Information Audit by CAMRIS

In a nutshell An information audit identifies the information needs of an organization and matches them against existing services and resources. It is considered the first step in the development of a knowledge management strategy, or in this case an evidence-based decision strategy for the development sector. The steps in the information audit are:
Pre-planning and planning Data collection Data analysis Data evaluation Communicating recommendations Implementing recommendations The information evaluation system as a continuum Definition One widely accepted definition of an information audit is "a systematic evaluation of information use, resources and flows, with a verification by reference to both people and existing documents in order to establish the extent to which they are contributing to an organization's objectives" (Orna, 1999). Although there appears to be no universally accepted definition of information audit, this definition from the Association for Information Management in the United Kingdom seems to incorporate most of the critical elements required by such an activity. Our only addition would be the element of information use for explicit resource allocation. While this is implied in the idea of responding to an organization's objectives, it makes more explicit the pathway for reaching those objectives.		Difference between data and information There is a conceptual difference between data (an organized set of observations that can be communicated) whereas information is much more than that. Quoting Bateson: "Information consists in those differences that make a difference" Although Bateson's work spans many disciplines (a multi-disciplinarily which makes his work unique), he began as an anthropologist (though he would have described himself as a biologist) and is most likely to be described today as a cybernetician. He is most famous for his theory of double-bind which provides a framework to understand the development of schizophrenia, but it is his learning and communication theory based on definitions of difference, context and pattern. Bateson, Gregory. (1979) "Mind and Nature: A Necessary Unity", Wildwood House Ltd. p.98-99
Our revised definition of an information audit therefore is: "An information audit (IA) is the systematic collection of standardized data to evaluate data collection, information or knowledge generation in an organization. Using data collected from documents and persons, the IA establishes what information is required and currently being utilized to make resource allocations that impact an organization achieving its objectives." Purpose The information audit is therefore a process used to: Identify the information needs of the organization Assign priorities and/or a level of strategic importance to these information needs Identify the resources and services currently utilized to meet these needs Identify information flows from the external environment to the organization which inform how well the needs are being met, including the OR needs and validation of the other data flowing into the M&E system Analyze gaps, duplications, inefficiencies, and other areas where changes might be necessary Review resource allocation as it relates to information transmitted and data collected The IA addresses the nature, scope, coverage, periodicity, validity and other meta-data characteristics of all of the data and meta-data currently available and that might be collected with the purpose of being utilized for decision-making and resource allocation within the organization. In sum, provides a highly specific level of understanding of how data feeds the task activities performed by the units and how they are data driven. The use of other informational tools such as geographic information systems should also be explored in that the ability to introduce deficiencies in the data system that lead to improved health status of the population. Understanding the relationship between data and action is vital to the continued improvement in the performance of personnel. Preliminary activities An information audit includes the following preliminary activities: Selection of criteria to identify the best and worst performing cases in the current information system; Develop case studies of what organizational units (e.g. districts, etc.) could be classified as the best and the worst instances of current performance; Conduct careful information audits of the selected cases; Develop a plan to extensively redesign the existing systems. A new system which eliminates existing information gaps and incorporates new elements to monitor, manage, and evaluate the impacts of increasingly complex development problems within the context of a decentralizing national information system. Devise and conduct a information audit on one of the system`s modules as the first step in designing and developing an improved M&E information system. The new M&E will be based on the results of the outcome of the ongoing information audit. Methodology Because the audit process needs to be designed to minimize obstacles and maximize opportunities within the organization, it is important to be able to capture in very clear objectives what the user expects from its data systems and to understand the organization in depth. Step 1: Pre-planning and planning The planning effort for an information audit needs to start with a general group process which outlines the organization and the objectives of the information audit as it relates to the organization. A complete review of prior activities in the data information sector would provide a sound base and opportunity to assure transparent input into the information audit process. At the same time, another important element is to enlist senior management support from decision level. A general agreement on a methodology needs to be generated such that implementation can begin. Pre-planning Mapping and identification of information flows - The methodology begins with mapping out and identifying all information flows. After key data inputs and points of collection are listed and the flow of data charted, the second part of the process is to identify strategically significant information resources and gaps. Identification of strategically significant knowledge assets - The next planning step within this category is to identify the tasks and activities that produce strategically significant knowledge assets based upon the data collected in the field. We then identify the process by which this data is transformed into information at the local level, the district level, and the national level. Planning Determination of the scope and resource allocation - The next step of the planning process would determine the scope and resource allocation program for this activity. This involves resource allocation for only the prototype information audit, not the development of a new information system. The existence of the current information system exercise, provides a good quality check for the quality and appropriateness of the information audit. The old and new information systems will need a baseline measure against which to measure progress. The broad range of the rate of current implementation of the current M&E information system suggests that there is still much to do in this area to reach the goal of countrywide implementation. In this case we would expect to identify the resources that are available to conduct the activity and define which parts of the organization will be included. After the priority district(s) is selected, the approach would be to select one of the best performing and one of the poorest performing units (communities, health centers, etc.) in order to collect the information required by the audit. The basis for allocating a budget would be somewhat dependent upon which unit of analysis is selected. A clear idea of the size of the district and allocation of resources would be needed before starting the prototype. First, field trips would be required to collect data. Before such trips a close examination should be made for local leadership to help the process. Some data processing time for data analysis preparation of reports and review needs to be included. A decision about the potential use of outside consultants would be appropriate as well as what resources from within the ministry would be available in terms of persons and time. Methodology selection for data analysis - The next step involves selecting a methodology for how the collected data would be analyzed and evaluated, After an initial case method approach is defined and administered as the contextual basis for comparison a series of structured interviews with key individuals at local level would complete the primary data collection techniques suggested. It is important to remember that during the information audit process all concerns expressed by individuals at the local level need to be recorded and responded. This not only ensures a higher quality of data input but the feedback will generate support for future implementation of the program minimizing resistance to suggested changes. During this methodological design phase standardized data collection instruments need to be developed and pre-tested. Once this is accomplished, training for individuals who will be doing the fieldwork will be undertaken. Given that these will be primarily situation and case analyses activities, problems of statistical sampling will not be present unless the decision to examine client's response at the population level is taken. We believe that at this time interviews with communities currently in the system will be sufficient to provide the kind of information we are searching for in the information audit. It is strongly recommended that teams composed of stakeholders from the government level take primary responsibility for the data collection phase. Their knowledge of both the political and information/evaluation environment within which they will be collected and utilized will be important to the process. Development of communication strategies - The next step involves developing communication strategies that will facilitate cooperation and support before, during, and after the audit. We must convince ourselves and the public that the detailed effort required for a good information audit is first possible and second will have significant payoffs for a better operating and managed projects. This will involve identifying key stakeholders and decision makers at each level from the local health center to the ministry and beyond. The primary method for this identification will be personal interviews of stakeholders from within and without the system. Step 2: Data Collection (Inventory and Testing) This stage involves collecting the data required to meet the objectives made explicit in the planning stage. We expect data to be collected in this process through personal interviews with a semi to fully structured questionnaire. The types of data to be collected include data relating to information required to perform tasks and/or activities that collect and process data at all levels the level of importance of data and resource allocation activities related to the use of data at all levels starting with its utility to the individuals that collected the information in the field. Step 3: Data Analysis The primary objective of this activity is to identify gaps duplications and endpoints in the current M&E information system and to identify where there are problems in how the data are collected and turned into information. The first stage of data analysis will be information flow mapping. Here the data are followed from the point of collection to their compilation, presentation, and utilization. After the information mapping is completed, other inputs are sought to ascertain the strategic significance and use of the data. The final analytical framework is a general analysis of the flow and utility of data within the system. We would suggest that detailed and careful recording of the exact wording of all responses be undertaken such that a content analysis could be performed in support of the questionnaire data. The ultimate goal of the significance content analysis is to generate the following types of reports: Tasks supported by each information resource and data item and the use of that data by the resource or task manager. The importance of information resources to the allocation of resources. The relationship of the prior two steps to organizational objectives and strategies. The identification of tasks or objectives for which data are not provided. The identification of cases where there is duplication of resources and/or data that are not identified by the system. From this analysis mapping, gaps and inefficiencies in the system will be evident by visualizing information flows. Such an analysis often makes it possible to identify bottlenecks and inefficiencies in the system. Other problems include information gatekeepers who control access to data and information "dead ends" where much data is going in but no decisions are being made based on the data that can be identified. We should also be able to identify where there are gaps in the provision of critical resources and over provision or excess amounts of services. Finally, balances and biases in the data and information provision process should be identified in the analysis. Step 4: Evaluation and Interpretation of Data Data are evaluated and interpreted within the context of the organization. This requires understanding the data in the context of organizational constraints, culture and resources. In general, problems that are identified relate to the provision of data and its conversion to information, and include some of the following examples: Information is used by individuals to increase their power or base of operations and is kept from others. This often leads to the biased distribution of resources and/or the use of substandard resources both of which need to be identified in the information audit. Gaps in the provision of resources are important and should be identified during the information audit. Issues of information overload or excessive amounts of data coming in which are not utilized or reflected in related activities. A lack of transparency and accountability would be documented during the information audit process. The lack of traceability or the ability to find out who exactly is responsible for generating, developing, and utilizing data/information. From this data evaluation paradigm, it is important to emerge with the following questions answered from the information audit. First, does a data problem exist that has strategic significance? Does that problem affect the achievement of organizational objectives and ultimately impact the health of the population? Is there a specific reason that can be identified for the problem? If so, does it matter with respect to the solution? What are the cost implications of letting the problem continue or to resolving the problem? What alternatives are there to dealing with the problem in ways that can be effective? Finally, what are the implications of changes introduced into the system, who will be affected, what other services will be impacted, and what barriers will be present? For each problem that is encountered there may be multiple solutions, and it is important that these solutions be presented such that the best solution is taken. Ideally, we should be able to identify a priority score which objectively weights solutions and sets priorities with respect to cost. Thus, the data analysis section of our information audit should conclude with a set of recommendations that identify where the data problems are, who or what is responsible for the problems, possible solutions for resolving the problems, and the costs and human and other resources needed for resolution. Step 5: Communicating Recommendations to Appropriate Stakeholders and Policymakers Mediation and communication strategies are important throughout the entire information audit process. It is critical that once recommendations have been formulated that they are communicated to persons integral to the process of resource allocation and decision-making. Changes in the system will be required if the information from the information audit is to be appropriately utilized. This almost always includes a change in the way resources and personnel are allocated and this creates some resistance to the process. Since this information audit is intended to be a prototype which ultimately would be applied to all of the different units of analysis, we would recommend that a seminar be held or, if appropriate, a series of seminars where the written report could be reviewed with its applications to other districts the major point of discussion. Written reports, as well as a database including all of the documents that had been examined and utilized, would be produced as well. Step 6: Implementation of Recommendations The findings of the information audit essentially would imply changes in strategy and implementation. Management will need to examine the interrelated effects of these changes particularly with regard to how they would apply to the project. Ultimately, a comprehensive implementation plan and post-implementation review strategy will be needed to minimize resistance and assure that everyone is onboard. Since this activity is designed to be the baseline for a much larger effort involving all of the municipios (or local units), recommendations here will be in the form a proposal for an expanded system including a nationwide Information Audit. The fact that this proposal will be evidence-based and a prototype already exists implies a high potential for success. Step 7: The Information Management Evaluation Continuum The final stage of our prototype Information Audit involves placing the information in the context of the overall data and information system. In the past, information systems were separated from evaluation systems because of the long period of time needed to evaluate the impact of specific interventions on the quality of life of the population. We believe that this process is no longer necessary and that surveillance, management, and other routine information systems should be combined. This combination would result in a continuing systems approach which would be utilized to add, supplement, and adjust the information resources database to reflect changes in significance, tasks, and organizational structure. Although less important in a prototype system, any information audit taken to a larger context would include decisions regarding data maintenance. Additionally, regular visits from objective outside reviewers are required to examine and evaluate the process. Another information audit should be implemented in five years to evaluate the changing needs of data and the population that is to be served. Conceptual Framework The information audit for health information systems in Africa was first utilized in the design of the Niger national health information systems developed in the early 1980s. (Bertrand et al, 1988). Although various similar methodologies had been available from the systems and information sciences prior to this, we believe that this was the first application to social sector problems at the national level in Africa. Since the early 1980s, the field has grown slowly; a number of publications documenting this slow progress are available[1]. The evidence-based paradigm currently popular in medical circles is central to addressing the need for effective data and knowledge management as the basis for all social sector interventions. Data turned into information and processed as knowledge should be the basis of all decision-making related to the allocation of resources within the development sector. Controlling the acquisition of and access to data is a complicated and difficult process, particularly in the domain of decentralized and developing information systems. Many organizations, including units within large organizations operate in such a way that they are effectively independent of one another. Consequently, although they should ideally rely on a single set of data and information sources, individual units often design and implement their own data collection activities. The result is quite often a hodgepodge of different data collection systems, reporting mechanisms, and methods of data use for decision-making. For efficient and useful management of development project there must be a clear and visible alignment of the information that is required by policy makers for resource allocation. Our challenge in this prototype study is to identify precisely what data are needed to achieve organizational objectives that are ultimately intended to improve the quality of life of the population. The information audit should effectively determine the current information environment by first identifying the information required to meet the needs of the organization at the local level. The strongest reason for this is that only if the local level needs are met will the quality of information collected be sufficient to feed the information system. Figure 1 illustrates the model of how data are processed and collected internal to an organization, then filtered, further transformed, reused, stored, transferred and/or made public The data–to–information process creates knowledge that is both explicit and tacit. Explicit knowledge is the output of tasks and activities that can be documented as reports, databases, procedures and actions. Generally, it is relatively easy to capture, store and communicate. It is analogous to domain specific knowledge in that it is the description of things known to the organization. Tacit knowledge resides within each worker who needs to use data for allocating time and resources and achieving objectives. It consists of lessons learned from doing the job, along with the experience created through understanding the environment within which the job is undertaken. Tacit knowledge is more like analytical learning processes in that it implicitly combines knowledge and means of knowing of a specific person with approaches to answering questions for which there may not be a single known or correct answer. Any information strategy should develop means of assessing all kinds of tacit (domain) and explicit (analytical) knowledge (Buchanan, S.J., 1998) The information audit process allows us to map information as it flows between the population served and the entities providing the service. Because of the decreased amounts of time required to process and report data that modern information technology has made possible, data systems and information systems can become real-time evaluation systems, making them more important to routine management issues including disease surveillance and response. The outline that we propose is a structured flexible framework that will be adjusted in the field to meet the needs of the organization. This will allow us to understand the context of the data and information system as well as to tailor objectives to the organizational resources that are available. For an example of Information Audit carried out by CAMRIS please click here. [1] An abbreviated bibliography is included at the end this document.