Chapter III

IMPLEMENTING MONITORING AND EVALUATION SYSTEMS

The principles and procedures outlined in chapters I and II are distilled from a growing international experience in determining how well development programmes are working. The techniques discussed are not particularly difficult; to a large extent they are an application of long‑accepted planning methods and common sense. While implementation requires attention to some technical aspects, experience to date indicates that the main deterrents to successful implementation are non‑technical. These include lack of a firm politico‑administrative commitment to monitoring and evaluation, inadequate attention to cost and time considerations and absence of an appropriate institutional structure. Of these, the principal problem lies with what could be called "political will".

A.  Politico‑administrative commitment

It must always be assumed, when embarking on monitoring and evaluation, that decision makers genuinely wish to know how well their Drcgrarrmes are functioning and what results are being obtained. If decision makers cynically establish programmes without concern for whether they are effective, there is no place for monitoring and evaluation. The problem lies, rather, with circumstances in which decision makers feel that they can be assured of effectiveness without monitoring and evaluation systems, or fear that systematic monitoring and evaluation will show that programme performance is not as good as desired.

When decision makers fail to make a commitment to systematic monitoring and evaluation because they do not see its necessity, it is usually because they do not appreciate the nature of the data. Political leaders, particularly, appear to prefer "seeing things with their own eyes' to basing judgements on data produced by someone else. Common sense would appear to confirm that an on‑the‑spot visit to a programme is a better way to appraise progress than reading reports filled with "abstract" data. This is often accompanied by a lack of realization that what is seen is not necessarily a representative sample, nor are the observations systematic.

To avoid this problem, it may be necessary to devote considerable time to explaining to decision makers how information produced by the system can help supplement information obtained from other sources, including "seeing things with your own eyes".

The more common difficulty is when an initial commitment to monitoring and evaluation is challenged by the first data obtained. Few programmes, if any, are successful in achieving all of their goals, particularly at the beginning. A general rule of thumb for development programmes is that their results are neither as good as their proponents claim nor as bad as their critics allege. Indeed, one purpose of monitoring and evaluation is to detect substandard results in time to correct them. Nevertheless, when faced with initial data showing that certain goals or targets have not been met by a programme, many decision makers no longer want to obtain the data. While this is similar to avoiding bad news by killing the messengers who bring it, it also responds to a legitimate fear that publication of data showing substandard performance in a portion of a programme will lead to cancellation of the entire programme.

Since one means of dealing with data that indicate undesirable results is to discredit them, the research design is frequently a target of criticism. It is therefore extremely important to undertake a rigorous analysis of the technical soundness and appropriateness of the proposed model before proceeding with its implementation. Concurrently, it is necessary to convince decision makers who will use the data of the technical feasibility of the system.

The second means of overcoming fear is to ensure that the results of investigations reach decision makers responsible for the programme, but not necessarily the general public. Confidentiality of research results is one major means of distinguishing between pure and applied research. In pure research, the norm is publication of findings in pursuit of dissemination of truth. The purpose of research applied to monitoring and evaluation is to help improve the performance of programmes with socially worthy objectives. Publication of results showing partial failure might jeopardize the whole programme. For many decision makers, the data themselves are less worrisome than the possibility that they may be published.

It is therefore imperative that persons responsible for the research make sure that data, particularly of an adverse nature, reach decision makers who can take necessary steps, but are not published or disseminated outside the programme. This guideline begs the question of the responsibility of researchers to see that their findings are utilized. If there is an expectation that decision makers will take both positive and negative findings into account, there is no moral problem. If, on the other hand, there is evidence that findings will be suppressed, there will be a moral issue. A means of ensuring that findings will be taken into account is to present them in a balanced way. It is as bad to over‑emphasize negative findings as to minimize them.

B.      Cost factors

Questions of whether the benefits of monitoring and evaluation outweigh the costs may also hinder implementation. The difficulty is that the major benefit of monitoring and evaluation is avoidance of programme planning and administrative mistakes which result in wastage of resources. Thus, the major benefit of monitoring and evaluation would be seen in terms of reduced cost in the programme, but not as something easily demonstrable in a positive sense. Nevertheless, there are a sufficient number of known cases in which failure to diagnose problems in programmes has led to wastage of development resources to justify an investment in monitoring and evaluation as part of any programme. The issue is how large a share of programme resources should be devoted to monitoring and evaluation.

There are many general rules, such as one which prescribes that up to 1 per cent of the total project cost be devoted to monitoring and evaluation. But the best rule is that monitoring and evaluation should be as economical as possible while still supplying sufficient information to permit reduction of risk in decision-making. For some programmes, in which the reduction of high risks is deemed essential, resources devoted to monitoring and evaluation may have to be substantial. In most cases, however, the fewer the resources devoted to monitoring and evaluation (and to administration and management generally), the better. 

In general, experience has shown that costs can be reduced by (a) obtaining only absolutely necessary information; (b) applying only the most straightforward methods and techniques of research; and (C) using the data secured for purposes supplemental to monitoring and evaluation in order to reduce relative cost. Monitoring and evaluation would be most costly if the necessary data were obtained and analysed and then, for any reason, not used by programme decision makers. For this reason it must be clear from the outset that the system will be used. Indeed, if, after the system has been implemented, it is clear that decision makers are not and do not intend making use of the information generated, it would be preferable to discontinue the systematic monitoring and evaluation procedure.

C.      Institutional arrangements

Another possibility of improving the economy of monitoring and evaluation operations is to establish them on a regular, institutionalized basis. This could be done by setting up central evaluation units within the governmental bureaucracy. Depending upon which are the important levels of decision making in the political and administrative process, these units could be located at the national level, for example, within the national planning agency, and/or at regional and subregional levels.

The main function of evaluation units would be to provide government agencies with assistance in designing and implementing monitoring and evaluation systems for their programmes and to co‑ordinate the data‑acquisition efforts undertaken in connexion with various programmes. The establishment of central monitoring and evaluation units with functions so defined could enhance the efficiency and effectiveness of programme monitoring and evaluation in three ways. First, the staff of those units could be specifically trained for the functions assigned to them. Thus, national expertise would be developed in the field of programme monitoring and evaluation. The absence of such expertise in the past has often impeded the successful implementation of data‑acquisition systems.

Secondly, to the extent possible, standardization of the monitoring and evaluation concepts and methods used in different programme contexts would make it possible to combine the various data sets for purposes of re‑analysis and re‑evaluation. Thus, problems and issues not directly related to specific programmes, but rather to the development strategy or policy on which they are based, could be studied by retrieving the necessary data from a central information pool. Besides permitting a fuller exploitation of the information obtained, a central compilation of all data materials would improve the representativeness and accuracy of findings. In this way the usefulness of monitoring and evaluation as instruments of policy making, development planning and programme management would be enhanced.

Thirdly, the establishment of central monitoring and evaluation units would be consistent with the principle that responsibility for carrying out the monitoring and evaluation tasks should be located as close as possible to the focal points of decision making, in order to facilitate and expedite information feedback. Thus, while the tasks of monitoring and ongoing evaluation are best carried out by the programme staff, as stated earlier, the responsibilities for ex ante and ex post evaluation should rest with a central monitoring and evaluation unit. As an integral part of the government bureaucracy, this unit has direct, regularized and institutionalized access to the relevant bodies of policy making and development planning.

The longer‑term perspective of building up national information systems should always be considered when programme monitoring and evaluation systems are established, even though in most cases these will still be set up on an ad hoc basis at the present time.

PART TWO

FIELD APPLICATIONS OF THE SYSTEMATIC MONITORING AND EVALUATION APPROACH

Chapter IV

BACKGROUND INFORMATION ON THE SYSTEMS

An increasing number of countries are implementing systematic monitoring and evaluation as part of integrated development planning. Various approaches have been adopted. In most cases, the major obstacle has been obtaining the data required for the tasks of monitoring and evaluation. A number of countries have overcome this obstacle by designing techniques for acquiring the necessary data in a quick and. inexpensive way. While each of these programmes might have selected any number of data acquisition methods, all followed similar procedures.

The procedures described in this part of the source‑book are primarily based on the monitoring and evaluation experiences of the following programmes:

(a) Brazil. Social welfare projects of the rural extension programme of the Associaçao Brasileira de Credito e Assistencia Rural (Brazilian Credit and Rural Assistance Association (ABCAR));*

(b) Mexico. Programa de Desarrollo Socio‑Econ6mico de los Altos de Chiapas (Chiapas Highlands Socio‑economic Development Programme (PRODESCH)) of the Mexican Federal Government and the Government of the State of Chiapas;

(c) Panama. Training and promotional programmes for rural development of the Direcci6n General de Desarrollo de la Comunidad (Directorate General of Community Development (DIGEDEC0M)) and the Ministerio de Desarrollo Agropecuario (Ministry of Agricultural and Livestock Development (MIDA));

(d) Venezuela. Programa Integrado de Desarrollo Agropecuario (Integrated Agricultural and Livestock Development Programme (PRIDA)) in the Guarapiche river basin; this was executed by the Centro Nacional de Capacitaci6n y Investigaci6n Aplicada en el Desarrollo Regional y Local (National Centre for Training and Applied Research in Regional and Local Development (CIADEC)) of the Oficina de Coordinaci6n y Planificaci6n (Office of Co‑ordination and Planning of the Presidency (CORDIPLAN)).

In addition, work has been conducted along similar lines in Egypt (Ministry of Land Reclamation), Nicaragua (PRODESAR), Paraguay (Consejo Nacional de Progreso Social) and Costa Rica (Direcci6n Nacional de Desarrollo de la Communidad).

The method outlined in part one was chosen by the separate programmes for basically similar reasons. It is useful to describe these reasons in some detail in order to illustrate some of the major issues in monitoring and evaluation.

A. Reasons for data acquisition

1.      Lack of available data series

The programmes could not draw on any existing data series which would permit evaluation. While all of the countries had well‑developed national censuses, these usually did not break down data by units comparable to those of the programmes. In one country, for example, census data could be obtained for the State but the programme in question was concentrated in only part of the State. In anouther country, the programme covered the entire country, but concentrated only on low‑income residents. Thus, census data could not be used to assess change.

Moreover, in most of the countries, the published census data were over 10 years old. In one country in which a census had recently been made, the data were still being tabulated and were not expected to be available for several years. No other relevant statistical series existed in any of the countries. As a result, the alternatives were either to collect new data or to have no reliable data at all to assess programme results.

2.      Complex data requirements

All of the programmes concerned were integrated approaches to socio‑economic development. Thus the data required for monitoring and evaluation were highly varied. As a minimum, reasonably accurate economic data as well as information on health, education and social participation were needed. None of these data could be obtained without a special effort.

3.      Minimal resources for data collection

Most of the programmes covered had very tight budgets, so that any funds spent on research had to be diverted from the beneficiaries of the programme. Any data collection, therefore, had to be at minimum cost. In practice this meant that the personnel for the research effort would have to be drawn from existing programme staff, and their data collection activities would have to be consistent with their other programme activities.

4.      Lack of staff specially trained in research

Only one of the programmes, ABCAR, had staff specially trained in research and even its trained staff was small. In most cases, the programming and planning personnel had had no previous experience with research design or analysis, although a number had acted as interviewers for previous studies. As a result, it was necessary to collect data in a way that would facilitate their use by an enthusiastic but inexperienced staff. Moreover, the methods for collection had to be sufficiently clear‑cut to be understandable by the field staff doing interviewing arid sampling, few of whom had ever done so before.

5.      Shared experience

A positive factor was that there was some awareness in the various programmes of each other's experience. In each case, United Nations technical co‑operation, funded by the United Nations Children's Fund or the United Nations Development Programme, was available, providing a direct link among the different efforts, In addition, there was an exchange of documentation among the programmes and, in the case of one programme the staff was sent to another national programme under a United Nations study tour. As a result, the programmes tended to adopt a similar approach to data acquisition.

B.  The implementation experience

The results of the exercises have been mixed. In some cases, the evaluation system was established only to be abandoned when major changes were made in the programmes being evaluated or when there were major shifts in personnel. In other cases, the methods continue to be used in differing degrees. In each case, however, a major base‑line study was undertaken as a minimum and in most, at least one set of restudies was carried out. The data that were obtained proved highly useful in modifying and improving programmes. In one case, the results helped prove to a sceptical new government that a socially oriented programme was more efficient than several economically oriented programmes. In another case, the results helped generate a major change in programme management.

In all cases, the relative cost of the effort was sufficiently low to permit continued inclusion in the programme and the data were timely enough to affect programme decisions. Moreover, it proved possible to quickly train staff to collect and analyse information.

The problems in implementation tended to be of two types. First, outside factors impeded the programmes themselves. Thus, changes in government often meant that the programmes being evaluated were abandoned in favour of new efforts. Secondly, staff assigned to the evaluation efforts sometimes were replaced. In some cases this was for political reasons, in others it was due to personal changes. In one case, the staff member in charge of the evaluation was recruited by the United Nations to provide technical assistance in evaluation to another country. Staff changes generally resulted in delays due to a lack of technical expertise at key points in the process. In one instance, data were not used at all after the staff change because no trained person was available to process and analyse the data which had been collected.

1.      Responsible agency

In contrast to what was proposed in part one as desirable relationships between planning, evaluation and administration, in only one of the four programmes was the direction of the evaluation effort closely connected to planning. This was in Mexico, where the responsibility for planning and directing the Programa de Desarrollo Socio‑econ6mico de los Altos de Chiapas (PRODESCH) was vested in the office of the Governor of the State. This office was directly concerned with overseeing the evaluation, and thus nominally could link results to planning and management. In Panama, the evaluation system was the responsibility of a group of staff whose main function was training, and the relationship with the planning staffs in both DIGEDECOM and MIDA was more informal. In Brazil, the programmes were executed by state affiliates of ABCAR, which is itself a federal quasi‑governmental agency. The evaluation was directed by ABCAR's central research group as a service to both the state affiliates and planners in ABCAR's central office. In Venezuela the evaluation was undertaken on an informal "subcontract" basis by an autonomous government institution whose main function is training.

In most cases, the fact that the responsible agency for evaluation was not closely connected with the decision‑making process in the programme meant that the communication of results was more difficult, since additional effort had to be made to make the decision makers aware of the results and explain their significance.

2.      Costs and sources of personnel

The direct costs of the evaluation systems in all cases tended to be low for the same reason: the personnel directing the evaluations were regular staff, and the personnel collecting the data were also either regular staff or were seconded from other agencies. In the case of PRODESCH, the interviewers were teams directed by local extension agents and largely composed of field assistants. In Panama, the interviewers were community development promoters released from other work for two weeks. In Brazil, depending on the state, the interviewers were either extension personnel released from other duties for a short period, or programme trainees who conducted interviews as part of their training. In Venezuela, the data were collected by the teaching and research staff of CIADEC together with trainees in one of the major courses at the Centre.

For the most part, the costs were therefore indirect and can only be calculated in terms of "lost opportunities" for other activities which might have been undertaken instead of data collection. At the same time, the involvement of personnel in data collection may also have indirect benefits. For example, the exercise of conducting systematic interviews often acts as a mechanism for self-shy;training because the personnel are forced to confront a development problem in an integrated manner and to interact with programme beneficiaries. Field personnel may thus have an improved sensitivity to the inherent problems of a programme. An example can be found in one of the countries where interviewers were field staff of the programmes. One interviewer was the area head of an agency providing food assistance. One of the variables in the evaluation was the use made of the food assistance that was provided. During the course of standardized interviewing, a programme beneficiary was asked whether the food that was provided had been eaten. The respondent indicated that everything had been good except the powdered milk, which had caused stomach cramps. The food assistance official was then prompted to ask why this was the case. He learnt that when the food was delivered, no one had explained to the target population, who had never used powdered milk before, that the powder had to be mixed with water to create a liquid. Consequently, the population had consumed the milk in powdered form. The interview led to a modification in food distribution procedures to ensure that the incident was not repeated.

3.      Time constraints

All of the programmes had time constraints for obtaining data, in terms of either work schedules or budget cycles. In the case of Panama, it was important to collect and analyse data before the beginning of a new major training effort. In Mexico, data requirements were geared to the annual budgeting cycle, while in Brazil the deadlines were dictated by both the budget cycle and planned reorganization. In Venezuela, the time constraints were defined by a major international loan package. In all cases, the information produced was most valuable if it was made available on or before certain dates.

4.      Types of information

The procedures used were designed primarily to obtain data on programme outputs and results. Data on programme input delivery were usually available through normal accounting and reporting procedures already existing in the agencies involved. Thus, no special effort was made to develop new reporting systems and the entire effort in data acquisition was directed towards assessing results.

The methods used to acquire and analyse these results are described in considerable detail in the following sections. It should be borne in mind that these procedures are not perfect and, in most cases, involved compromises in terms of precision and coverage that were dictated by the constraints of time and resources. It should also be recognized that the model followed by all of the cases, while applicable to their conditions, may not be so in all conditions. It should be further recognized that, in view of the large number of assumptions and considerable experience that underlie the methods presented here, the presentation is brief. Although brief, it is sufficiently detailed in order that a person with reasonable knowledge of research methods might apply the techniques directly or adapt them to the needs of a given programme.

Chapter V

BASIC STRUCTURE OF THE MONITORING AND EVALUATION SYSTEMS

The structure of a monitoring and evaluation system is determined by the information that is required for regular review and appraisal of programme results, that is, by what is to be measured for assessing programme progress and impact. Since the purpose of all the programmes for which systems were designed was to improve the social and economic conditions of a defined population, the system had to measure the degree to which these conditions had improved. Thus, information had to be collected which could show the results achieved and impact of inputs supplied by the programmes.

As has been noted previously, two types of information are usually desirable in order to permit systematic monitoring and evaluation: (a) data on results of programmes in terms of subjective change (changes in the attitudes, motivations and behavioural patterns of individuals and groups) and objective change (changes in the socio‑economic living conditions of individuals and groups); and (b) data on methods for producing these results. The analysis of results is basically concerned with observing levels of certain indicators at a given point in time. Analysis of the process through which results are obtained is essentially dynamic in that it concerns behaviour of change agents over time. Thus, he monitoring and evaluation systems used here had to be designed so that both types of analysis could be included.

The procedures for designing a system with these two types of analysis in mind are illustrated by the case of Panama. [1] Here, the specific programme to be evaluated involved training field‑level personnel of the national community development agency (DIGEDECOM) and the Ministry of Agricultural and Livestock Development (MIDA), as well as local leaders in community development concepts and skills. Thus, it involved general upgrading for rural development programmes. In this sense, two programmes had to be evaluated: the specific training activities and the rural development programmes for which the training was an input. The direct effects of the training programme were seen in terms of the trainees, the indirect effects in terms of the people with whom the trainees worked. In both instances, there was concern with measuring both results and procedures. On this basis, a schematic picture of the four aspects of the programmes to be measured can be made; this is presented in figure VI.

Collecting information on these four aspects would permit evaluation in terms of a number of relationships, each of which would supply a different type of conclusion. If each cell in figure VII (AC, AD, BC, BD) represents a different aspect, then five types of analysis could be made:

The evaluation system in Panama, since it dealt with both a training programme and the rural development programme in which training was an input, was somewhat more complex than those designed in Mexico, Brazil and Venezuela. The latter three systems were only concerned with relating field results to field procedures. Nevertheless, the method by which a training subprogramme was related to a larger programme could be added at any time that this became important.

In the four programmes, the first step was to determine how data were to be analysed and where they were to be collected. This was largely dictated by the desire to be able to infer causal relations at the point of interface between the programme inputs and the beneficiaries, as noted in part one, chapter II, section .

The most acceptable type of comparison was seen to be the before‑after comparison in the same locations. Thus, for example, if there was a difference in income between two measurements, change could be assumed to have taken place. It would then be possible to observe what changes in the delivery of programme inputs, programme procedures and other factors had also occurred and relate these programme changes to results, in order to determine what factors appeared to have caused the observed results. While the data acquisition procedures were primarily designed for a before‑after comparison, they would also permit a comparison of results with targets and could permit interunit comparisons.

In each case, a decision was made to generalize to the level of the community, since this was the point at which the development programmes interacted with beneficiaries.

Taking into account the types of comparison to be made, the unit for generalization and the types of information to be collected, monitoring and evaluation systems consisting of three elements were designed in each of the four countries.

First, a set of base‑line case studies was carried out among a randomly selected sample of all communities being affected, or to be affected, by the programme. These were designed (a) to provide a point of comparison for measuring changes over time, and (b) to provide data for a planning diagnosis.

Secondly, a systematic restudy was to be made of these base‑line communities to determine changes which would have occurred over time, thus providing the central core of the evaluation by relating these changes to inputs and programme procedures. In order to ensure that continuous data were provided, these restudies were to be staggered over a number of years. Thus, for example, in Mexico, 23 base‑line cases were studied in 1973 and 1974. During 1975 and 1976, the years included in the current stage of the programme, 10 cases each were to be restudied in order to provide an estimate of programme progress and impact. This is exemplified graphically in figure VIII.

Thirdly, reporting methods were designed to keep a record of inputs and procedures for comparing and relating them to results, such as the evaluation of training courses in Panama.

Results as of year 0 would be estimated by intercase comparisons; results in year 1 by noting changes in restudied communities 1, 14 and 7; results in year 2 by noting changes in communities 2, 5 and 8, and in year 3 by noting changes in communities 3, 6 and 9. The over‑all result of the programme would be determined by comparing the values observed in year 0 with results obtained in a new major set of base‑line studies in year 4. This information, coupled with data on input delivery, would provide a regular basis for judging programme performance.

Chapter VI

DESIGN AND IMPLEMENTATION OF BASELINE STUDIES

The first stage in building an evaluation system typically involves design, execution and analysis of the baseline studies in order to establish the frame of reference for subsequent comparisons on which evaluation will be based. Since for these comparative purposes the data to be collected subsequently must be similar to those collected in the baseline studies, the methods of selecting and conducting these baseline studies and their content are extremely important. In effect, the principal conceptual work for the evaluation of a programme must occur at this stage, since the nature of the entire monitoring and evaluation system will be determined here. Moreover, the largest volume of data that is collected at any one time is obtained in the baseline studies. This stage will therefore involve the largest number of personnel and require the greatest amount of time. As a result, it is the most costly stage in the design and implementation of the system. This cost factor led to the decision to make a double use of the baseline studies in each of the four cases under review here. On the one hand, data would be used for subsequent comparisons; on the other, data would be used for a diagnosis of the existing situation of potential beneficiaries of the programmes for planning and programming purposes. In several cases in which programmes were already under way, the baseline data would affect implementation rather than formulation of the programme plan.

The importance of this stage makes it necessary to proceed in a logical and efficient manner through a sequence of steps in order to define the content of each system. Many of these steps are common to any form of applied social research, but since the main purpose of the effort is practical evaluation of programmes, the details and procedures in each step differ somewhat from academic practice. Thus, in addition to normal care to ensure that the study design yields scientifically valid information, it was necessary in the case of the four country studies to keep in mind three rather practical questions:

The objective of the design process was to ensure that the system remained practical, rapid, efficient and inexpensive. Achieving this involved certain costs. First, the usefulness of the data for purposes of deriving or testing theories was limited. Since certain short cuts were made in order to achieve the wide coverage necessary to monitor and evaluate complex programmes, measurement of individual variables could not be very precise. Moreover, sampling procedures did not yield, within cost constraints, the most precise sample possible for individual variables.

On the other hand, while research procedures used in each system were rather unsophisticated, they had the advantage that relatively unspecialized people could utilize them with a minimum amount of training. The use of non‑researchers for implementing the system had an inherent advantage in that a practical approach to data collection would be maintained, since the people responsible were not likely to be detoured into theoretical or methodological discussions. [2]

A. Step 1: Determination of what to measure

The first step in any design of a baseline study consists in determining what variables to measure, i.e., specifying the substantive content of the study. In academic research this usually means elaborating a set of hypotheses. By contrast, for the type of research necessary for a monitoring and evaluation system, the step consists in determining what information policy makers, programme planners and administrators require in order to ascertain whether or not the programme is functioning properly and why this is so. This step is perhaps the most difficult, since programmes do not always specify their objectives clearly and in measurable terms. Indeed, many of the objectives of a given programme are not even stated formally. In addition, although most development programmes are multidisciplinary, the programme personnel often exhibit a particular professional bias towards obtaining one or another type of information deemed necessary for monitoring and evaluation. Thus economists may tend to be interested only in economic data, while sociologists may be interested only in social data.

In each of the four countries, the procedure for determining variables on which data were to be collected consisted of a dialogue between those charged with evaluation and those responsible for the operation of the programme to be evaluated. For the evaluation system of the ABCAB social welfare projects in three Brazilian states, [3] the research team held several day‑long meetings with state‑affiliated agency heads and field‑level supervisors to discuss what information was needed. On this basis, a specific list of necessary information was prepared. The dialogue centred on a series of questions:

In these discussions the evaluation personnel from ABCAR had a dual role, that of questioning as well as orienting the field supervisors to a multidisciplinary approach to project monitoring and evaluation,

In Panama, similar meetings were held with the directors of planning units of DIGEDECOM and MIDA, the two agencies involved. For the Guarapiche river basin study in Venezuela, separate meetings were held with local programme heads of the various agencies in the integrated rural development programme. In Mexico, the system design was developed as part of a seminar on evaluation techniques sponsored by UNICEF for the local operational supervisors of the agencies that were involved in the integrated development programme (PRODESCH). [4] As part of this seminar, the information needs of each agency and sector of the programme were discussed and defined.

As a result of such discussions with the programme staff, a master list of required information was put together for each programme. The items on the list naturally varied from programme to programme according to the objectives of each but there were a number of common items, since each of the four programmes was concerned with rural development. In general, the following types of information were required:

Each item of required information was included because of its relation to a project goal. For example, in Mexico health programmes executed under PRODESCH involved community work through the use of food incentives which should have resulted in dietary changes in the Chiapas highlands. In Brazil, if ABCAR programmes to develop a system of small health posts (mini‑posts) in rural areas were effective, there should have been change in such health practices as use of latrines, home cleanliness and personal hygiene. In Venezuela, the integrated rural development programme that works through credit users associations (uniones de prestatarios should have resulted in increases in use of certain technological inputs and in production per hectare. In Panama, the effectiveness of leadership training programmes should have been observed in a qualitative change in the type of local leadership, from the traditional chief, whose role was defined ascriptively, to a modern democratic leader, whose role was defined by achievement.