ST/ESA/78

Chapter I
DEFINITIONS AND CONCEPTS OF SYSTEMATIC MONITORING AND EVALUATION

Monitoring and evaluation have become widely used terms in the literature on programme planning) implementation and review. They are frequently used as interchangeable synonyms, but actually refer to different activities and processes. Although closely interrelated and interdependent, monitoring and evaluation need to be distinguished clearly if they are to be systematic and reliable tools for decision‑making in programme management, programme planning and policy formulation.

A. Definitions

Before entering into a discussion about the methodological issues and practical problems involved in designing and implementing monitoring and evaluation procedures, it is necessary to define precisely the key terms to be used and to explain the implications of the definitions set forth. This should help to avoid terminological ambiguities and misunderstandings, particularly when the discussion concerns other works on programme monitoring and evaluation.

Monitoring and evaluation systems are discussed with special reference to integrated socio‑economic development programmes. Thus, the definition of monitoring and evaluation has to be related to a description and specification of the basic characteristics of such integrated programmes. Section 1 deals with the essential attributes of development programmes in general and the distinguishing features of integrated socio‑economic development programmes in particular. The meaning of the terms monitoring and evaluation is the topic of section 2

1. Integrated socio‑economic development programmes

The term "'programme" is meant to refer to a planned complex of activities, including utilization of resources, undertaken to reach specified objectives within a certain time‑limit and cost‑frame.

For operational purposes, programmes must be planned as a sequence of subunits comprising defined sets of implementation activities for defined implementation purposes. These subunits are called "projects", 1/[1]

Development programmes and the strategies from which they are derived, can be said to be integrated if they take full account of the close interrelationship between economic and social factors and the interdependence between the various sectoral subprocesses of development in: (a) analysing the existing impediments to socio‑economic change and progress; (b) formulating development goals and objectives; and (c) planning practical measures needed for promoting developmental change. Owing to their multifaceted approach to development analysis and planning, integrated development programmes typically comprise several sectoral project components designed to be mutually supportive and/or functionally interrelated.[2]

Example An example of an integrated development effort is the activities undertaken by the Chilalo Agricultural Development Unit (CADU) in Ethiopia.[3] The CADU programme was intended to serve small‑scale landowners and tenant farmers. It included the following project components:[4]

(a) Improvement of milk collection and grain distribution in order to establish markets with fair prices that would make it attractive for farmers to increase their production;

(b) Use of extension workers and model farmers for stimulating the target group to buy and use high‑yield varieties of wheat and fertilizers;

(d) Engagement in agricultural research and experimentation for the purpose of obtaining a continuous flow of well‑tested methods and means as a basis for extension work;

(e) Motivation of the local people to assume increased responsibility for the development work by helping them to learn to help themselves, in order to make development a self‑sustaining process;

(f) Training of staff for similar, subsequent programmes.

Another example worthy of mention in this context is the Matourkou programme in Upper Volta. It comprises three main sets of activity: "Firstly, it is a pilot project for rural development. In a model village and 10 communities or villages in the surrounding areas, it has encouraged farmers to adopt a settled form of agriculture instead of the prevailing cropping systems, and has introduced some modern farming techniques including animal traction, improved seeds and tools, and the use of fertilizers and pesticides. In addition, it has provided the communities with health and educational assistance. Secondly, the project trains extensionists at various levels, and 'technical agents' in agriculture, forestry, irrigation and livestock. Thirdly, it has established and equipped a 1,000 hectare farm for experimentation and applied research in horticulture.) seed multiplication, livestock breeding and training for animal traction.[5]

As can be seen from the examples, the projects included in a programme may be linked to each other in either a causal or supportive way:

Causal (means‑end linkage

In this case, a project is intended to produce the inputs to, or the pre‑conditions for the functioning of, subsequent ones. It thus relates to the programme objective only indirectly. That is, the project affects the overall programme objective through its impact on other projects included in the programme. This type of linkage can be seen in the second example presented above, the Matourkou programme. The results of the applied research projects will provide inputs to the training of extensionists and technical agents. Their improved knowledge and skills are, in turn, inputs to the training of farmers which may help to improve their production techniques, increase their productivity, and ultimately lead to an improvement of their socio‑economic living conditions. This chain of events, i.e., the causal relationships that exist between applied research, training of trainers, extension work with farmers etc. and the programme objective, can graphically be depicted as follows:

Supportive linkage

When the projects comprising a programme are mutually reinforcing so that one project supports the effects and impact of the others and vice versa, the linkage can be termed supportive. An example is the relationship that exists between the above‑listed CADU projects (a), (b) and (c). on the one hand, and (e), on the other. The linkage can be illustrated as follows:

Programme objective I in this case is to increase farm production the strengthening of popular participation in the development effort is stated as programme objective II.[6]

Programme objectives, as already indicated, state the reason or purpose for the programme or, in other words, the changes the programme is expected to initiate and accomplish in its environment. If the ultimate objective is to satisfy human needs and improve peoples living conditions, that is, if the programme seeks economic growth and change for social purposes, it is said to be a socio‑economic programme".

Under present circumstances, programmes aiming at such objectives as satisfaction of needs, improvement of living standards and levels of well‑being must be primarily oriented towards alleviating the poverty of the masses of low‑income people residing in rural areas and making their development self‑sustaining.[7] It is in the rural areas of the developing nations that poverty has assumed the most desperate dimensions, and that, according to World Bank estimates,[8] 80 per cent of the 750 million people live who are considered to exist either in absolute or relative poverty.[9] The most affected groups, the poorest among the rural poor, are small‑scale farmers, sharecroppers, landless workers and their families. A target‑group‑oriented approach to rural development has been advocated with increasing vigour in recent times as a direct and immediate way of reducing, and ultimately eradicating, poverty, hunger, malnutrition, illiteracy, underemployment and unemployment in these groups.[10] The notion of target groups is intended to assist in defining the size, shape and developmental needs of the rural poor. It is also intended to introduce into the programme planning, monitoring and evaluation processes specific considerations about the way in which the poor are benefiting from developmental efforts.

Integrated socio‑economic development programmes, when defined as suggested here, pose two specific monitoring and evaluation problems:

(a) Their multidisciplinary and multisectoral approach to development requires that at certain critical points in the programme implementation process, careful assessments should be made as to whether and to what extent the linkages between the various project components have been established in the intended way;

(b) Since their main purpose is to improve the welfare of described target groups, any meaningful monitoring and evaluation will have to provide an answer to such basic questions as what impact the programmes have on the living conditions of their intended beneficiaries and whether that impact is self-sustaining, i.e., of a relatively lasting nature.

In view of these questions that monitoring and evaluation must answer, it is useful to define "programme result as the anticipated and. intended, as well as unanticipated and unintended, observable changes issuing from the programme in the respective target group or area. These programme results can be classified as follows:

Outputs These are (a) physical outcomes produced by the programme, e.g., number of schools built, kilometres of road constructed, hectares of land irrigated, number of co‑operatives established; and (b) measurement of services provided, e.g., in terms of persons trained, home visit made, credits granted;

Effects These are direct and immediate consequences of the programme outputs, e.g., higher crop yields, increased school attendance rates, changes in levels of information, preferences or skills;

Impact These are changes in the standard of living of the target group or within the target area stemming from the programme. They may take place at two levels: (a) the individual or family level, where they may include changes in income, housing, nutrition or health status, and (b) the community level, where they may comprise changes in the structure of social services – including health care ‑ in the economic system ‑ including production and infrastructure ‑ or in social interrelationships and patterns of communication.

The results produced by one project component may be inputs to other projects of the same programme if, as was pointed out earlier, there exists a means‑end linkage between them. In an integrated development programme, for example, a principal input could be the work of field extension workers. It is often necessary to provide these workers with a preliminary orientation and technical training that is geared to the specific needs of the programme. The results of such a training project, namely, the knowledge or skills imparted to extension workers, will constitute at the same time inputs to subsequent projects undertaken within the framework of the programme. In such a situation, it will be extremely important to evaluate very carefully the results of the training project in order to be sure what the input to the subsequent projects is and whether it meets the requirements laid down in the programme plan.

Within a given programme, two types of factors are important: programme inputs and programme activities. Programme inputs consist of resources and manpower that are provided for the purpose of attaining programme objectives. Programme activities are actions undertaken by the programme staff in order to execute the programme plan. These actions involve the utilization of programme resources, such as machinery, technical instruments, medicine, teaching or training materials.

Programme activities could also be called 'conversion factors" because programme inputs are transformed by them into results. Fertilizer, school books or credits by themselves do not produce any developmental change. They need to be put to use in a specific way, i.e., combined with certain activities. For example, "twelve months of extension work" as an input into an agricultural development programme can produce a variety of results, depending on the method of action adopted by the extension worker. He/she may rely on local leaders as change agents or may organize group discussions and demonstrations for farmers. Different techniques tend to cause different effects. This example illustrates that inputs and activities determine programme results; one is ineffective without the other.

No programme operates in a socio‑economic vacuum. Rather, it operates under certain economic, social and political conditions which it seeks to infIcan1 which influence it. These conditions can be termed "programme environment". Within its environment, the programme will often be just one of several factors causing change. Programme monitoring and evaluation are more complicated when outside factors can be expected to produce the same results as programme activities.

Example Studies of community development have often found that exposure to ordinary mass communications media, such as radio and newspapers, is frequently related to participation in voluntary communal activities. The question for the programme evaluator then becomes whether any increase in community participation was a result of action by community development personnel or merely of routine exposure to mass media. In order to determine which is the real cause of change, information on both contact with community development workers and level of exposure to mass media must be related to a particular activity. For such an analysis, data must be collected on both the programme input (here, community development work) and the other change factor (here, exposure to radio).

It is generally useful to distinguish between those factors of the programme environment which support developmental change and those which impede programme progress. The former, referred to above as "other change factors", can he called "external support factors, the latter 'external constraint factors. To mention just one example, the ethnic homogeneity of the programme target group has often proved to be an external support factor, since it facilitates social interaction and intercommunication, while extreme ethnic heterogeneity is usually a major external constraint factor.

The relationships between the various programme elements and the programme environment, as they have been defined above, are depicted graphically in figure I.

2. Programme monitoring and evaluation

The term evaluation is understood to mean a process by which programme inputs, activities and results are analysed and judged against explicitly stated norms. The norms may be the stated programme objectives, work schedule, budget etc.

Three types of evaluation can be distinguished according to the programme phase or element focused upon:[11]

(a) Ex ante (or pre‑programme evaluation is undertaken before programme implementation. It is usually carried out in order to assess the developmental needs and potentials of the target group or region, test programme hypotheses or determine the feasibility of a planned programme or project;

(b) Ongoing (or concurrent evaluation is undertaken during programme implementation. It analyses the relationship between outputs on the one hand and effects or, to the extent possible, impact, on the other. Its main purpose is twofold: (i) flexible adaptation of the programme to changes either in its environment or in the over‑all development goals, and (ii) early detection of programme deficiencies and immediate design of intervention strategies;

(c) Ex post evaluation is carried out after programme implementation. It assesses programme effects and impact, and aims at obtaining information on (i) the effectiveness of the programme in achieving its stated objectives; (ii) it contribution to the achievement of sectoral or national planning targets an development goals; and (iii) the self‑sustaining character of the changes resulting rom the programme.

The term monitoring usually refers to the process of routine periodic measurement of programme inputs, activities and outputs undertaken during programme implementation. Monitoring is normally concerned with the procurement, delivery and utilization of programme resources, adherence to the work schedule or progress made in the production of outputs. Although primarily concerned with measuring these aspects of programme performance, monitoring also aims at determining the following causal relationships: (a) the relation between programme inputs, activities and outputs, and (b) the influence of external constraint/support factors on programme performance and outputs. The main purpose of monitoring is to indicate as early as possible any short‑comings with regard to delivery of inputs, execution of activities or production of outputs, in order that corrective measures can be undertaken in time. As such, monitoring is primarily a device for improving programme management.

The relationship between the different programme phases and various steps in the monitoring and evaluation process is shown in figure II.

There are usually three categories of information that monitoring and evaluation are expected to provide:

(a) Information on programme performance which would indicate whether programme implementation is proceeding in accordance with the programme plan and budget;

(b) Information on programme effectiveness which would provide answers to such questions as whether and to what extent the programme has achieved its objectives, and on what external conditions it depended;

(c) Information on programme efficiency which would permit determination as to whether programme results were produced in the most economical way, i.e., by maximizing outputs or minimizing inputs.

The main focus of this source book will be the evaluation of effectiveness. This involves such questions as: What impact does the programme have on the living conditions of its intended beneficiaries? How were its results achieved? Are the results of a self‑sustaining nature? How do they compare with the stated objectives of the programme?

B. Concepts

Systematic monitoring and evaluation as presented here combine three basic assumptions:

1. Monitoring and evaluation should be a tool for decision making;

2. Monitoring and evaluation should be a continuous process of problem definition, measurement, analysis and judgement;

3. Monitoring and evaluation should be an integral part of the programme process.

Not all of these assumptions are universally accepted. They will be explained in greater detail, since they underlie the conceptual basis of the following discussion on designing and implementing monitoring and evaluation systems.

1. Monitoring and evaluation as a tool for decision making

The basic premise here is that monitoring and evaluation should be designed in such a way that the information obtained facilitates and supports rational decision making on the programme.[12] In the definitions given above, explicit reference has therefore been made to the way in which the data secured by the various types of monitoring and evaluation could be fed back into the decision-making process.

Decisions on development programmes are taken at different political and administrative levels and in different functional contexts. Some are concerned with policy formulation, others with programme planning and still others with programme management. Decision makers who participate in the programme process in different capacities will each have distinct information requirements. Nevertheless, all information secured must fulfil three general criteria for monitoring and evaluation to serve as a tool for rationalizing decision-making. It must be:

(a) Relevant i.e., geared to the specific needs of decision makers;

(b) Timely, i.e., available and accessible at the time decisions are taken;

Policy makers may, for example, he interested in obtaining reports on the contributions made by various types of programmes to the achievement of national development goals. Such reports would provide them with guidelines on the selection of future development strategies, setting of development priorities and allocation of scarce resources. On the other hand, programme planners, who are primarily concerned with the technical soundness of programmes, might ask for information that would indicate an eventual need for replanning of ongoing programme operations and allow formulation of new programme designs in the light of past experience. Programme managers would most likely have rather comprehensive requirements for information relating to all critical aspects of the programme process.

In addition to being geared to the information requirements of decision makers, the data secured by monitoring and evaluation have to be available at the right time if they are to serve as a basis for decision making. Moreover, the basic purpose of monitoring and evaluation would largely be undermined if the information provided was not accurate. Reliability and validity are two important aspects of data accuracy. Procedures used for data collection and analysis are said to be reliable to the extent that repeated measurements of a single variable lead to similar results. The validity of data depends on whether they really measure what they are expected to measure.[14]

2. Monitoring and evaluation as a continuous process of problem definition measurement analysis and judgement

In order to produce relevant, timely and accurate data, monitoring and evaluation activities should he designed as a repetitive cycle comprising four essential elements:

(a) Problem definition i.e., specification of the topics to be studied and the information to be obtained;

(1) Measurement i.e., collection of data;

(a) Judgement i.e., comparison of findings against programme norms (e.g., objectives) and assessment of their degree of concordance.

On the basis of judgements that are made, decision makers select future actions from alternatives. The implementation of these actions will pose new problems to be studied in subsequent rounds of monitoring and evaluation. Although the topics and content of the various study cycles may differ, their basic structure will in most cases consist of the elements shown in figure III.[15]

Monitoring and evaluation should he designed as a repetitive process because programme results ‑ the assessment of which is of primary concern here ‑ are produced throughout the entire life cycle of a programme. They are influenced by factors and forces that operate in all programme phases. Thus, in order to determine precisely which elements contribute to or impede the progress of a programme, a steady flow of information on its performance and achievements must be obtained. This means that monitoring and evaluation have to be carried out on a continuous basis covering the entire life cycle of the programme.

Of course, continuous monitoring and evaluation of a development programme is an ideal. In actual practice, this ideal will inevitably be modified, especially in the light of the available financial and manpower resources. The data acquisition procedures actually established will usually be close approximations of continuous monitoring and evaluation. For various reasons, however, it may often be preferable to collect the information on a periodic, rather than a continuous, basis.

If the emphasis is on assessing programme effects and impact, the data acquisition system that is designed should minimally plan for (a) execution of a base‑line study; (b) periodic replications of the base‑line study. A simple type of "before‑after" measurement, consisting of a base‑line study followed by a single replication, would not be sufficient. The results obtained from such a measurement would not permit an assessment of which programme inputs and activities were ineffective or which linkages between project components were missing, and why this was so. The available information could not be used for decision-making in connexion with programme management. Consequently, many advantages of monitoring and evaluation would be lost and the reason for having a data acquisition system at all would be subject to question.[16]

Base‑line studies, the first step in implementing monitoring and evaluation, are undertaken either before or immediately after the commencement of programme implementation. They are meant to serve two main purposes: diagnosis of the existing situation for programme planning purposes; establishment of a bench‑mark against which to measure future changes in the living conditions of the target group or in the target area.

Data on changes deriving from the programme will be collected through the replication of the base‑line study at later points in time. The replication needs to be done very carefully so that differences between the original and new data can be interpreted to indicate programme effects or impact.

The number of replications or restudies undertaken will depend mainly upon three factors: (a) the nature of the programme process; (b) the information requirements of the decision makers; (c) the resources available. These factors also determine the number of and intervals between the routine measurements for monitoring purposes.

3. Monitoring and evaluation as an integral part of the programme process

To be a successful and effective tool for decision making by providing a continuous flow of relevant data, a monitoring and evaluation system must first be designed and established as an integral part of the programme process.[17] There are three main reasons for this.

(a) The salient feature of evaluation that distinguishes it from other research processes is that facts (e.g., about programme performance or results) are judged against explicitly stated norms. Thus) successful evaluation will only be possible if the programme plan is formulated in a precise and detailed way, facilitating a clear determination of the programme objectives and procedures needed to achieve them. In designing the programme plan, consideration must therefore be given to the way in which the programme should he evaluated;

(b) In order to ensure continuity and comprehensiveness of data acquisition) adequate resource provisions have to he made in the programme budget;

(a) In order to facilitate a prompt feedback of results to programme managers, responsibility for monitoring and ongoing evaluation should be assigned to the programme staff, since they are the principal users of the information during the programme implementation.

This last point is a rather controversial one in the literature on programme monitoring and evaluation. It is sometimes maintained that evaluators should be emotionally detached from the programme and consequently that studies on programme performance and achievement are best performed by outside experts. The various reasons for one or the other strategy are discussed in greater detail in chapter II, section C. It is emphasized that the two approaches are not mutually exclusive but, rather) complementary to one another since each serves a different purpose.

[1] For definitions of these two terms and those introduced below, see United States of America Agency for International Development, Project Evaluation Guidelines, 3rd ed. (Washington, D,C., l97b); and World Health Organization, Evaluation of Environmental Health Programmes Report of a WHO Scientific Group, Technical Report Series No. 526 (Geneva, l973)

[2] An extensive discussion of the concept of an integrated or, as it is also called, unified approach to development analysis and planning is to be found in "'Report on a unified approach to development analysis and planning; note by the Secretary‑General" (E/CN.5/519).

[3] Bengt Nekby, ed., CADU An Ethiopian Experiment in Developing Peasant Farming (Stockholm, Prisma Publishers, 1971).

[5] Food and Agriculture Organization of the United Nations, Review of Field Programmes 1971‑1975 (Rome, August 1975), pp. ll6-117.

[7] For a more detailed discussion of this issue, see Uma Lele, The Design of Rural Development Lessons from Africa (Baltimore and London, Johns Hopkins University Press, 1975), pp. 12l.

[8] World Bank, Rural Development Sector Policy Paper (Washington, D.C., February 1975), p. 4.

[9] A widely accepted definition of absolute and relative poverty is that suggested by the World Bank. As a criterion for identifying absolute poverty, the Bank proposes an annual per capita income equivalent to $50 or less. The standard set for determining relative poverty is an income above $50 but less than one third of the national average. See World Bank, op cit. pp. 4 and 17.

[10] See, for example, International Labour Office, Employment, Growth and Basic‑Needs; A One‑world Problem (Geneva, 1976); Jan Tinbergen, co‑ordinator, Reshaping the International Order A Report to the Club of Rome (New York, E. P• Dutton 1916);, "Basic services for children in developing countries (E/ICEF/L.1342).

[11] Similar definitions are proposed in World Bank, Technical Workshop on Monitoring and Evaluation of Rural Development Projects and Programs Copenhagen, Denmark, 6‑10 December l976 SummaryReport (Washington, D.C., 1977), p.2.

[12] A decision‑making approach to monitoring and evaluation is presented in the following studies: Michael C. Latham, Planning and Evaluation of Applied Nutrition Programmes (Rome, Food and Agriculture Organization of the United Nations, 1972); B. J. Mardel, "Building a better MIS", Financial Executive, August 1970, pp. 65‑82; Herbert 0. Turner, "Principles and methods of program evaluation”, Focus Technical Co‑operation No. 3, 1976, pp. 26‑30; Carol H. Weiss, Evaluation Research Methods for Assessing Program Effectiveness (Englewood Cliffs, New Jersey, Prentice‑Hall, 1972); World Health Organization, op cit.

[13] These three requirements of a successful monitoring and evaluation system are strongly emphasized by B, J. Mandel, loc. cit, pp. 66‑68. A fourth requirement mentioned by Mandel, namely, economy, is discussed in a later context; see chap. II, sect. A.

[14] See Claire Selltiz, Lawrence S. Wrightsman and Stuart W. Cook, Research Methods in Social Relations (New York, Holt, Rinehart and Winston, 1976). chap. VI.

[15] A similar model depicting the evaluation mechanism in relation to the system being evaluated is set forth in World Health Organization, op cit. p. 13.

[16] This point is especially stressed by Michael C. Latham, op cit, p. 71.

[17] See, in this connexion Herbert D. Turner, loc, cit., pp. 26‑27; and World Bank, Technical Workshop on Monitoring and Evaluation... pp. 4-5.

PART ONE. A SYSTEMATIC APPROACH TO PROGRAMME MONITORING AND EVALUATION

Chapter I DEFINITIONS AND CONCEPTS OF SYSTEMATIC MONITORING AND EVALUATION