Chapter II
DESIGNING A MONITORING AND EVALUATION SYSTEM
This chapter will formulate guidelines for the
design of monitoring and evaluation procedures which are decision‑oriented,
systematic and focused on controlling and assessing programme outcomes. The
guidelines set forth are based on theoretical considerations and practical
experience.
Establishment of a monitoring and evaluation
system entails two major activities: (a) formulation of a workable system
design, and (b) planning operational and resource implications of the system.
The steps involved in this process consist in finding clear and detailed
answers to the following series of questions:
What are the norms against which to compare
monitoring and evaluation results with regard to programme performance and
outcomes?;
What information has to be obtained in order to
make decisions on the programme?;
How is the information to be collected? By which methods? Where? When?;
How should the data be analysed?;
To whom should the monitoring and evaluation findings be reported?;
Who will be responsible for carrying out monitoring and evaluation?;
How much should be invested in monitoring and evaluation?
The main issues to be considered when answering this list of questions
are elaborated in section C below. Preparatory to this, the principles that
should govern the deliberations about and selection of procedural alternatives
at each step of the designing process are set out in section A. Section B
examines a step to be taken before the actual designing process is begun. This
step relates to the problem of how to identify the decision makers who are to
use the monitoring and evaluation findings and how to co‑operate with
them in setting up the dataacquisition system.
A. Principles of design
The monitoring and. evaluation methods
conceptualized here are expected to respond to four imperatives. The following
three have already been discussed in chapter I:
(a) Monitoring and evaluation should be strictly
decision‑oriented;
(b) Monitoring and evaluation should be systematic;
(c) Monitoring and evaluation should be an integral part of the programme
process.
To these a fourth should be added:
(d) The costs of monitoring and evaluation should be as low as possible.
Since monitoring and evaluation are to assist
decision‑making on the programme and are thus intended to improve
programme operations, they should not divert programme resources to the extent
that operational activities are impaired. While there are no universal rules
for establishing cost limits for monitoring and evaluation, a rule of thumb
alight be that the cost should not exceed 1 per cent of the total programme
budget. In general, the resources required for monitoring and evaluation can be
reduced by keeping the amount of data to be collected at a minimum; and
applying the most efficient procedures possible for data collection and
analysis.
In addition, if the data secured can be, used for other purposes besides
monitoring and evaluation, the relative cost is decreased. If, for example,
monitoring and evaluation are integrated with staff training, or evaluation
studies are also used for action research, their cost could be attributed to
several purposes and thus be more easily justified.
Given these four imperatives, the following
principles should be observed in forming the basic shape of the monitoring and
evaluation system, in order that it may be capable of fulfilling the objectives
and functions expected of it. The system should be:
(a) Formulated in close consultation with its
potential users to ensure that their information requirements receive due
consideration;
(b) Designed as early as possible so that it
can be properly incorporated into the programme process‑;
(c) Designed as simply as possible by
collecting only the most necessary information and using the most
straightforward methods in order to reduce costs to a minimum;
(d.) Formulated in a methodical way by
considering all the steps involved in the designing process and taking them in
a logical sequence. This will determine the systematic character of monitoring
and evaluation.
B. Identification
and involvement of potential users
Involving potential users in the design of the
monitoring and evaluation system will not only help to clarify their
informational requirements, as pointed out earlier, but may also help to assure
support for the system and utilization of its findings.
Agreement among designers and users on the
basic approach to be adopted will considerably decrease the possibility of
later opposition to the data‑acquisition activities.[1] Such opposition may assume the form of either
political or professional criticism on the relevance, accuracy and usefulness
of monitoring and evaluation. If connected with a curtailment of resources, it
will hamper, if not interrupt, the planned continuity of data collection and
thus seriously distort the whole process of information gathering.
Besides strengthening support for monitoring
and evaluation, the broad participation of all decision makers concerned may
also enhance the utilization of findings. The fact that decision makers have
been involved in the design process will tend to increase their commitment to
respond to the information reported to them. They will be more apt to
reconsider and, if necessary, revise their decisions and actions in the light
of the conclusions drawn from the evaluation studies.[2]
For all these reasons, it is useful to
identify, as early as possible, all the policy makers, administrators and
programme staff members who are to take decisions on the programme and to
invite them to participate actively in designing the monitoring and evaluation
procedures. One way of institutionalizing their participation would be to
establish a working group composed of representatives of decision makers and to
consult this group on each major step the system designers intend to take.
In collecting data on programme effects and
impact through interviews with members of a specific target group, it has
proven especially important to consult also with their local leaders. This has
proven useful for two reasons. First, local leaders are often the best resource
persons for finding out how members of the target group generally view the
development problems that the programme is expected to resolve. Understanding
their way of looking at these problems is relevant not only to the formulation
of monitoring and evaluation questionnaires but also to programme planning.
Secondly, local leaders cannot be expected to generate and secure the co‑operation
of the people in the data‑acquisition process unless they understand the
rationale for gathering data. In explaining to the people and their local
leaders the purposes and advantages of monitoring and evaluation, emphasis
should be placed on making them aware of the fact that controlling and
assessing programme outcomes can be useful for them also. For example, it
should be pointed out that monitoring and evaluation results can demonstrate to
them whether and to what extent the changes introduced by the programme are
really advantageous to them, what the programme benefits are and to whom the
benefits actually accrue.[3]
C. Steps involved in the design process
The following discussion of the steps involved
in setting up the monitoring and evaluation system illustrates further the
importance of close consultation between designers and users. The design
procedures proposed have been developed in accordance with the principles set
forth in section A of this chapter.
Step 1 Preparing a logical programme
framework [4]
Before any opinion can be formed on how a
programme should be monitored and evaluated, it is essential to know what the
programme is intended to do and how it is expected to operate. A careful
description of the programme, its objectives and work plan, must therefore be
the first step in designing the procedures for programme monitoring and
evaluation. If this description is made methodically, it will result in a
logical programme framework. Preparing such a framework of the programme design
requires undertaking three main tasks:
(a) Definition of programme objectives in measurable terms;
(b) Explication of the premises underlying the programme plan of
operation;
(c) Selection of indicators of programme inputs, activities and
outcomes.
As far as the definition of objectives is concerned,
it must be stressed that there is no requirement that all programme and project
objectives should be expressed quantitatively. Rather, the programme purposes
may be stated in terms for which either quantitative or qualitative measures
(indicators) could be established. It is essential, however, that in either
case clear and unambiguous formulations are made which do not allow any
uncertainties to exist about the programme purpose. In order to make the
achievement of stated objectives susceptible to periodic measurement and
progress control, it is suggested that the following questions be clarified
when defining programme objectives:
(a) Definition of the content of the
objectives: What is the programme intended to do?;
(b) Definition of the target group or region: Who will benefit?;
(C) Definition of benefits that will be
foregone, by whom and where: Who will pay?;
(d) Definition of the time‑frame: When
are benefits and costs likely to occur?;
After the programme objectives have been specified, the next task in the
preparation of a logical framework is the explication of the assumptions on
which the programme plan is based. Hypotheses shoul be formulated on three aspects of the programme
process:
(a) The functioning of the projects comprising
the programme: How are project inputs/activities to be converted into results?;
(b) The interlinkages between the various
project components: How are the objectives of each component supposed to relate
to one another and to the over‑all programme purpose(s)?;
(c) The influence that external
constraint/support factors exercise: How are external factors supposed to
affect the programme process?
Since programmes are usually designed within
the broader context of a development programme, evaluating their effectiveness
only in terms of what progress they have made towards achieving their own
stated objectives would not be sufficient. In addition, it is necessary to
assess what they have contributed to the attainment of the broader
developmental goals and how they compare in this respect with other programmes,
especially those following alternative courses of action. One of the programme
hypotheses. should therefore deal with:
() The relation between programme objectives
and plan goals: How does the programme intend to further the attainment of the
over‑all goals of development?
The greater the specificity with which
programme objectives and hypotheses are stated, the less complicated it will be
to establish programme indicators. Indicators are objectively verifiable
measures of facts and events. They may be either quantitative or qualitative.
Where the focus of,monitoring and evaluation is on control and assessment of
programme performance and outcomes., indicators may be usefully defined by
answering such questions as: What facts or events will indicate that the
programme is progressing according to plan? What changes in the programme
environment will show that programme purposes are achieved?
Sometimes, however, it will be impossible to
observe and measure programme results directly. In such cases, indirect or
proxy indicators must be found. These are objectively verifiable facts or
events which can be taken to indicate the existence or non‑existence of
the phenomenon to be studied.
Example: Since mortality rates are
difficult to determine over short time periods, they are usually poor
indicators of effectiveness of health programmes, even though the programme
objective is reduction of infant mortality. A proxy indicator might be the
percentage of births which are attended by trained health personnel, and the
frequency and type of (preventive or curative) health facilities used. This
asses that the usually observed negative relation between use of health
services and infant mortalitywill continue.
When proxy measures are used, it is important to ensure that the
hypothesized linkage between the non‑directly and directly observable
variable has been verified by previous research.[5] The same holds true for those indicators
which, while relating to a single crucial characteristic of the phenomenon to
be measured, are supposed to present an adequate picture of it as a whole.
The important advantage of using key indicators
is that less data have to be obtained. There is also a disadvantage, namely, a
decrease in accuracy. The information provided will be more reliable if data
pertaining to several indicators of the same phenomenon are collected at the
same time. However, the use of such multiple indicators will considerably
increase the number of variables to be studied. There exists no genera]. rule
for choosing between use of key indicators with less accuracy and use of
multiple indicators with greater accuracy. This choice must be made‑for‑each
programme individually, in view of the specific conditions under which it will
have to function. Two considerations are of special relevance in this
connexion:
(a) The information requirements of the users:
The use that will be made of the monitoring and evaluation findings could be
one determinant of the required degree of data accuracy;
(b) The experimental character of the
programme: A programme that uses innovative techniques will require more
thorough monitoring and evaluation than one that moves along already proven
lines.
Since the objectives pursued and the strategies adopted vary from
programme to programme, it is not possible to propose any catalogue of
"best" indicators for measuring programme performance and outcomes.
What can be done, however, is to generalize some of the formal requirements
that must be fulfilled if the selected indicators are to be useful.[6]
(a) The indicators established for measuring
the outcomes of integrated socio‑economic development programmes must be
sensitive to development on the local level, since the primary objective of
these programmes is to improve the living standards of a specific target group
or area;
(b) The indicators must vary with the programme
aspects they are supposed to measure, i.e., with progress made in programme
implementation or attainment of objectives;
(c) They must be objective, i.e., directly observable and measurable;
(d) They should be simple, i.e., the data on
them should be readily available.
In the process of preparing a logical programme
framework, one may find it convenient to use the matrices shown in figures IV
and V as a guideline. Placing the required information into each matrix cell
will help to check whether the tasks involved in the process have been
completed properly and whether all relevant questions have been answered: Are
all programme objectives defined? All assumptions explicated? All indicators
selected?
From the foregoing discussion, it is obvious
that the logical programme framework is not only a crucial instrument for
monitoring and evaluation but also an important programme planning device. This
underlines once more that the design of the monitoring and evaluation system
should be an integral part of the programme planning process.
Step 2 Specifying information requirements
what to measure
An integrated socio‑economic development
programme deals with a potentially infinite number of variables that can be
measured in order to analyse whether or not it is functioning properly and why.
Collection of these data could easily become the most expensive programme operation.
For reasons of cost as well as of keeping the monitoring and evaluation system
to a manageable size, great caution must be exercised when determining the
topics to be studied. As a general rule, only those data should be collected
which are absolutely necessary for rational decisionmaking on the
programme.
Delineation of the crucial topics of monitoring
and evaluation requires that, on the basis of the logical programme framework,
three questions be clarified in consultation with the appropriate decision
makers:
(a) What exactly do the decision makers want to know about the
programme?;
(b) For what purpose do they want that information?;
(c) When do they need it?
For reviewing and screening the requests put
forward in response to these questions, a matrix can be designed according to
the format shown in figure VI. The matrix has several uses:
(a) It will reveal very clearly whether and to
what extent the data needs of the various users are distinct or overlapping.
Two factors have to be checked in this connexion: (1) the content of the
information, and (ii) the date by which it should be available. The more
coincidence and conformity there are, the fewer the data that will have to be
collected and, consequently (as will be discussed further in step 6), the fewer
the monitoring and evaluation reports to be prepared;
(b) It will also show whether, for all the
information requested, there is a predefined practical use;
(c) Categorizing the information according to
content and purpose will permit system designers to decide on the type of
studies to be undertaken: ax ante evaluation, monitoring, ongoing and/or
ex post evaluation.
In discussing information requirements with
decision makers, they must be given at least a preliminary indication of the
proposed time‑frame of the system.
This implies answering a number of questions: What is the planned length
of the intervals between the periodic routine measurements for monitoring
purposes? What is the maximum possible number of replications of the base‑line
study? The time frame, which depends in its structure on both the manpower
and financial resources available, will be a decisive determinant of the users'
information needs. The greater the number of monitoring and. evaluation studies
envisaged, the more attention will probably be paid to the detailed working
of the programme process; the fewer the studies, the more oriented.
to results and. to broader concerns
will be the questions suggested for research.
Appropriate indicators must be found. for each
information element that is recognized as being of crucial importance for
decision‑making. Programme inputs, activities and objectives require
a simple transcription of the respective measures from the logical programme
framework matrices to the information matrix. When the variables that make
up each of the chosen indicators are entered. into the last column of the
information matrix, a master list of all the data to be studied
in the various monitoring and evaluation rounds will be obtained..
This list constitutes the informational blueprint of the system.
The master list should consist of two parts:
(a) a rather fixed core of basic data relating, for example, to the socio‑economic
conditions of the target group, which will be restudied periodically in order
to assess change deriving from the programme; and (b) a flexible and open‑ended
part. Since it is conceivable that the purposes of the programme may be
modified in the course of its life cycle or that changes in its environment
occur or unforeseeable problems arise in connexion with its implementation,
adequate provision has to be made for a possible redefinition of the topics to
be studied and the variables to be measured. It thus follows that the
substantive content of the monitoring and evaluation system, as well as all its
other aspects, must be planned with a certain degree of flexibility in order to
ensure adaptability, continued relevance and usefulness.
Step 3 Identifying the sources of
information where to measure data
The next step is to discuss with system
designers and other knowledgeable persons the variables on the master list one
by one, in order to determine where each of them could be observed and
measured. The main purpose of this exercise is to reduce the amount of data to
be collected from primary sources, i.e., through specially organized research
activities, by identifying secondary sources of information that already exist and
are easily accessible.
Step 3.1 Secondary sources of information
Before it is agreed to rely on certain
secondary sources, the usefulness of the data they contain must be carefully
checked.. Three consideration are important here:
(a) How current are the data?;
(b) How accurate are they?;
(c) Are the categories into which the data are grouped., e.g., the
category of "worke?' or "household", comparable to those used in
programme monitoring and. evaluation?
Possible secondary sources that exist in nearly every country and should
be
thoroughly investigated include the following:[7]
Census reports;
Statistical reports of the sectoral ministries
and their respective offices at various levels of the governmental bureaucracy;
Reports of the national, regional and subregional planning authorities;
Statistical reports of the district or block development offices;
Monitoring and evaluation reports on similar
development programmes conducted previously;
Social science research reports relating to
developmental problems with which the programme is dealing.
Especially in instances where it is desirable to observe certain
development trends over a longer period of time or to compare local‑level
data to the corresponding figures for the regional or national level, it will
be very useful to have at hand secondary data that cover a broad area and a
number of years.
Step 3.2 Units of research
Because, in practice, secondary data sources
will not cover all the required variables, the proper target group or the appropriate
time frame, the verification of some programme aspects will always require
special data collection efforts. In some cases this can be a rather simple task
requiring only a few modifications in the normal administrative recording and
accounting procedures. However, in other cases, data collection may pose
complex and difficult problems. In the first place, it will involve a choice
with respect to the unit of research: Should the main unit be the individual,
the household, or a locality such as a village or neighbourhood?
The choice of unit depends mainly on two factors:
(a) what the programme is expected to do, i.e., the programme
objectives.
Example: Is the programme expected to bring water to a central point
in the village or to each village household?
(b) How the programme is intended to operate, i.e.,
the functioning of the programme process.
Example: A programme aiming at improving the literacy rate in a
certain rural area may not work with the villagers directly, but may operate on
the district level instead. The district office for community development may
be provided with the necessary financial and material resources for organizing
the construction of school buildings on a selfhelp basis, while the
development of appropriate curricula and the acquisition of text‑books
may be arranged through the education office.
If programme results alone were to be measured
for the purpose of monitoring and evaluation, then a sample survey of
households or individuals would probably be adequate in most cases. However,
measurements of resource delivery and utilization or more generally of progress
in programme implementation are also necessary. These measurements sometimes
become complicated owing to the fact that programmes usually operate through larger
aggregates, such as neighbourhood, village or district, or through
organizations, such as co‑operatives or farmers' or women's
organizations. In this event, resource delivery and programme activities
sometimes,have to be studied at one level, and the results at another. A rule
of thumb for determining in what ways programme operations can be related to
programme outcomes is to ask the following question: What is the lowest
politico‑administrative unit that includes both the level of programme
operations and the level at which programme results become effective?
Examples: A
health care centre may be serving several villages which constitute a development block. The
block would in this case be the lowest unit to be studied;
In another programme, fertilizer and seeds are distributed through the
village co‑operative. The village would then be the appropriate unit to
be selected.
Whatever the nature of a development programme,
it is important to define very carefully the lowest unit at which changes
deriving from the programme can be directly related to the functioning of the
programme operations, in terms of programme resources and activities. For
consistency, this lowest unit will be termed the "community level" in
the following discussion. In cases where the programme aims at changing
community conditions, e.g., the infrastructure and communication facilities of
a village, the community will be the appropriate research unit for studying
both programme operations and outcomes. If, however, the programme beneficiaries
are individuals, households or specific social groups, for example, female
agricultural labourers, a second choice becomes necessary. Subunits within the
community, the main research unit, must be defined. What the right subunit is
depends primarily on the identity of the intended target group, as was
emphasized earlier. The question to be asked is: Who should be benefiting from
the programme?
Step 1 Deciding on the research design how
to collect and analyse data
The issues relating to this step need to be determined only in principle
at the time of designing the monitoring and evaluation system. Their details
may be worked out later, when the system is implemented. Nevertheless, three
crucial questions must be clarified at this time:
(a) How should progress in achieving stated programme objectives be
measured?
(b) Would complex enumeration or sample selection be preferable?;
(c) What should be the principal methods for measuring and analysing
data?
Step 4.1. Measuring change
Measuring progress in achieving stated
objectives means measuring changes deriving from the programme in its
environment. More precisely, it means assessing improvement in the living
conditions of the target group or region.
In order to analyse programme effects and impact
some kind of comparison is needed. There are three possible approaches:
(a) Comparison of observed conditions against
set targets in connexion with either the programme's stated objectives or the
over‑all goals of development;
(b) Comparison between the programme area and
other areas with a similar socio‑economic structure;
(c) Comparison based on change over time in the
programme area. Each approach has certain advantages and disadvantages. In
practice, all three are used, but the method adopted most often is that of
comparison based on change over time in the same place.
Methodologically, evaluation against targets is rather simple. It is
merely a matter of placing an observed value against a set standard in order to
find out how close the two have come to each other. In the ideal case, planning
targets should be empirically determined so as to avoid the problem of either
overly ambitious or overly conservative targets. However, this is not always
possible, especially during a programme's initial stages, as the necessary
empirical data may not be available then. Indeed, even where data are
available, realistic targets may be politically unpalatable.
Example: In
one country,‑an integrated rural development programme set a
four‑year family
income target at $12,000 per year, even though a careful
study of rural income in agrarian reform areas
had shown average family income
to be only $3,550, with the median even lower
at $2,205. Achieving the target
in four years, even under the best of
circumstances, was next to impossible.
In contrast, another country set a family
income target for a resettlement
zone at the level of average family income in
the original areas from which
settlers came, a target very easy to reach.
Thus, while comparison of observed results against targets can be a
simple and useful. operation, it would not be advisable to base programme
evaluation solely on this type of analysis.
Evaluation based on comparisons among different
regions or localities, while also rather simple, has similar disadvantages. For
example, comparisons can be made between communities where a programme has
worked and where it has not worked, and it can be concluded that differences
between the two types of communities represent the impact of programme
operations. While these differences may indeed represent programme effects, it
would also be conceivable, especially if measurements have been made only at
one point in time, that they were caused by other change factors. They could
probably also be attributed to factors on the basis of which the programme area
was defined.
Example: PRODESCH, a socio‑economic development programme
in the Chiapas Highlands of Mexico,[8] had by 1973 worked in about 300 of the 600
villages in the highlands area. In theory, it would be possible to estimate the
impact of the first two years of PRODESCH by comparing the 300 worked villages
(or a sample thereof) with the 300 non‑worked villages. However, the 300
worked villages tended to be close to major population centres, larger and more
involved in the stateÕs economy, while the 300 non‑worked villages tended
to be relatively inaccessible, smaller and. less involved in the monetary
economy. As a result, many differences between the two types of villages
probably would have existed even without the PRODESCH inputs, and an evaluation
of the programme based solely on this type of comparison would have been
misleading.
Thus, while
comparisons over space can be helpful, exclusive use of this technique for
evaluation is not recommended unless base‑line data exist on the pre‑programme
conditions of the people or criteria are established regarding the
comparability of the two types of communities ‑ worked and non‑worked
‑ included in the evaluation study.
The third method of
comparison is similar to a "before‑after" design. It is more
accurate than the two approaches described above, since analysis is based on
Tepeated measurements of change over time within the programme area. Problems
due to selection factors or unrealistic targeting are thus reduced or
eliminated. This method is also the most appropriate for continuous monitoring
and evaluation, since it allows for periodic data gathering.
The disadvantage of
this method is that the socio‑economic development observed in the
communities during a certain period of time may be the result of the over‑all
socio‑economic development of a larger geopolitical region in which the
communities are located. To minimize this short‑coming, special analysis
techniques may be designed for internal, or interunit, comparisons. To control
for over‑all socio‑economic development, they would utilize data
related to the degree and intensity o the various activities of the programme
in relation to the observed results.
Weighing all
advantages and. disadvantages carefully, one may conclude that the most
reliable procedure for measuring programme effects and impart would be a
combination of the second and third methods: an analysis of both worked and
control communities involving measurements at different points in time, one to
establish base‑line data and others to indicate the progress made in
programme implementation and achievement.
The term "case study" is used here in
a very precise manner. In contrast to its usage in social welfare practice, in
which cases are chosen according to predetermined criteria, the cases studied
for monitoring and evaluation purposes must be randomly selected. The total
population from which they
are to be drawn consists of two subpopulations or strata, namely, all the
communities that lie either within the programme area, or, according to certain
explicit specifications, within the delineated control area. In each case
selected, data are obtained from a variety of sources: a random sample of
individuals or households, interviews with community leaders and systematic
observation of social, political and economic activities.
An individual case study cannot describe the
pattern of results for the entire programme area. Thus, the appropriate method
is a sample of case studies representative of all communities or other politico‑administrative
units participating in the programme. For this reason, data collection should
be based upon a network of case studies, randomly selected.[9]
Sample selection is proposed because in most development programmes the number of
communities to be covered is far too large to allow investigations of each
individual case. Although there is definitely great merit to in‑depth
studies, based on complete enumerations, disadvantages by far outweigh
advantages when they are undertaken in connexion with programme monitoring and
evaluation. Owing to the enormous amount of data collection and analysis
involved, in‑depth studies are usually quite expensive and bound to
produce results with a considerable time‑lag. Thus, their usefulness for
monitoring and evaluation is rather limited.
To ensure that the cases selected are truly
representative of over‑all conditions, the sample must be random.
Representativeness of a sample also depends on its absolute size: the larger
the sample, the more it will be representative, provided that an unbiased
sampling procedure is used.
The question then becomes "how large a sample?". Two
considerations determine the size of the sample of case studies for systematic
monitoring and evaluation:
(a) the method of sampling, and (b) the number of resources available for data
collection and analysis. In reality, the number of resources available is the
more important determinant. Initially, this implies determining how many people
are available for how long to collect and analyse data. For example, suppose a
development programme includes 500 villages, and each case study, as experience
has shown, requires one week by a team of five to collect the data, plus
another week to tabulate and pre‑analyse them. If a total of 100 persons
are available for two weeks each to participate in the conduct of case studies,
it is clear that no more than 20 cases can be studied.
Step 4.3 Data collection
The more commonly used techniques of data collection include:
In‑depth interviews;
Standardized questionnaire interviews;
Direct observation;
Participant observation;
Group discussion;
Record keeping by the respondents themselves;
Physical measurements.
Which of these methods proves to be the most appropriate in a given
research context depends mainly on two factors: (a) the research unit(s) and
(b) the research topics.
A programme concerned with institution‑building
for local‑level development illustrates the first point. The question to
be studied is whether and to what extent local leaders are willing to support
an expansion of the co‑operative movement and a strengthening of rural
workers' organizations. The appropriate method for obtaining data on their
respective views and attitudes would be the in‑depth interview. The
interviewer's role here is simply to focus the leaders' attention on the
various crucial issues involved and to encourage them to elaborate on these
issues as extensively as possible. The order in which the topics are covered
and the manner in which they are dealt with is, in this type of interview,
largely at the discretion of the respondents.[10] Thus, the interview takes on the character of
a free‑flowing conversation, allowing the respondents to express their
views and concerns in the way they wish to. In the present case, given the role
of local leaders and their expectations of the role of their discussion
partners, this would be a better method than confronting them with a rigid,
standardized questionanswer schedule.
The relationship between research topic and
method i illustrated by the method used in obtaining data on the views of
farmers about the usage of profits earned by co‑operatives to which they
belong. The most adequate research method for investigating the issue is the
group discussion. The reason is that under normal circumstances the farmers
will always discuss this specific problem in a group context, usually at a
general meeting of their co‑operative. Therefore, to question farmers
individually, for example, in their homes, about their opinion on the
utilization of co‑operative profits would be putting them into an
unrealistic and. artificial situation. Unfamiliarity with the research
situation would definitely be reflected in the farmers' responses and. thus
cause a distortion of the research results.
The above examples demonstrate a general rule to be followed when
selecting the research method: The method should always enable the research
situation to resemble as closely as possible the everyday life context in which
people encounter and. discuss the problems on which data are to be gathered.
This rule also applies to the formulation of
interview schedules. Its implications with regard to the questionnaire design
are threefold:[11]
(a) The questions should be formulated in such
a way that they can be easily understood by the target group. Each question
should usually contain a single easily comprehensible idea;
(b) The questions should be worded in the local
language or dialect, using typical idioms and phrases of the target group;
(C) In designing the question‑answer
sequence, especially when including fixed‑alternative (closed) questions,
proper attention should be paid to the prevalent social norms of
intercommunication. In many countries, for example, a rigid
"yes/undecided/no" schedule may be socially unacceptable.[12]
After an examination of the various possible
research methods in the light of the above‑mentioned considerations has
revealed several of them to be appropriate, the most straightforward one should
be selected. This will usually be the one which is the easiest to apply,
involving a minimum amount of data collection, and thus the fastest and most
economical method.
Step 14.4 Methods of data analysis
Both data collection and analysis need to be
accomplished as quickly and efficiently as possible, since for the purpose of
decision making there is as little value in outdated as in incomplete data. In
addition, prompt analysis of the information collected reduces the amount of
staff time that must be devoted to the tasks of monitoring and evaluation.
In analysing the data obtained, two extremes must be avoided. On the one
hand, data analysis can be so general that the conclusions arrived at become
rather weak. The data are then said to be under‑analysed. On the other
hand, data can be analysed so fully that results become trivial, in the sense
that unimportant findings are given equal standing with important ones. In that
case, data are over‑analysed. The solution is to strike a balance between
these two extremes, based on the quality of the data and the information needs
of the decision makers. If this is done, data analysis can be straightforward
and useful. It should be kept in mind that certain types of data, especially
those which measure qualitative aspects of development, are less precise; other
types, such as economic data, are usually more precise. One should not over‑analyse
less precise data nor underanalyse those which are precise. A good rule is
that if a. result is significant, it should be observable in the data, without
requiring extensive use of sophisticated statistics and other types of
analytical procedures.
The process of
analysis of socio‑economic data consists in the reduction of raw data to
empirically supported conclusions. In order that this process may take place
expeditiously, care must be taken to ensure that analytical procedures are kept
simple. Two main bodies of analytical procedures are to be distinguished: (a)
"descriptive statistics", and (b) "explanatory statistics".
The first analyses data on a variable‑by‑variable basis, the second
examines interrelationships among variables.
By looking at each
variable individually and describing its characteristics, it should be possible
to draw some preliminary, tentative conclusions. In a monitoring and evaluation
system where base‑line data are acquired prior to the initiation of the
programme, such conclusions could help programme planners to determine more
precisely the nature of the developmental problem that the programme is
intended to resolve and to formulate more realistic programme objectives. In
cases where base‑line data are collected only after the programme has
been under way for some time (for example, where monitoring and evaluation have
just been introduced into a longstanding programme), the descriptive analysis
can also include preliminary judgements of programme effectiveness by comparing
base‑line results with previous targets. Similarly, comparisons between
communities that have attained "highest" and "lowest"
results for each variable can indicate the degree of relative effectiveness of
the programme.
The main tools of
descriptive analysis are measures of central tendency, such as means, medians,
quartiles and proportions, calculated for each community. The results obtained
will indicate directions for explanatory analysis.
In explanatory data
analysis, the main task is to identify relationships among variables. The most
important relationships to be tested are those listed in the logical programme
framework matrices. Thus, the analysis has to ask whether a statistically
significant relationship exists between:
Project inputs/activities and achieved results;
Various project objectives;
Project outcomes and over‑all programme
results;
Programme results and achievement of national/regional
goals of development;
Certain external constraint/ support factors
and the observed programme results.
An explanatory
analysis may be conducted by utilizing a number of methods of varying
sophistication to establish these relationships. The method to be used will
depend on the quality of the data, the level of precision obtained and the time
available for analysis. All methods have several principles in common. The
basic principle for establishing relationships is that of association.
If a variable occurs in the same way and place as another variable, the
two may be said to be positively related. For example, if a].]. communities
with higher than average incomes have higher than average community
participation, and all communities with lower than average income have lower
than average participation, the two variables can be said to be related.
Similarly, if the value of one variable is always higher when the value of
another is lower, the two can be said to be negatively related.
A second principle in
explanatory analysis is that association does not in itself indicate cause and
effect. Cause and effect can only be determined by relationships over time or
on the basis of logic. Suppose, for example, that two exactly similar
communities had. the same levels of income in 1968. In 1969 an agricultural
extension programme was begun in one, with all other programmes remaining the
same. If in 1970 the income of the community with agricultural extension was
double that of the other, it could be stated, with reasonable assurance, that
agricultural extension caused an increase in income.
A third principle is
that since several variables could be related to each other, in varying degrees
of strength, explanatory analysis should attempt to identify those which are
most significantly associated with one another. This is essential for drawing
useful policy conclusions. If, for example, in a number of communities it is
found that a dependent variable A, e.g., "economic growth", is
related to two independent variables B and C, e.g., "strength of co‑operatives"
and "women's participation in the labour force", and that in all the
communities studied the association between A and C is stronger than that
between A and B, it would follow that, in order to speed up economic growth,
prime importance has to be placed on encouraging women to take up employment
outside their homes.
The first stage in any
analysis process is the tabulation of raw data. Without proper procedures this
can be a very time‑consuming process, whether tabulation is done manually
or by computer. The problem particularly centres on tabulation of individual
interviews, since these are the most numerous sources of data. For example, in
a system with 20 case studies, and with 25 interviews per case study, there
would be 500 individual questionnaires. If each questionnaire had 75 variables,
there would be 37,500 individual pieces of data to be dealt with.
A procedure which has
been found appropriate in a number of countries is that of assigning the task
of tabulating individual questionnaire data to the field investigators who
collected them. First, individual computations are made by the interviewers on
the questionnaire itself. Then, since the main research unit is the community,
the individual responses are tabulated on a community basis before the
interviewing team leaves the area. This has two advantages: (a) the
interviewers will be able to identify incomplete and possibly erroneous data
and return to the respondents in order to rectify them, and (b) at this central
or programme level, there will be fewer data to analyse. According to the
figures of the above‑mentioned example, only 20 data sets instead of 500.
When working with base‑line
data, the analyst must keep in mind that these represent observations at a
single point in time. Thus, they only permit comparisons among communities and
with targets. In contrast, restudies, which form the core of the monitoring and
evaluation system, in combination with base‑line data, represent
observations at two points in time. Hence, they allow a communityby‑community
comparison over time. In step 4.1, this type of comparison, was found to be the
most appropriate for measuring change, including regress in programme
implementation and in achievement of programme objectives.
Most of the problems
discussed above can be reduced to a common denominator. They relate to the
basic question concerning the possibilities and limitations of data analysis.
The different aspects of this question have been discussed here only in a
general way. In each case of programme monitoring and evaluation they would
need further specification, according to the sampling and data collection
procedures actually selected. Clarification of the way in which data obtained
through various procedures can be analysed is especially important. It helps
determine to what extent the monitoring and evaluation results will match the
information needs of decision-makers, in both specific and general terms.
Step 5 Timing of
research when and how often to collect data
It has already been
emphasized that in order to produce relevant, timely and. accurate data,
monitoring and. evaluation activities must be designed as a continuous process
of data collection, analysis and judgement. The heart of systematic monitoring
and evaluation is thus the replication of the case studies included in the base‑line
survey. Only through this approach can an accurate assessment of the
relationships between programme resources, activities and. results be obtained.
In essence, the replication or restudy procedure consists of returning to
previously studied sampling units, periodically collecting data that relate to
principal programme conversion factors and results, comparing these data with
base‑line findings, assessing differences between various sets of
measurements, deriving conclusions from them about progress made in programme
implementation and achieving stated objectives.
There are two
different aspects to the timing of repeat studies: determining the length of
research intervals and setting dates for data collection.
The research interval
is the time between two measurements or, in other words, the reporting period
covered by a restudy. In deciding upon the length of the interval, several
criteria must be taken into account:
(a) Resources
available for monitoring and evaluation purposes Staff time and financial
means allocated to information‑gathering activities limit the number of
studies that could be conducted;
(b) Life span of
the programme Dividing the number of years over which the programme will
operate by the maximum number of possible studies provides a first figure for
the length of the research interval;
(c) Time the
programme needs for producing outputs effects and impact If these various
periods are found to be longer than the research intervals computed under point
(b), there might be no use in undertaking the maximum possible number of
monitoring and evaluation studies. If studies are carried out before the
programme has entered into a next phase, i.e., before outputs are produced,
outputs have led to effects or effects have caused impact, no new information
will be obtained. If, however, programme phases are expected to be shorter than
research intervals, priorities must be set: Should the data system focus on the
programme implementation process, or should it rather concentrate on programme
effects and impact? Should the emphasis be on monitoring, ongoing or ex post
evaluation? The answer to this question depends on the main purpose
monitoring and evaluation are intended to serve: Should they assist in
programme management, planning or policy formulation?;
(d) Length of the
study cycles This is the time required for completing data collection,
analysis and reporting of findings. Before commencing a new study cycle, the
preceding one should generally be completed in order to prevent raw data from
piling up and remaining unanalysed;
(e) Administrative
calendar of the programme The periodicity of studies should also be related
to key administrative events in the programme. Changes in programme management
are frequently restricted by planning and budgetary procedures. Reports may be
required by political and administrative bodies. Data should be collected so as
to be available at these critical points in the politico‑administrative
cycle.
After all these criteria have been carefully
considered. and a decision has been taken as to the desired length of the
research intervals and the subsequent number of restudies, the next problem to
examine involves the timing of data collection. This is a crucial problem in
all cases where the phenomena to be measured are affected by seasonelity or any
other time factors, e.g., festivals, vacation periods, production, planning or
budgetary cycles.[13]
If the variables are time‑dependent,
great care must be exercised in order to ensure that the data can be collected
at the critical points identified. An important aspect in this connexion is
ascertaining that enough qualified and trained field investigators are
available at the time when the data must be collected.[14] Failure to recognize that the phenomena to be
studied are influenced by seasonality and periodicity will distort the
monitoring and evaluation results and lead to false conclusions and judgements
about the effectiveness of the programme.
Because it is desirable to provide decision
makers with a constant flow of information, it may sometimes be preferable to
stagger the restudies over time. For example, in a programme whose current
stage will run for three years, and whose base‑line survey includes 30
case studies, it could be appropriate to restudy 10 cases each year in order to
assess continuously its performances and outcomes.
Step 6 Communicating monitoring and
evaluation results how to report findings
The evaluators' task is not completed until the
findings of monitoring and evaluation research have been communicated to the
relevant decision makers. Reporting of results is one of the most crucial
phases in the monitoring and evaluation process,since it is here that the link
between research and decisionmaking on the progratmie must be established.
The pre‑condition of successful feedback of information is threefold.
First, the monitoring and evaluation report must be capable¥ of attracting the
decision makers' attention. Secondly, decision makers must have confidence in
the accuracy of the data presented to them. Thirdly, the findings and
conclusions derived from them must clearly show the possible alternatives for
future action. Finally, whether and to what extent monitoring and evaluation
actually serve as a tool for rationalizing decisionmaking, as they are
intended to do, depends largely upon how their findings are reported back to
the decision makers.[15]
The array of audiences who are to receive monitoring and evaluation
reports can easily be identified again by looking back to the information
matrix set up at step 2 of the designing process. This matrix also indicates
what exactly the users want to know about the programme. Whatever their
particular information needs, users should always be given enough details on
the monitoring and evaluation procedures, especially the sampling techniques
and methods of data collection and analysis, in order that they may form their
own opinions on the reliability and validity of the information obtained. A
single report containing all the significant findings on programme performance
and outcomes would naturally be the most economical and at the same time the
fastest way of reporting.[16] But presentation of a common body of information
would probably be rather voluminous in many cases. It could discourage many
decision makers from examining the material closely, and thereby lead to non‑utilization
of findings. The solution lies in developing a very detailed and clear table of
contents that enables the reader to locate immediately the figures and
conclusions pertaining to issues that most concern and interest him.
As is evident from the foregoing, the question
of what information to report is closely linked with that of how to present
information most effectively. The following reporting procedures promise to
enhance the utilization of monitoring and evaluation findings:
(a) Results should be set out with reference
to the logical programme framework Reference to the programme framework,
i.e., the definitions, indicators and hypotheses established there, will help
to avoid terminological ambiguities and disagreement on what the programme is
intended to do;
(b) Findings should be reported promptly. A
speedy feedback of information is especially important with regard to short‑term
decision‑making, e.g., programme management. Any delay in reporting will
lower the influence potential of monitoring and evaluation;
(c) phasis
should be placed on assessing the effectiveness of the project
components included in the programme The issue to be decided is
usually not whether or no to have a programme but, rather, what approach to
adopt towards development planning. Monitoring and evaluation, therefore,
should not so much aim at an over‑all "go/no‑go"
assessment of the programme, but concentrate primarily on finding out which of
the programme elements functioned successfully, where things went wrong and why
this happened. Such detailed information on the programme process would clearly
point out corrective measures that are needed and thus guide decision makers in
selecting among alternative courses for future action.[17]
The foregoing discussion underlines once more a fact that has been
repeatedly stressed: If monitoring and evaluation are to serve as a tool for
decision‑making, they must provide information that is relevant, i.e.,
geared to the specific needs and interests of decision makers.
Step 7. Assigning responsibilities who is to
perform the monitoring and evaluation tasks
Since systematic monitoring and evaluation are designed to assist in the
tasks of planning and managing a development programme rather than in the
academic study of such a programme, principal responsibility for its conduct
should rest with the programme staff rather than with special outside
consultants.
It may be argued that the programme staff might
not always be completely impartial in their assessments, nor might they have
the necessary technical competence to execute effectively all aspects of the
systematic monitoring and evaluation process. In such cases, it may be useful
to have outside experts in methods and procedures for monitoring and evaluation
to train programme staff, as well as to participate in special technical
activities.[18]
However, the continual presence of outside experts as a normal
monitoring and evaluation procedure could, in fact, prove to be detrimental to
over‑all programme operations.
In the first place, outside personnel do not
have the same knowledge of the local conditions and populace as do programme
staff. This consideration becomes especially important where the development
activity has been designed to promote socio‑cultural progress as well as
economic growth. If the programme staff have been performing their tasks
conscientiously and effectively, they have become trusted and respected by the
programme's beneficiaries. In contrast, repeated visits by outside personnel
may create an impression among the local people that these individuals have
come to conduct studies (or experiments) on them for purely personal purposes.
Once programme credibility has been challenged, maintenance of programme
momentum becomes difficult.
A second difficulty arising from the use of
outside personnel to carry out systematic monitoring and evaluation is that many
of the individuals connected with a development programme (programme staff,
counterparts, volunteers etc.) may tend to look upon such periodic review and
assessment as little more than a check upon their own functions and/or as a
conscious attempt to detect programme difficulties or short‑comings.
Staff morale is as critical to effective operations as over‑all programme
credibility.
With respect to the design and execution of a
programme monitoring and evaluation system, there are certain instances in which
outside assistance may be of great advantage. These would include formulation
of the monitoring and evaluation strategy, and the execution of the ex ante and
ex post surveys. In all such instances the role of the outside expert
should be that of an adviser. He should work with the programme staff, rather
than supplant their mandated responsibilities, inspect programme operations or
audit programme accounts.
There is a second aspect to the question of who
is to perform the monitoring and evaluation tasks. It relates to the problem of
who should make evaluative judgments. Should the evaluators' role, besides
collecting, analysing and reporting data, also include the function of passing
judgements about programme performance and achievements? Or should this be left
to the decision makers? There are good reasons for and against each of the two
strategies.[19]
Whichever is selected, clarification
of mutual expectations is absolutely necessary in order to avoid friction in
the role relationship between evaluators and decision makers.