29 January 2024
The first step in planning an evaluation is to determine what to evaluate. This is not necessarily very easy at the international level.
In national public service, the issue seems to be more clear-cut, since governments have direct control over the delivery of services. Evaluating delivery of a service can be as simple as answering the question, "was the service delivered?" It becomes more complex if the question is asked "did the delivery of the service make its intended change on the lives of its beneficiaries?"
For an international program intending to have an effect at the national level, the difficulty is compounded by the fact that the direction of causality is even more indirect. The first question that often has to be asked is "did the organization's input change the way the service was delivered?" The next question is "did the delivery of the improved service make its intended changes?"
An example is a project of the IAEA. Combating Desertification in the Sahel (RAF/5/048). The project is intended to induce the use of nuclear techniques in research on water supplies, soil and nutrients. This is supposed to lead to increased availability of water and better soil conservation. This, in turn, is supposed to increase crop yields and retard desertification. The distance between the outcome expected of the IAEA project and the desired impact is very great.
This distance from the action can only addressed by carefully plotting the assumed relationship between the program's actions and the results. In one of the classic empirical studies of voting behavior (the American voter studies of the late 1950's), the concept of a "funnel of causality" was proposed. As one got further away from the action (in that case who you voted for), the possible causal factors became more general and more numerous. The further up the funnel, the less secure any conclusions about causality would be.
The objective of the first step in an evaluation is to define results, as objectives and outcomes, in such a way that they are realistic, attainable, plausibly in a causal relationship with the program or project and, above all, measurable.
The introduction of results-based programming and budgeting in organizations of the United Nations system has focused attention on the need to do a careful job of specifying objectives and outcomes. While the terminology varies somewhat between organizations (the UN Secretariat doesn't use the term "outcome" but rather uses "expected achievement" defined as "A desired outcome or result involving benefits to end-users, expressed as a quantitative or qualitative standard, value or rate. Accomplishments are the direct consequence or effect of the delivery of outputs and lead to the fulfilment of the envisaged objective."), the underlying concepts are the same. They are also used in the operational organizations. All are derived from the logical framework, although the terms used are slightly different. In this course, we use the terms for the log-frame employed by the IAEA, as well as most of the operational organizations and by the Organization for Economic Cooperation and Development (OECD) Development Assistance Committee (DAC). Their Glossary of Evaluation Terms is the standard that is usually used.
The highest level result that a program or project intends to achieve is termed an objective. Objectives are defined as:
A well-drafted objective meets a number of criteria:
The concept of outcome is central to evaluation of international programs: it is the direct result of what the program produces in terms of outputs and activities. It is not an objective, since it is not the final end-state, but rather it is the contribution that the program intends to make to achieving the desired end-state expressed in the objective.
The IAEA has been one of the most advanced and consistent in adopting results-based programming and has invested considerable effort in developing the concepts and training managerial staff in their use. The Agency's internal management training curriculum includes material on drafting outcomes and performance indicators. The Agency has developed a number of tools to help managers formulate objectives, outcomes and performance indicators. They can be used in planning any evaluation. One that has proven very useful is a Tool for Drafting Results-based Objectives.
An outcome:
A key difference between outcomes and output is that output is completed controlled by the program, but outcomes are outside control and can only be influenced. Thus, when a program or project promises to make an outcome happen, it is taking a significant risk. That is one reason that many programs have found it difficult to formulate outcomes.
Outcomes should be 'SMART'
1. Are they linked to Specific objectives?
2. Can progress be Measured (verified) for each of them?
3. Are they Achievable?
4. Are expected accomplishments Relevant to problem?
5. Can they be done in a Timely manner?
Outcomes should not be Activities or Trivial.
It is important that the distinction between output and outcomes is maintained.
The problem for evaluators is that, in the past, objectives may not have been well-formulated and outcomes not indicated at all. Consequently, the first step of the process often involves reformulating existing objectives and outcomes to match what was really intended and, often, formulating missing intended results.
In an evaluation of the UN's program of substantive support to Economic and Social Council in which I participated as a consultant, we confronted the dual problem that the stated objectives and "expected achievements" were actually activities, and that the Secretariat, dealing with sovereign governments who are members of the Council, had to be very careful in setting out what it intended to see happen (so that it wouldn't be perceived as seeking to manipulate sovereign governments.) For this, I worked with the program managers to see what they felt they could reasonably achieve as a Secretariat.
For operational organizations working in the development field, the danger is a bit different: it is assuming that the objectives of the national program and its international component are the same. In one sense they are, but the international component is not responsible for everything necessary to achieve the objectives. Instead, it is responsible for inducing changes that will enable the national component to function effectively.
The exercise of reviewing the objectives specified in the various programs of the United Nations or the UN climate change institutions is an example of how this can be done. You should make a clear distinction between the objectives that are essentially national and those that are uniquely international and also try to specify the outcomes that need to happen if the objectives are to be achieved in a reasonable period.
In looking at objectives and outcomes it is useful to consider that they are a form of a tree, in which the highest result is the objective, but that the intermediate results often work in sequence.
[1] Abraham Kaplan, The Conduct of Inquiry: Methodology for Behavioral Science, San Francisco, Chandler, 1964, p. 11.