ÒHOW WELL ARE WE DOING?Ó
A GUIDE TO
PROGRAMME AND PROJECT EVALUATION
FOR AGENCY
MANAGERS
A recent mayor of the City
of New York was famous for stopping citizens on the street and on subway
platforms to ask ÒHow am I doing?Ó
In getting answers, Mayor Koch was doing what can be called programmatic
self-evaluation.
Effective project and
programme managers ask themselves ÒHow well are we doing?Ó They want to know with certainty
whether and how well they are achieving their objectives. More importantly, they want to know
what works and what doesnÕt, so that they can build on what does and fix that
which doesnÕt.
Techniques of programme
and project evaluation done for internal use, which hereafter will be termed
Òself-evaluationÓ, allow managers to answer this fundamental question
precisely, opportunely and convincingly, at a reasonable cost in time and
resources. They are essential
tools in the Agency managerÕs repertoire.
This Guide sets out the
self-evaluation methodology that the International Atomic Energy Agency (IAEA)
has designed to meet its needs and reflect its experience with project and
programme management. It is
for Agency managers (project and programme) and staff who plan to do a self Ð
evaluation. It provides
easy-to-use guidance and methodologies for determining how well the managerÕs
programme, project or service is achieving its stated objectives and whether it
is having the desired result for Member States or meeting its clients
needs.
The guide is not intended
to be a cookbook to be followed rigidly.
Rather, it is like the general lesson plan that teachers use: the Guide
has to be interpreted and modified to fit the managerÕs specific circumstances,
much as lesson plans have to be adapted by the teacher to the specifics needs
of the students. However,
it does represent the policy of the Agency and is to serve as the basis for
self-evaluations for the sake of consistency.
Agency managers are
expected periodically to determine whether they should undertake a
self-evaluation and to use the Guide in making that determination as well as in
planning and implementing the evaluation.
While the focus is on
projects, the self-evaluation process in this Guide can be tailored and applied
appropriately to other levels in the programme hierarchy (Programme and
Subprogramme). It also can be used
to look at administrative services and on-going functions (e.g., training) as
well as to contribute to larger-scale programme evaluations conducted by the
Office of Internal Oversight Services (OIOS) or by external parties.
The Guide consists of an
introduction, the four phases of an evaluation, and a set if appendices:
á
The introduction describes the context in which the Agency is doing
evaluation, the main concepts in use and the process that self - evaluation
follows.
á The four phases
describe the process of carrying out a self-evaluation, from planning to data
collection and analysis, reporting through follow-up. The Guide uses three different approaches (or levels) of
sophistication based on the kinds of evaluation needs that project managers
have identified: simple, standard and extended. It also describes a number of main methods that can be used to obtain
the necessary information and analyze it taking into account the three
approaches.
á The appendices offer related tools, some related to the three modules
and others generic to the evaluation process.
To complement this, a training workshop for managers who want to undertake self-evaluations
has been developed. While the Guide can be used directly by an
Agency manager (and his/her staff), it is also the basis for a training
workshop that the Agency makes available to its staff. The workshop sets out the concepts and
context for self-evaluation, presents the optional approaches and related tools
managers in the Agency have found useful for answering the central
question: ÒHow well are we doing?Ó
É
This section of the
Guide outlines the context for self-evaluation, defines underlying concepts and
highlights the process of self-evaluation.
Results-based Approach
The Agency now uses results-based programming as the basis
for its programme management approach.
Evaluation is one of the three key components in this, because it
demonstrates objectively whether the results promised were actually delivered.
Evaluation results provide feedback to the other two components of that
process, namely (a) programme planning, including formulation, and (b)
programme implementation, including performance monitoring and assessment of
delivery. (See Display 2)
The results-based approach responds to the concerns of
Member States to ensure that the public resources they provide to international
organizations are used in the optimal way. It also responds to the increasing importance of the work of
international organizations like the IAEA in dealing with problems that
transcend national boundaries. To
ensure continued support, the Agency not only has to deliver the results it
promises, but it has to be able to show clearly that it has delivered them.
The Agency has a policy on programme evaluation,
SEC/NOT/1879 issued on 4 March 2002, which incorporates, among other things,
self-evaluation. It is based on a
hierarchy of evaluations, as shown in Display 1 below.
Display 1. The evaluation hierarchy
At the apex of the pyramid are the independent external
evaluations managed by the Member States (e.g, the UN Joint Inspection Unit and
the External Auditor). Then there
are internal independent evaluations undertaken by the Office of Internal
Oversight Services (OIOS). Next
are participatory evaluations that involve both Agency and Member State
evaluators. Then, at a lesser
level of formality, are the self-evaluations performed by the Member States of
their own programmes that are supported by the Agency and, at the base of the
pyramid, self-evaluations undertaken by Agency managers themselves.
The first priority for a programme manger, of course, is to
deliver programme results.
Self-evaluation facilitates this because it:
- enables
Agency managers to determine how well the project is doing and what
adjustments are needed to improve performance; and
- helps
develop support for the programme by being able to demonstrate to
stakeholders that the resources they are providing are doing the job.
Timing
While evaluation can be done at the end of the programme
management cycle, it can also be part of the cycle at different times. It is particularly important when
strategic objectives are being reviewed and when there are questions about the
effectiveness of the project in obtaining the desired results. While the specific stimulus for
deciding to do a self-evaluation can be varied, the most common are when:
á
there have been major external changes that require the
Agency to adapt its programmes to new challenges, or
á
it is anticipated that a programme needs to move in new
directions during the next biennium programme or medium-term plan period, or
á
a manager senses that something is not going as well as
expected in areas under his or her purview.
Ideally, however, the project manager will have considered
when he/she would like to do a self-evaluation at the time of project
formulation, building that into the project design itself.
In formal terms, evaluation is the process of determining whether the
programme entity has achieved or is achieving over time its stated objective and,
therefore, is having the desired impact for the intended beneficiaries. A self-evaluation is an internal
process primarily for the use of managers to improve their performance results.
In contrast, programme evaluations conducted by OIOS are in-depth
independent exercises that involve a larger number of stakeholders, often
external evaluators, and usually have a public purpose.
The primary focus of each, however, is on results in the collective sense and their relation to the
impact on Member State (or internal client) needs. Whereas Òperformance assessmentÓ focuses on whether the
outcomes in a given biennium occurred as a result of the services or products
provided by Agency programmes and projects, evaluation focuses on Òprogramme
resultsÓ over a multiple year time period that refers to three different things
taken together:
á
the output
(service or product) of a programme or project,
á
the outcome (changes)
that is induced by the output, and
á
the objectives
that are achieved (often referred to as ÒimpactÕ) because of the effect or
consequences on Member State (or internal client) needs.
In practical terms, a major focus is on the
outcome, which is the direct result of the AgencyÕs work. An evaluation seeks to verify whether
the outcomes have occurred and determine whether they plausibly have led to the
kind of impact that would show the objectives sought had been or are being
achieved.
A successful evaluation starts by
understanding what the key concepts of output, outcome and impact mean in terms of oneÕs specific programme
or project. It requires that each
be defined in ÒmeasurableÓ (verifiable) terms.
Impact refers to the achievement of
objectives. In the AgencyÕs usage,
an objective is the
description of the end-state that should be observed at a certain point in
time. It is the result or consequence of changes in the
situation of a beneficiary population.
In the instruction
for the preparation of the 2002-2003 programme, the example was given of the
programme on Radiation Protection and Safety of Radiation Sources: ÒTo increase the capability of Member States to
achieve and maintain a high level of safety in the use of radiation sources and
radioactive materials and their transport and protect humans and their
environment against the effects of radiation exposure.Ó
For
this objective to be achieved, Member States would have to modify their rules,
procedures, training and supervision of personnel in order to ensure
safety. These changes would be the
impact of the programme.
Impact can be the result of many different factors, of which
the Agency will only be one. An
evaluation should be able to link the outcome induced by the Agency with the
changes that have taken place as they lead, over the longer term, to attainment
of the objectives and thereby to the desired effects on Member State needs.
An outcome is the main result of Agency work. As the AgencyÕs document setting out
the results-based approach puts it: ÒAn outcome
is the specific and observable changes to be induced by the Agency's actions
during a biennium if the objective is to be achieved.Ó
An outcome is the direct
and typically intermediate effects of a programme's output. It is a change in the population
directly affected by the output.
It is something that happens and can be described. It can be an action (e.g. a product is
used) or a condition (radiation measurement instruments are in place). The type
of outcome depends on the programme and what it intends to accomplish.
The outcomes together with
the objective are the central focus of evaluation. Was the intended change induced by the output produced by
the AgencyÕs work? Did the outcome
provoke intended impact (was it plausibly related to an observed change)? Is the objective being fulfilled? Why is it or isnÕt it? Is that resulting in meeting the Member
States needs?
Output consists of the
products produced and services provided by the Agency in order to induce
outcomes. It is a focus of
evaluation in the sense that it is necessary to determine whether the output
led to the outcomes and the objectives.
Effectiveness and Efficiency
Two concepts are used in
looking at the relationship among output, outcome and impact:
Effectiveness refers to the extent to which the output leads to the
outcomes and thence to the achievement of objectives.
Efficiency refers to the relationship between inputs (in the
form of resources like staff, funds, equipment) and output.
Of
the two, effectiveness is the more important. Put another way, an inefficient project that is effective is
better than an efficient project that is ineffective. Of course, the optimal situation is to have projects and
programmes that are both effective and
efficient. The evaluationÕs first
purpose is to assess effectiveness.
Once this is done, it can look at the issue of efficiency.
