PHASE I:  PLANNING THE SELF-EVALUATION PROCESS

Phase I of the self-evaluation process is about planning.  When this phase is done properly, self-evaluation can be effective at a reasonable cost in time and effort.  Planning a successful self-evaluation requires answering four questions in sequence:

·        What do we need to know?  What are the objectives and outcomes we expect to be able to observe? (subsection A below)

·        How much information do we need?  Can we make do with a simple design or do we have to have more detail? (subsection B below)

·        Which alternative self-evaluation approach is best for collecting the information?  (subsection C below)

·        How are we going to obtain the information? (subsection D below)

As each question is answered, the scope and approach of the self-evaluation defines itself. 

This section of the Guide helps Agency managers deal with each question by: describing the kind of planning needed for a successful evaluation, providing tools that can be used, and introducing three alternative approaches (models) for self-evaluation: simple, standard, complex.  It concludes with the Terms of Reference which builds on and reflects the answers to those basic questions.

A. What do we need to know?

Every project or programme has one or more objectives and outcomes with associated indicators, stating what is intended or what is "expected" to happen if it is successful. 

Step 1: Determine the period to be covered.  In answering the question "What is the time period that should be evaluated?", recognize that the longer the time horizon for the evaluation the more information that will be needed, as noted in Display 5. 

There are some obvious options for determining the period that should be covered by the self-evaluation:

·        the period since the most recent programme (PPAS) or other evaluation;

·        the period since a major re-orientation of the programme of which the project or programme is a part;

·        the period of the medium-term strategy.

 

Since most evaluations will cover more than a single biennium, the scope of information collection is likely to be a four to six-year period or longer.  A four-year horizon might involve comparing information at the beginning of the period, the middle and the end of the period.  A six-year horizon could involve comparing information at four or more points to show trends (the beginning, the end of each intermediate biennium and the end of the period.. 

It is critical to set a clear timeframe for the evaluation, since this will determine how much information will be needed.  This will also influence the choice of which of the three self-evaluation modules, as outlined in Display 5, to use.

Taken together, the answers found through these two steps will help determine what kind of information we need to collect during the course of the evaluation.  Specifically, information related to the performance indicators will be collected over the period of time determined to be appropriate.  Of course, additional information may also be needed to aid interpretation, but these are the central information elements that need to be obtained about the programme results for an evaluation to be successful.

Step 2: Determine the objectives and outcomes.  To answer the question, determine what, realistically, the objectives and outcomes are.  Under a result-based programming approach, the outcomes are (supposed to be) specified in measurable (i.e, verifiable) terms in the biennial Programme and Budget, in the context of programme objectives established over a specified time period.  Well formulated, the outcomes and associated performance indicators as well as the objectives provide a clear standard against which to judge.

 When this has been done properly, this phase is simple: what was programmed and agreed to by the Member States merely has to be listed.

For example [1] , in the case of Subprogramme C.1. Energy Modeling, Databanks and Capacity Building, the following are specified in the 2002-2003 Programme and Budget:

Outcome	Performance Indicator
·         Increased use of Agency recommended engineering and management practices, quality assurance guidance, supporting databases and training methodologies for improvement of nuclear power reliability and competitiveness	·         Number of Member States using and requesting direct assistance for application of Agency recommended engineering and management practices, quality assurance guidance, supporting databases and training methodologies

For this subprogramme, if Member States use Agency recommended practices and the  "programme results" can be observed, the evaluation is relatively simple.

Results have only been formulated with this level of precision for biennia beginning in 2002.  For the immediate term, evaluations will have to take into account the fact that objectives and outcomes in the past were not as precise. 

On the other hand, when the outcomes and associated performance indicators and the objective are not properly formulated they need to be "revisited."  Therefore, it will be necessary to "reconstruct" what the intended results really were in the past, so that a judgment can be made about whether the results have been attained.  For detailed guidance on reformulating objectives and outcomes, play the PowerPoint presentation entitled "What Do We Need to Know?" which shows how to use the "conversion tool" found in Annex B.

For example, in the case of Subprogramme C.2. Energy-Economy-Environment (3E) Analysis, the following are specified in the 2001 Programme and Budget:

The objective is not easily measurable (contribution is not defined and what a "full and fair hearing" means is not clear.  The performance indicators do not tell whether the objective has been achieved.  And, of course, the outcomes that will lead to the achievement of the objective are not specified.

As results-based programming takes hold, this step will become increasingly easy and, in fact, the performance monitoring and assessment system being set up by the Office of Programme Support will increasingly provide the basis for improved programming.

In short, the Agency managers needs to review his or her own objectives, outcomes and performance indicators, reformulating them when necessary, so they can be used subsequently as the basis (i.e, "standard") for measuring programme results during the course of the evaluation.

B. How much information do we need?

Once the outcomes to observe have been set out, it is possible to determine how much detail is needed.  Three steps need to be taken:

Step 1: Determine the extent outcomes can be observed.  This is done by asking and answering the question "To what extent can the outcomes, which have occurred, be observed simply and directly?" 

If the outcomes are explicit and specific and the performance indicators are obvious, the task is straightforward. 

An example of the former would be if the intended outcome is that states sign safeguards agreements, the observation is simple: either they did or they did not.   In this case it is only necessary to verify that fact by gather the necessary documentation and counting the number of signatures.

If the outcomes have to be inferred from data, however, the task is more complex. 

An example of the more complex, would be if the intended outcome is increased favorable coverage of nuclear power issues in the world press, it would be necessary to examine comparatively over time the coverage in major newspapers.  This will require doing bibliographic research, analyzing the content of newspapers and other periodicals that have been identified at different time periods and determining whether there has been a change in coverage favorable to the Agency.

Step 2:  Determine the "cause-effect" link. To what extent will it be necessary to (a) estimate the relationship between Agency output and the outcomes and (a) draw a relationship between the outcomes and impact.

If the managers need to be able to draw causal inferences in order to respond to perceived problems or make convincing conclusions, more detail will be needed

For example, if national reports need to be examined and their content compared with Agency recommendations, more information will have to be obtained and processed than if the test is whether resolutions adopted by the Board contain recommendations made in Secretariat reports.

Step 3:  Estimate resources available.  To do this, ask and answer the question "How much time is available for the evaluation and how many resources can be released to undertake it?"

Self-evaluations require a commitment of resources, which are always scarce, and a careful estimate of how many staff members can be deployed over what period of time can help determine how far the evaluation can go.

To help determine the time, resource and management considerations, three different approaches or models are described in Display 4 below in terms of resource estimates associated with each model.

C. WHICH Alternative Approach IS BEST? Three Models

Broadly speaking, there are three models for self-evaluation and the choice between them is dictated by both the types of information that must be gathered and the time and resources available.  Display 4 shows how these factors can be used to select a model.

For many self-evaluations, the simple approach will suffice, given time and resource availabilities; for most, the standard approach is most appropriate; and for critical self-evaluations, the extended approach should be employed.

Display 4.  Time, organization and resource levels for the three models
Factor	Simple	Standard	Extended
1.     Performance indicators	Directly observable	Can be observed largely from internally available information	Need externally-provided information to measure
2.     Time needed	Up to a week	Up to a month	Multiple months
3.     Staff needed	One nearly full-time or two part- time	One nearly full-time, one or more part-time	One full-time, several part-time
4.     Time Horizon	Up to 2 years	2-4 years	2-6 years
5.     Organization needed	Informal	Team with designated leader	Team with designated leader and engagement of other stakeholders

Some of the detailed considerations and the implications of each approach by phase are shown in Display 5 below.

Display 5.  Three models for self-evaluation in terms of phases
Phase	Simple	Standard	Extended
1. Planning
a. What to measure	Outcomes that are easily observable, over a short time horizon	Outcomes that require some effort to observe, over a medium time horizon	Multiple outcomes that are not easy to observe, over a medium- to longer-time horizon
b. Time and cost	Short time, few resources	Longer time, ability to invest some resources	Longer time, ability to invest considerable resources
2. Implementation
a. Methods for obtaining data	Content analysis and access recording	Content analysis, access recording, plus interviews and other methods within the Agency	Content analysis, access recording, internal interviews plus interviews,  questionnaires and other methods both within the Agency and externally
b. Type of data analysis	Comparison of actual with intended outcomes	Comparison of actual with intended outcomes, relationship between outputs and outcomes, Estimate of connections with probable impact	Comparison of actual with intended outcomes, relationship between outcomes and impact, relative effectiveness of output, efficiency of use of inputs.
c. Complexity of findings	Simple conclusions and recommendations	Conclusions about causality, recommendations based on effectiveness	Conclusions about causality, recommendations about relative effectiveness and efficiency.
3. Reporting	Internal, limited to immediate staff and supervisors	Internal, available to Agency senior management	Internal but made available to key external stakeholders
4. Follow-up	Programme adjustments	Programme adjustments incorporated into next P&B and strategic/medium-term plan	Major programme adjustments, reprogramming and planning