CiPA Logo

PADM 5345. Evaluation of International Programs and Projects

Current as of June 6, 2017

home button

syllabus button

resources button

participants button

Syllabus

Master Schedule of Sessions and Assignments

The in-person sessions in Spring 2018 will meet on Tuesdays and Thursdays in Martha van Rensselaer 280 from 13:25-14:40.

On-line participation will be available.

 
Course Approach

The course examines results-based management of the programs and projects of international organizations, public and not-for-profit, with an emphasis on the evaluation aspect.  To do this, it examines how results-based management concepts and evaluation theory can be applied at the international level, takes the student through the process of planning an evaluation, examines the main analytical techniques that are usually employed and, as a final step, undertakes critical examinations of evaluations of specific projects and programs made by the main organizations at the international level.

2015 was International Year of Evaluation!

  There will be a simulation of the planning of an evaluation in results-based management and exercises related to each of the main analytical techniques. At the Maxwell School of Syracuse University, in Spring 2011, the evaluation simulation was of the Internet Governance Forum. In Fall 2010 it was the evaluation of the Departmet of Economic and Social Affairs of the United Nation. In 2009 it was a simulation of an evaluation of the Internet Governance Forum and of the current United Nations climate change institutions (the Global Environment Facility, the UN Framework Convention on Climate Change's Clean Development Mechanism, UNEP and the United Nations Convention to Combat Desertification.) For Fall 2011, it was the Gender Equity Programme of the Food and Agriculture Organization of the United Nations. For Spring 2012, it was the Intergovernmental Panel on Climate Change (IPCC). For Spring 2013 at Cornell, the subject was two real evaluations, one of the UN Convention to Combat Desertification and the other of the Research Cooperation Program of the Swedish International Development Cooperation Agency. The subject for Spring 2014 and also for Spring 2016 was the Intergovernmental Panel on Climate Change. In Spring 2015, it was UN Women. In addition, as a final assignment, each student will design an evaluation of a program or project of her or his choice.  The interactive part of the course takes place through a weekly live discussion session which is also available over the Internet.  Each student is expected to participate in either the in-person or through the on-line faciltiy.  The professor will hold regular office hours, either over the Internet or in M van Rensselaer 186. His office hours for Spring 2018 are Tuesdays and Thursdays from 9:30-11:30 or by appointment (even if coming during office hours it is better to schedule an appointment). The Professor can also hold office hours on request over Skype.
Course Organization The interactive part of the course will be a live discussions on Tuesday and Thursday at 13:25 in M Van Rensselaer Hall 153. Students are expected to participate in this each week. It is expected that a recording from session will be posted the day after the session for those who could not attend or for review by those who could. The lecture, readings and an audio or video presentation will sometimes be put up on the site by Tuesday of the week prior to the session. The on-line participation through Webex can be arranged and for that students should have access to a personal computer that has sound and if possible a microphone.
Readings

The course has one required text RealWorld Evaluation, Second Edition, by Michael Bamberger, Jim Rugh and Linda Mabry. Mike Bamberger was the chief sociologist for the World Bank for many years and Jim Rugh is head of evaluation for CARE. The book can also be a useful reference in the future.  Other required readings will come from evaluation materials of international public and non-governmental organizations, that are mostly available over the Internet.

Responses to Session Questions For each of the weekly sessions, there are a series of questions to be addressed. Students are expected to provide short, one-sentence answers to each of the questions and send them to the graduate assistant for the course who will assemble them on Sunday before class.

Simulation

simulation logo

The first half of the course will include the simulation of planning and implementing an evaluation of an organization to be chosen.

Participants will be part of an external evaluation firm hired to undertake the evaluation for the next meeting of the organization's board in April. There will be weekly discussions of progress with the Managing Director of the firm or his designated facilitator.
Office Hours

Prof. Mathiason's office hours for Spring 2018 will be on Tuesdays and Thursdays from 9:15-11:30 in MvR 3M29.

Grading

The course will involve a final paper, consisting of an evaluation design of an international program or project. Details on the assignment are found on the final paper page. The paper will be a detailed evaluation design written as Terms of Reference. Please note that a first draft must be submitted on time for comments. If that is not done, the grade is reduced.

Final paper
60%
Participation in the simulation
30%
Participation in the on-line chats or in-person discussion sessions
10%

 

Master Schedule of Sessions and Assignments

Date Section of course/Content of the session
January 25

1. The craft of result-based management (RBM) and its use in international organizations

That session will review concepts basic to all results-based management and their evaluations of public institutions and discuss why this is critical for international programs and projects. The session will also review the policies and approaches to results-based management of the main groups of international organizations: the World Bank, UNDP and other UN Funds and programs, the Specialized Agencies and the United Nations Secretariat. It will look at the standards adopted by the United Nations Evaluation Group. The emphasis will be on how concepts are modified to suit the specific environment of the international organizations.

The RBM process I:  This section will follow the main steps in doing strategic planning in international organizations.  The class will simulate the planning of an evaluation in an international organization.
January 30, February 1

2. Determining objectives, outcomes and the output to make them happen and the role of evaluation in the UN System

The session will explore how objectives and outcomes should be defined, as well as the output to induce them. These are also the critical elements for evaluation, since they define what has to be evaluated.  It will include a focus on the logical framework (log-frame) which is the main tool used for this stage of planning, as it is applied in international organizations. It will also discuss the role of evaluation in international organizations. Problems to be addressed by the organization due on 30 January.

February 6, 8

3. Determining performance indicators and means of obtaining information

The session will explore how information on results can be obtained for subsequent monitoring and evaluation and the tradeoffs involved.  It will include planning for information acquisition. First draft of logframe due on 7 Febrary

February 13, 15

4. Evaluation Quality Assessment

The session will explore how information on results can be obtained for subsequent monitoring and evaluation and the tradeoffs involved.  It will include planning for information acquisition. Final draft of logframe due on 14 February

February 22 This session will summarize the design process to be used in evaluations.
The RBM process II:  This section will follow the main methods of information collection for monitoring and evaluation
February 27,March 1

5. Obtaining information from documents

The session will begin to explore the use of content analysis techniques for drawing information from published document.
March 6, 8

6. Obtaining information from surveys

The session will explore the use of sample surveys and questionnaires to obtain information on program and project impact.

Topic of final paper due. Send it as a Word Document giving the title and a brief statement of why you are interested in the chosen program or project to Professor Mathiason. Please format the document title with your last name, first name and the name of the program or project.

 March 13, 15

7. Obtaining information through case studies

The session will explore the use of case study methodology as a means of undertaking evaluation

 

March 20, 22

8. Analyzing information

The session will consist of an overview of main simple data analysis techniques used in international evaluations: descriptive statistics, cross tabulation and analysis of variance and correlation and regression.  Examples will be drawn from actual evaluations. Each group will present its proposal for the use of surveys, including who to survey, sampling method and a questionnaire on one aspect of the survey. Outline of final paper due by March 25. [Please format the document title with your last name, first name and the name of the program or project.

The RBM process III: This session will look at evaluation models in international practice:  This section will include actual cases of evaluations conducted by the main types of international organizations.  In advance of each session, the participants will be expected to make a critique of the evaluation being examined.
March 27, 29

9. Evaluation of a United Nations Program

The session will explore an evaluation of the Global Programme on Money-laundering of the United Nations Office on Drugs and Crime as well as of the Division for Economic and Social Council Support and Coordination.
April 10, 12

10. Evaluation of World Bank projects

The session will examine critically an evaluation produced by the Bank's Independent Evaluation Group ( formerly the Operations Evaluation Department). Three groups will present their ToRs

April 17, 19

11. Evaluation of a UNDP project

The session will examine critically an evaluation produced by UNDP of an ILO project. First Draft of final paper due on April 21 but not later! Please format the document title with your last name, first name and the name of the program or project. Two groups will present their ToRs.

April 24, 26

12. Evaluation of an NGO project

The session will examine critically an evaluation produced by World Vision. Three groups will present their ToRs

May 1, 3

 

13. Presentation of results

The session will include a discussion of how results should be presented. Several class members will present their ToRs.

Summing Up

May 8

 

14. Summing up: The Future of Evaluation in the International Public Sector

The session will discuss common problems and drawing of general conclusions about evaluations.

May 11 Final paper due on May 11!

John R. Mathiason.
Copyright 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2011, 2012, 2013, 2014, 2015, 2016, 2017 John R. Mathiason. All rights reserved.
Revised: February 6, 2016