Handbook on Impact Evaluation

Handbook on Impact Evaluation

-

Documents
262 pages
Lire
YouScribe est heureux de vous offrir cette publication

Description

Public programs are designed to reach certain goals and beneficiaries. Methods to understand whether such programs actually work, as well as the level and nature of impacts on intended beneficiaries, are main themes of this book. Has the Grameen Bank, for example, succeeded in lowering consumption poverty among the rural poor in Bangladesh? Can conditional cash transfer programs in Mexico and Latin America improve health and schooling outcomes for poor women and children? Does a new road actually raise welfare in a remote area in Tanzania, or is it a 'highway to nowhere'?
This book reviews quantitative methods and models of impact evaluation. It begins by reviewing the basic issues pertaining to an evaluation of an intervention to reach certain targets and goals. It then focuses on the experimental design of an impact evaluation, highlighting its strengths and shortcomings, followed by discussions on various non-experimental methods. The authors also cover methods to shed light on the nature and mechanisms by which different participants are benefiting from the program.
For researchers interested in learning how to use these models with statistical software, the book also provides STATA exercises in the context of evaluating major microcredit programs in Bangladesh, such as the Grameen Bank. The framework presented in this book can be very useful for strengthening local capacity in impact evaluation among technicians and policymakers in charge of formulating, implementing, and evaluating programs to alleviate poverty and underdevelopment.

Sujets

Informations

Publié par
Publié le 13 octobre 2009
Nombre de visites sur la page 43
EAN13 9780821380291
Langue English
Signaler un problème

Handbook on Impact
EvaluationHandbook on Impact
Evaluation
Quantitative Methods and Practices
Shahidur R. Khandker
Gayatri B. Koolwal
Hussain A. Samad© 2010 The International Bank for Reconstruction and Development / The World Bank
1818 H Street NW
Washington DC 20433
Telephone: 202-473-1000
Internet: www.worldbank.org
E-mail: feedback@w
All rights reserved
1 2 3 4 13 12 11 10
This volume is a product of the staff of the International Bank for Reconstruction and Development / The World
Bank. The fi ndings, interpretations, and conclusions expressed in this volume do not necessarily refl ect the views
of the Executive Directors of The World Bank or the governments they represent.
The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors,
denominations, and other information shown on any map in this work do not imply any judgement on the part of
The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries.
Rights and Permissions
The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without
permission may be a violation of applicable law. The International Bank for Reconstruction and Development /
The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions
of the work promptly.
For permission to photocopy or reprint any part of this work, please send a request with complete information
to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400;
fax: 978-750-4470; Internet: www.copyright.com.
All other queries on rights and licenses, including subsidiary rights, should be addressed to the Offi ce
of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail:
pubrights@worldbank.org.
ISBN: 978-0-8213-8028-4
eISBN: 978-0-8213-8029-1
DOI: 10.1596/978-0-8213-8028-4
Library of Congress Cataloging-in-Publication Data
Khandker, Shahidur R. Handbook on impact evaluation : quantitative methods and practices / Shahidur R. er, Gayatri B. Koolwal, Hussain A. Samad.
p. cm.
Includes bibliographical references and index.
ISBN 978-0-8213-8028-4 — ISBN 978-0-8213-8029-1 (electronic)
1. Economic development projects—Evaluation. 2. Economic assistance—Evaluation.
I. Koolwal, Gayatri B. II. Samad, Hussain A., 1963- III. Title.
HD75.9.K52 2009
338.90072—dc22
2009020886
Cover design by Patricia Hord.Graphik Design.Contents
Foreword ................................................................................................................................xiii
Preface ..................................................................................................................................xv
About the Authors ......................................................................................................................... xvii
Abbreviations ................................................................................................................................ xix
Part 1 Methods and Practices ................................................................................... 1
1. Introduction..............................................................................................................3
References .......................................................................................................6
2. Basic Issues of Evaluation ....................................................................................7
Summary ........................................................................................................7
Learning Objectives .......................................................................................7
Introduction: Monitoring versus Evaluation ...............................................8
Monitoring .....................................................................................................8
Setting Up Indicators within an M&E Framework ......................................9
Operational Evaluation ...............................................................................16
Quantitative versus Qualitative Impact Assessments .................................18
ve Impact Assessment: Ex Post versus Ex Ante
Impact Evaluations ...................................................................................... 20
The Problem of the Counterfactual ............................................................22
Basic Theory of Impact Evaluation: The Problem of Selection Bias .........25
Different Evaluation Approaches to Ex Post Impact Evaluation ...............27
Overview: Designing and Implementing Impact Evaluations ..................28
Questions .....................................................................................................29
References 30
3. Randomization .......................................................................................................33
Summary ......................................................................................................33
Learning Objectives .....................................................................................33
vContents
Setting the Counterfactual ..........................................................................34
Statistical Design of Randomization ..........................................................34
Calculating Treatment Effects .....................................................................35
Randomization in Evaluation Design: Different Methods
of Randomization ........................................................................................38
Concerns with Randomization ...................................................................38
Randomized Impact Evaluation in Practice ...............................................39
Diffi culties w.................................................................47
Questions .....................................................................................................49
Notes .............................................................................................................51
References 51
4. Propensity Score Matching ................................................................................53
Summary ......................................................................................................53
Learning Objectives .....................................................................................53
PSM and Its Practical Uses ..........................................................................54
What Does PSM Do? ...................................................................................54
PSM Method in Theory ...............................................................................55
Application of the PSM Method .................................................................58
Critiquing the PSM Method .......................................................................63
PSM and Regression-Based Methods .........................................................64
Questions .....................................................................................................66
Notes .............................................................................................................67
References 68
5. Double Difference.................................................................................................71
Summary ......................................................................................................71
Learning Objectives .....................................................................................71
Addressing Selection Bias from a Different Perspective:
Using Differences as Counterfactual ...........................................................71
DD Method: Theory and Application ........................................................72
Advantages and Disadvantages of Using DD .............................................76
Alternative DD Models ................................................................................78
Questions .....................................................................................................82
Notes .............................................................................................................84
References 84
viContents
6. Instrumental Variable Estimation ......................................................................87
Summary ......................................................................................................87
Learning Objectives .....................................................................................87
Introduction .................................................................................................87
Two-Stage Least Squares Approach to IVs .................................................89
Concerns with IVs .......................................................................................91
Sources of IVs ..............................................................................................95
Questions .....................................................................................................99
Notes ...........................................................................................................100
References ...................................................................................................100
7. Regression Discontinuity and Pipeline Methods .........................................103
Summary ....................................................................................................103
Learning Objectives ...................................................................................103
Introduction ...............................................................................................104
Regression Discontinuity in Theory .........................................................104
Advantages and Disadvantages of the RD Approach ...............................108
Pipeline Comparisons ...............................................................................110
Questions ...................................................................................................111
References 112
8. Measuring Distributional Program Effects ....................................................115
Summary ....................................................................................................115
Learning Objectives ...................................................................................115
The Need to Examine Distributional Impacts of Programs ....................115
Examining Heterogeneous Program Impacts:
Linear Regression Framework ...................................................................116
Quantile Regression Approaches ..............................................................118
Discussion: Data Collection Issues ...........................................................124
Notes ...........................................................................................................125
References ...................................................................................................125
9. Using Economic Models to Evaluate Policies ..............................................127
Summary ....................................................................................................127
Learning Objectives ...................................................................................127
Introduction ...............................................................................................127
viiContents
Structural versus Reduced-Form Approaches .........................................128
Modeling the Effects of Policies ................................................................130
Assessing the Effects of Policies in a Macroeconomic Framework .........131
Modeling Household Behavior in the Case of a Single Treatment:
Case Studies on School Subsidy Programs ...............................................133
Conclusions ................................................................................................135
Note ............................................................................................................136
References ...................................................................................................137
10. Conclusions ..........................................................................................................139
Part 2 Stata Exercises ............................................................................................ 143
11. Introduction to Stata ..........................................................................................145
Data Sets Used for Stata Exercises .............................................................145
Beginning Exercise: Introduction to Stata ................................................146
Working with Data Files: Looking at the Content ...................................151
Changing Data Sets ....................................................................................158
Combining Data Sets .................................................................................162
Working with .log and .do Files ................................................................164
12. Randomized Impact Evaluation ........................................................................171
Impacts of Program Placement in Villages ...............................................171
I Program Participation ............................................................173
Capturing Both Program Placement and Participation ..........................175
Impacts of Program Participation in Program Villages ...........................176
Measuring Spillover Effects of Microcredit Program Placement ............177
Further Exercises ........................................................................................178
Notes ...........................................................................................................179
13. Propensity Score Matching Technique ..........................................................181
Propensity Score Equation: Satisfying the Balancing Property ...............181
Average Treatment Effect Using Nearest-Neighbor Matching ................185
Average Treatsing Stratifi cation Matching ........................186
Average Treatment Effect Using Radius Matching ...................................186
Average Treatsing Kernel Matching 187
Checking Robustness of Average Treatment Effect ..................................187
Further Exercises ........................................................................................188
Reference ....................................................................................................188
viiiContents
14. Double-Difference Method ...............................................................................189
Simplest Implementation: Simple Comparison Using “ttest” .................189
Regression Implementation ......................................................................190
Checking Robustness of DD with Fixed-Effects Regression ...................192
Applying the DD Method in Cross-Sectional Data .................................193
Taking into Account Initial Conditions ....................................................196
The DD Method Combined with Propensity Score Matching ................198
Notes ...........................................................................................................201
Reference ....................................................................................................201
15. Instrumental Variable Method .........................................................................203
IV Implementation Using the “ivreg” Command ....................................203
Testing for Endogeneity: OLS versus IV ...................................................205
IV Method for Binary Treatment: “treatreg” Command .........................206
IV with Fixed Effects: Cross-Sectional Estimates .....................................207
IV with Fix Panel Estimates ......................................................208
Note ............................................................................................................209
16. Regression Discontinuity Design ....................................................................211
Impact Estimation Using RD 211
Implementation of Sharp Discontinuity ..................................................212
I Fuzzy Discontinuity 214
Exercise .......................................................................................................216
Answers to Chapter Questions .................................................................................... 217
Appendix: Programs and .do Files for Chapter 12–16 Exercises ........................... 219
Index ......................................................................................................................... 231
Boxes
2.1 Case Study: PROGRESA (Oportunidades) in Mexico ................................................10
2.2 y: Assessing the Social Impact of Rural Energy Services in Nepal .............13
2.3 Case Study: The Indonesian Kecamatan Development Project .................................15
2.4 y: Monitoring the Nutritional Objectives of the FONCODES
Project in Peru ...............................................................................................................17
2.5 Case Study: Mixed Methods in Quantitative and Qualitative Approaches ...............19
2.6 y: An Example of an Ex Ante Evaluation ....................................................21
ix