Impact evaluation

Sujets

• Économétrie

Informations

• Publié par: mtoledan
• Publié le: 14 août 2011
• Nombre de lectures: 180
• Langue: English

Extrait

Impact evaluation

Frequently interventions in economics take the form of a change in the economic environment of a given population or of a program proposed to a given population with specific needs. Programs mey be either mandatory or compulsory. Here are some examples : – Counseling program for the unemployed with risk of long term unem- ployment. Individuals identified as at risk of long term unemployment when falling onto unemployment are proposed to enter a counseling program that assist them in their search of a new job for a given dura- tion (six months). They may accept or refuse to join the program. – Welfare reforms. The welfare reforms of the 1990s in the US were cha- racterized by one or more of three core components : broad and tough work requirements, financial incentives to make work pay, and time limits on cash benefit receipt. The Connecticut’s Jobs First program was launched in 1996 and was one of the first statewide reform initia- tives to include all three, including, at 21 months, one of the shortest state-imposed welfare time limits in the country. – Conditional cash transfer. To solve the problem of low school atten- dance it is possible to pay children or their family to attend school. The transfer is done only when the children indeed attend school. This type of program is a frequent program to increase education in deve- lopment countries. – Class size. One issue is the link between class size and the performance

1

of children. One idea is that when reducing the class size, children can improve their performance. – Micro credit. Some people do not have access to credit, either because there is no bank or because they do not fulfill the condition to open an account and get a credit, for example because they are not rich enough. This may prevent these populations to make profitable investment. Micro-credit is a way to offer these people an access to credit. The principle is to form a lending group that bears the responsibility in case of default of one of its members. – Counseling Apprentices. High drop out rates in apprenticeship is a core problem, as it means exiting school without any diploma. Programs try to solve the problem by counseling apprentice in the choice of entre- preneurs and coaching then their relation with it. Many economic or social intervention either in development economics developed countries are of this type. One key point is thatwe know little about how these programs perform and work in practice. There is a broad variety of program intervention but our knowledge of what works and what do not works is still very limited. The purpose of these lessons is to show that it is possible to precisely define and measure the causal effect of these programs. We will see however that this is difficult and that there is one specific and fundamental problem of evaluation which is identification. We will see various methods to solve this identification problem, and what are their limits. One specific and interesting

2

tool that can be use dis is that ofRandomized Field Experiments. We will see how to implement these methods and what can be learned from them.

1 The Rubin Causal Model

We consider the case of a program that is proposed to an identified popu- lation. Part of the population will eventually enter the program. The program is frequently referred as thetreatment. We denoteT= 1 program participa- tion. We are interested in an outcome variabley. For example, we may be interested in consumption for a micro-credit program, or with the employ- ment status one year after the date of the beginning of the program for a counseling program. The Rubin Causal model considers that there are two random potential outputs associated with the program :y 1 andy 0 .y 1 is the output when individuals enter the program,y 0 is the output when individuals stay out. These two outputs exist fore each individual independently of the decision to enter the program. The Causal effect is the difference of these two output variables at the individual level. It is the contrast between the output when participating and when not participating.

c=y 1 −y 0

It has two main characteristics : 1. It is defined at the individual level. There is a distribution of the causal

3

effect of program participation in the population 2. It is not observable. For treated, we observe the output when treated (y 1 ) but we do not observe the output when non treated (y 0 ). For non treated, we observe the output when non treated (y 0 ) but we do not observe the output when treated (y 0 ). Observation for the output is

y=y 0 (1−T) +y 1 T

The last point is the fundamental problem of program evaluation : we miss one information to reconstruct the causal effect of the program. We have usually information on treated and non treated, all evaluation methods amount to invent the missing output for treated for example from observed output for the non treated. Relation with standard econometric models We may want to write the previous equation under the form of a regression equation

y i =a i +c i T i

Herea i =y 0i is the output when non treated andc i =y 1i −y 0i is the treat- ment effect We see that there are indeed two problems with this model. The first is that as is usual in econometrics there may be correlation between the treatment variableT i and the individual effecta i : individuals that have bad outcomes when non treated might be induced to enter the program. This is what we have in presence of heterogeneity, an issue that we have already 4

deal with. But in the present case there is another problem : the parameter is itself a random variable. Probably it is also correlated with the Treatment variable. This is why this model is called the model of essential heteroge- neity, because both the intercept and the coefficient are heterogeneous and potentially correlated with the explanatory variable. There are two main parameters we can be interested in : –The Average Treatment Effect (Δ AT E ). This is the mean value of the treatment effect :

Δ AT E =E(y 1 −y 0 )

–The treatment on the treated (Δ T T ). This is the mean value of the treatment effect over the population that indeed received the treat- ment :

Δ T T =E(y 1 −y 0 |T= 1)

These two parameters are different as long as the treatment is hetero- geneous and correlated with program participation. They give distinct in- formation. The TT parameter identifies the effect of the policy as it was implemented, while the ATE parameter identifies the mean effect of the pro- gram if it would be extended and made mandatory to the whole population.

5

2 Selectivity bias

What about OLS

OLS on the simple model

y=a+bT+u

is a natural idea. The OLS estimator of the parameterbwould bey T=1 −y T=0 and would tend toE(y|T= 1)−E(y|T= 0). We can see clearly that

ˆ limb=E(y|T= 1)−E(y|T= 0) =E(y 1 |T= 1)−E(y 0 |T= 0) = (E(y 1 |T= 1)−E(y 0 |T= 1)) + (E(y 0 |T= 1)−E(y 0 |T= 0))

As can be seen this is the sum of two components that areconfounded: the effect we would like to measure :

E(y 1 −y 0 |T= 1) =E(y 1 |T= 1)−E(y 0 |T= 1)

which is the TT parameter, and a population effect :

E(y 0 |T= 1)−E(y 0 |T= 0)

This population effect reflects the facts that the two populations of indi-

6

viduals that enter the program and that do not enter the program are not the same. Entering the program reveals characteristics that are associated to the performance. For example, If you consider class size. Assume that there is a program of reduction of class size. There is one regular class size : 25 and a reduced class size 15. When the program is developed, it is likely that the small class will be introduced in areas where students have already pro- blems,... Similarly for micro-credits individuals that will take a micro-credit are individuals that at least have projects. May be if they do not find a way to finance the project they will found another way to implement it, or to implement another one, because they are likely to have entrepreneur skills. In these two cases, it is clear that the population of treated and non treated do not have the same characteristics, and that the difference may be related to differences in the outcomes without the program. This type of bias is known asSelectivity Bias. What we know about these bias is that they are usually very important in practice : heteroge- neity among individuals is very important and strongly related with their decisions. The program that we implement have usually a small impact, so that Selectivity Bias is usually very important compared with the treatment parameter. This problem is the key problem of evaluation.

7

3 Randomized experiments

Key condition for the OLS estimator to be consistent :

y 0 , y 1 ⊥T

In this casey 1 −y 0 ⊥T, thus Δ T T =E(y−1−y 0 |T= 1) =E(y−1−y 0 ) = Δ AT E This is usually not true except in some specific circumstances. One im- portant and specific case isRandomized Experiments. In this case individuals are assigned to treatment and control groups. It is a case where it is necessary to organize the experimental design and the collect of the data. It is not just : take the data and apply an estimation method. It is necessary to assign individuals and monitor the evaluation in order to be sure that individuals and the observation obtained on them prior and after the program and the assignment are not correlated with the assignment. Key issues when implementing a Randomized Evaluation – How to implement these experiments in practice : It is possible to randomize at the level of individuals. There are various way to implement these methods. For example you can use a lottery. Frequently programs offer fewer places than the number of potential beneficiaries. How to allocate these places : random allocation is the best way in practice.

8

It is also possible to implement at the level of groups of individuals. One example is micro credit. Randomization can be organized at the village level rather at the individual level. Another way to implement the program is the so-called phase-in design. Frequently programs are developed over several years. In this case, it is possible to randomly choose the areas where the program will be developed first. The other areas are used as a random control group. Example : fruit tree productivity program. Program provide technical assistance to fruits producers to help them in both the production of the fruits, the increase of its quality and their commercialization. The program is to be developed over five years on 500 relevant perimeters : 100 perimeters each year. Randomized evaluation of this program can be implemented by randomly choosing the order of the implementation of the program. Another way to implement a randomized evaluation is to consider an encouragement design. It corresponds to the case where the treatment is not mandatory but compulsory. The population is divided in two sub- populations and the treatment is proposed to one of the two populations with some encouragement to enter the program. For example, there may be a subsidy in one of the two groups. This random assignment design is interesting as it is softer than a direct exclusion from treatment. It may solve a lot of potential problem of implementation. – Is it ethical. This is a key issue that need to be assessed extensively.

9

There are some principles : inform consent ; compensation for treated or control ; representative population. Different aspects : are individuals randomized out excluded from treatment forever or have they chance to enter the program later ? In some cases the implementation do not hurt individuals : for example when the program offers only few places for a large population or when the program needs time to be developed on the whole population. Frequently a convenient introduction into the operational development or implementation of the program. – Internal validity. Sometime the final objectivey 0 , y 1 ⊥Tis not reached although individual were randomly assigned to treatment and control. This may arise when there are non response bias : if individual in the control group have smaller incentives to respond to surveys, then although people have been randomly assigned, it is not true that the independence property for the observed outputs. Contamination bias : although individuals have been randomly assigned to a control and treatment group, they may change to avoid the treatment when they are in the treatment group or to get the treatment if they are in the control group. That was a problem for the Star project, a program devoted to test the effect of small class size : some individual move so that their children can enter a school with small size class. – External Validity. Is tit possible to generalize what has been lear- ned from the experiment to other populations, other periods of time. Usually the program is implemented during a period of time. What is

10

measured is the change in the situation of individuals from one date to another and then the experiment is ended. What is measured is the effect of the policy during a given amount of time. But what about the effect of the policy after. Sometime programs need time to reveal all their effects. Usually the time window for Randomized evaluation is small : a few years. – Power and sample size. How to design an experiment ? That is how many treated and control should be considered ? This is related to the power of the experiment. An experiment may have power or not. Having power means that the experiment will be able to detect even small effect of the program. More precisely, it will have power if it is likely to reject the assumption that the effect is zero even for small values of the true treatment effect. Assessing the power of an experiment is related to a pre-computation of the standard error of the estimator. When implementing a Randomized Evaluation the estimator is simply the difference of the mean over the treated and non treated population.

T=1T Δ ˆ =y−y =0

This estimator has a variance that can be computed as

V ˆ Δ(=)σ Y 2 ( N + 1 N )= 1V(y N ) P 1 (1−P) ) T C wherePis the proportion of individuals assigned to the treatment

11