Impact evaluation is an assessment of how the intervention being evaluated affects outcomes, whether these effects are intended or unintended. The proper analysis of impact requires a counterfactual of what those outcomes would have been in the absence of the intervention.
Quasi-experimental research designs test causal hypotheses. In Quasi-Experimental design the programme or policy is viewed as an ‘intervention’ in which a treatment is tested for how well it achieves its objectives, as measured by a pre-specified set of indicators.
There are different techniques of creating a comparison group, for example, difference in difference (DiD), propensity match scoring (PMS), regression discontinuity design (RDD), interrupted time series (ITS) etc.
Difference in difference technique (DiD)
Difference-in-differences (DID), also known as the ‘double difference’ method, compares the changes in outcome over time between treatment and comparison groups to estimate impact.
DID gives a stronger impact estimate than single difference, which only compares the difference in outcomes between treatment and comparison groups following the intervention (at t+1). Applying the DID method removes the difference in the outcome between treatment and comparison groups at the baseline.
DID usually is used to estimate the treatment effect on the treated (causal effect in the exposed), although with stronger assumptions the technique can be used to estimate the Average Treatment Effect (ATE) or the causal effect in the population.
In order to estimate any causal effect, DID estimation also requires that:
Intervention unrelated to outcome at baseline (allocation of intervention was not determined by outcome)
Treatment/intervention and control groups have Parallel Trends in outcome
Composition of intervention and comparison groups is stable for repeated cross-sectional design
No spillover effects
DID is usually implemented as an interaction term between time and treatment group dummy variables in a regression model.
Y= β0 + β1*[Time] + β2*[Intervention] + β3*[Time*Intervention] + β4*[Covariates]+ε