Cross-Regional Sequential Difference in Difference (CR-SEQDD): An Empirical Approach for Evaluating EU Thematic-Objective Interventions with Regional Data Aggregated at the National Level

Causal impact evaluations of EU Operational-Programme (OP) interventions aggregated at the national level are often used as a tool to inform policy makers, and the public opinion in general, about the overall contribution of the European Funds in achieving desirable results within the various thematic objective (TO) areas. Implementing these types of TO evaluations, however, is a very challenging task because, in the case of data aggregated at the national level, very often, no comparable “untreated” units of observation can be found for the analysis, due to the unique features of the EU member states and small sample sizes. For these reasons, standard quasi-experimental empirical methods can be very rarely implemented and identifying casual effects of the OP interventions without very strict and limiting casual-identification assumptions is merely impossible. Indeed, it should be understood that, in general terms, there is a clear trade-off between rigorous internal validity of casual-effect evaluations and the level of aggregation of OP interventions and result indicators: the more the analysis is focused on broad TOs at the national level, the more limited the internal validity tend to be; the more the focus is on specific interventions at a micro-level, the stronger the internal validity tend to be. This technical note discusses an empirical approach, called “Cross-Regional Sequential Difference-in-Difference” (CR-SEQDD) that exploits the regional variations in the intensities of the OP interventions, pertaining to a same TO, in order to estimate a dose response functions that, under very strict and limiting causal-identification assumptions, can be subsequently used to establish what part of a change in the nationally-recorded result indicator (Y) of interest is likely to be caused by the OP interventions and what part is instead due to a counterfactual spontaneous change. This is done by means of pairwise sequential difference-in-difference (DD) comparisons across regions with different intensities of the OP interventions. These DD estimations are then plotted against the related cross-regional differences in the intensities of the OP interventions and a fitting dose-response function is estimated to subsequently infer about the casual effect of the nationally-aggregated set of OP interventions considered in the analysis. Under sufficient data-availability scenarios, compared to different evaluation options, such as expert opinions or meta-analyses, the CR-SEQDD estimation procedure has the advantage of allowing a more consistent comparisons of the findings across different thematic areas, programming periods and EU countries.