First results from the program evaluation of the intensive training in Overgangsprosjeketet, Ny GIV

Through Overgangsprosjektet, which is a part of the larger initiative Ny GIV, students with weak academic performance midway through the 10th grade have since spring 2011 been offered customized intensive training. The aim has been to increase progress through and completion of upper secondary education. In this report we present the first results from the ongoing impact evaluation of intensive training.

The purpose of an impact evaluation is to find causal effects, i.e. the extent to which relevant results are different from what they would have been if the project were not implemented. We study whether progress in upper secondary education, academic performance in lower and upper secondary and motivation has changed for students who have participated in the intensive training or who attend schools where some students have done so, compared with what the outcomes would have been in the absence of the initiative.

These effects can not be measured directly, because we can not observe what would have happened in the absence of the initiative. However, the program has two characteristics that contribute to make it possible to evaluate: An explicit definition of the students in each school who make up the target group, and a gradual phasing in of different schools in three waves. Explicit delimitation in schools means that we can find almost identical students that respectively participate / not participate in intensive training, and compare later outcomes for these. The gradual phasing allows us to compare the evolution of the results in schools of different waves, which introduce intensive training at different times.

We find no clear effects of the intensive training. This applies both for comparisons of students within wave one schools and for comparisons of wave one schools with other schools that adopt intensive training later. We find that for some outcome measures there are differences between students/schools with intensive training and other students / schools that are non-zero in a statistical sense. However, as we study many different outcomes and use different analysis techniques, we must expect to find significant differences as a result of random variation, even if there are no effects. Which measures that differ vary with the technique applied. Moreover, in several cases where there are differences in results between students that participate in intensive training and others there are also evidence of other differences than the intensive training.

 

Since the first students receiving intensive training still are in secondary education it is too early to conclude whether there are effects on completion. The early outcome measures we study are strongly correlated with eventual completion of upper secondary. It is still possible that intensive training will have an effect on the share completing, despite the fact that we so far have not found any effects on early outcomes. We can not study this until 2016 and onwards, when at least four to five years have passed since the participants left lower secondary education.

We show that the selection of schools and pupils is based on widely varying criteria. This greatly complicates our effort to find effects of the intensive training. A majority of the schools selected students in such a way that it is impossible to find students who are attending / not attending, but otherwise comparable. Schools and local authorities in the different waves are systematically different, and it is difficult to fully take into account these differences.

This means that there will be some uncertainty as to whether our estimates capture differences that are independent of intensive training, and thus fail to identify the effect of the intensive training. Furthermore, our estimates are based on small samples or on groups where students receiving intensive training make up a small share. This reduces the statistical precision of our estimates. We can not on the basis of this analysis rule out either positive or negative effects of intensive training.