Quasi-experimental estimates of targeted learning support on student achievement

As part of the Ny GIV intiative, initiated by the Norwegian Ministry of Education and Research, students with weak academic performance midway through the 10th grade were offered intensive training in basic numeracy and literacy in the spring semesters 2011-2013. The intention has been to increase the progress through and completion of secondary education (VGO). This report is the second report from the ongoing impact evaluation, and presents preliminary results.

The purpose of the impact evaluation is to find causal effects of the intensive training, primarily whether it has increased completion of VGO. As the intensive training has been provided to students who completed primary school in the years 2011-2013 we have little data on completion of VGO and can say little about any effects on this outcome. We thus study effects on earlier outcomes, which previously have been shown to correlate with completion of the VGO, primarily grades from compulsory schooling and early measure of progress in VGO (completion of the first year, Vg1, and transition to the second year, Vg2). 

With the effect of intensive training we mean the difference between the actual results obtained and the results the students would have received in the absence of intensive training. Information about such effects provides a basis for evaluating the efficiency and prioritize between different policy measures. To learn about effects we must isolate these from other factors that have affected the results. We use two approaches to isolate effects: We compare the development in schools introducing intensive training at different times and we compare the results within Stavanger schools. 

The first comparison allows us to study the general effects of introducing intensive training. We have a large data set, consisting of several complete cohorts of students. As schools are introducing intensive training at different times, we can see whether changes in outcomes in schools introducing intensive training are different from changes in other schools. We find no evidence of such differences, thus no signs of effects of introducing intensive training, neither on marks nor early progress through VGO. There is some uncertainty in the estimates, but the vast data material makes this modest. We can thus rule out average effects on early measures of progress greater than about 1 percentage point. Approximately 10 percent of students participate in intensive training, but intensive training can have an effect also for students who do not participate. This makes it difficult to conclude about the effects on participants. The results do not indicate any effect for participants, and, under certain assumptions, we can rule out effects on these larger than about 5 percentage points. 

The second comparison is based on a sharp delineation of the target group in Stavanger schools, which allows us to compare initially almost identical students who are participating / not participating in intensive training. This comparison allows us to study the effect of changing the number of pupils participating. Early progress among Stavanger students in the target group is about 9 percentage points higher than among similar students not eligible to participate. A comparison with the corresponding difference in Bergen may indicate that this is an effect of intensive training. Since the target group represents 10 percent of students, this corresponds to an effect of almost 1 percentage point on average early progress through VGO among all students. 

When we find evidence of an effect in Stavanger, but not in general, it may be because intensive training is organized and conducted differently in Stavanger than elsewhere in the country. An increase in early throughput of 1 percentage point, from a level of 80-85 percent, is less than the target for Ny GIV, but still large enough that a similar effect on completion of VGO can be economically significant. We find no sign of any effects on marks or progress through higher levels of upper secondary. For progress at higher levels and completion of VGO the data is yet very scarce, not allowing us to draw clear conclusions. Any effects on this would be important in the final assessment of the intensive training.