Effects of an intervention promoting targeted mathematics teaching in secondary school

The purpose of this report is to describe and evaluate an intervention implemented in public lower and upper secondary schools in Oslo. The intervention targeted students with poor mathematics skills in 8th grade and the first year of upper secondary education (Vg1) and aimed to enable more students to complete upper secondary education (VGO) by improving their math skills. The intervention consisted of: 1) didactics training of teachers and 2) adapted instruction instead of regular instruction for students during two periods of 4-6 weeks (one in autumn and one in spring). In lower secondary school, adapted instruction took place in small groups, in upper secondary schools in regular groups.

The intervention was implemented as a randomized experiment, in which 24 of 48 lower secondary schools in Oslo and 9 of 17 relevant upper secondary schools were randomly selected to participate. We evaluate the intervention by comparing results in treatment and control schools. We are not yet able to study the completion of VGO, and therefore focus on the results on the standardized national test in numeracy in 9th grade (NP9) and the Oslo test in mathematics at Vg1.

We find that average NP9 in the target group in lower secondary school increases by 0.02-0.04 percent of a standard deviation but cannot conclude that this is an effect of the intervention. Most effect estimates are not statistically significant. However, we do find that the share of target students at the lowest mastery level is reduced by 2 percentage points, from an initial level of about 10 percent. This difference is significant at the 10 percent level. We do not find any signs of effects on students outside the target group. Based on the effect estimates and previous research, we estimate the economic benefit of the intervention. We cannot conclude that the intervention is cost-effective. At the same time, there is substantial uncertainty in the estimates, and we cannot exclude effects that make the intervention cost-effective.

In upper secondary, we find no effect on the students’ mathematics skills as measured by the Oslo test. However, we find evidence of increased completion of Vg1 for the first two cohorts (where we have data for completion of Vg1). We cannot yet conclude whether increased completion is an effect of the intervention. If in the long term, there is a corresponding effect on the completion of VGO, the intervention in upper secondary will have been cost-effective.

We also conducted a qualitative evaluation, based on interviews and surveys. This evaluation shows a change over time in the attitude towards the intervention. Participating teachers were initially positive to the content of the intervention but sceptical about features of the implementation. This changed to a more generally positive attitude. In the final year, the intervention became a part of regular teaching, and the schools' focus shifted to new projects. Teachers' responses about their teaching and observations of lessons indicate that the intervention had an impact on teaching. However, teaching during the intervention is not fully in line with the didactic principles of the intervention, nor are these principles exclusive to the treatment schools. Organizational conditions may have limited the effect of the intervention. There were challenges with turnover among teachers, and possibly with selecting the students who would benefit most from the intervention.

In summary, the intervention has largely succeeded in connecting with and motivating the treatment school teachers and is seen as relevant by these. On the other hand, we cannot conclude that the intervention has succeeded in improving the students’ results, although there are some signs of this. In the longer term, it will be possible to study any effects on actual completion of VGO.