Report from the Norwegian SILC web pilot 2019

The aim of the project was to explore how using web (CAWI) mode may contribute to cutting costs and increasing representativity of the data. Whereas the regular data collection for SILC is currently conducted by telephone only (CATI), the pilot was conducted by CAWI only. In the report, pilot results are compared to results from the regular CATI SILC from 2018 and 2019.

Chapter 1 gives an overview of the project and chapter 2 describes the pilot sample and questionnaire. The pilot consisted of three subsamples with differing characteristics in terms of knowledge of the survey and contact mode. For two subsamples the data collection was done by web only, but for one sample we did a short CATI interview and then switched to CAWI (multimode). The gross sample of the survey was approximately 5,600 selected respondents aged 16 or over. One subsample had participated previously (fourth wave CATI SILC 2018).

Chapter 3 reviews results from the data collection. Although the response rate for the pilot were higher than expected (45 per cent), it is clearly lower than in the regular CATI SILC. Moreover, the gross sample in the pilot only included individuals who were registered with an email address in a government register. As e-mail coverage is low for the elderly, the pilot included very few respondents over the age of 80. In terms of response bias, the young and persons with low education and low income were underrepresented in the pilot. These are groups that are underrepresented in the regular CATI SILC as well, but the bias is larger in the web sample. We found clear differences in bias and response rates depending on our contact strategies. The highest response rate and lowest bias was obtained for the pure web subsample who had not previously participated in the SILC survey.

Chapters 4 and 5 addresses questionnaire quality by looking at break-offs, the use of don’t know/refusals and by qualitative evaluations of questions. Despite an acceptable response rate, the rate of break-offs was higher than desired and clearly a larger problem than in CATI. Break-offs mainly occur when the questionnaire moves from personal questions to household mapping of work and housing cost. It is the youngest respondents who have the highest break-off rates, and the rate is also higher for respondents in households with more than one person, due to the questions about other household members. In addition to high break-off rates these groups also have poorer response quality when responding. Although the pilot included an option for young respondents to let other household members answer these questions, this did not lead to lower break-offs in the pilot.

Chapter 6 examines mode-effects. We find mode selection bias that may be difficult to adjust for using the current weighting procedure of the Norwegian SILC. Furthermore, we found evidence of mode measurement bias among the variables examined. The largest measurement bias was found for poverty indicators. Nevertheless, is seems that key SILC indicators such as AROPE are less affected.

Chapter 7 summarises findings in the project and gives recommendations for further work. We conclude that although mixed mode is a realistic option for SILC in the future, several measures should be taken before making this transition.