Modes of Data Collection

A self-completion future

One of the main aims of the ESS is “to chart stability and change in social structure, conditions and attitudes in Europe”. To make meaningful comparisons of survey estimates of attitudes across countries, it is necessary to implement high standards of scientific rigor (Jowell et al, 2007).

In pursuit of maximum data comparability, essential survey conditions are implemented as similarly as possible across ESS participating countries. For this reason, all countries up until Round 10 (2020-22) of the survey were required to conduct face-to-face data collection.

However, the Coronavirus pandemic made face-to-face data collection difficult to implement in all countries during Round 10. As such, special dispensation was given for self-completion modes to be implemented in this round only. A total of nine countries switched to a self-completion approach, while 22 countries were able to use the usual face-to-face approach.

In May 2022, the ESS General Assembly endorsed a plan to transition data collection from face-to-face interviews to a ‘web first self-completion’ design. Following a series of reviews and consultations with internal key stakeholders, the Core Scientific Team (CST) recommended that the change take place due to diminishing face-to-face interviewer capacity in some countries and improving data quality from self-completion methods.

Round 11 (2023/24) is therefore the final round in which face-to-face interviews will be conducted in the vast majority of countries.

Round 12 (2025/26) fieldwork will be conducted in all participating countries in both modes. In each country, half the sample will be interviewed via face-to-face interviews, with the other half receiving web or paper questionnaires.

This change is being implemented following a recommendation from the ESS Methods Advisory Board (MAB). Countries will conduct Round 12 face-to-face and self-completion fieldwork simultaneously to help better understand data collection mode effects.

Round 13 (2027/28) will see data collected solely through web and paper self-completion surveys in all participating countries.

To assist in this transition, the CST took advice from four experts - Andrew Cleary (Ipsos International Social Research), Peter Lugtig (Utrecht University), Laura Wilson (Office for National Statistics, ONS) and Ranjit Singh (GESIS - Leibniz Institute for the Social Sciences).

Background

The ESS has always investigated important methodological issues concerning survey data quality in a cross-national context. A programme of methodological research has been built into the project, in order to investigate major issues in surveys such as non-response, the reliability and validity of questions and the feasibility of mixing modes of data collection.

This research into mixed modes provided information on:

• whether mixed-mode data collection should be allowed;
• which modes of data collection should be allowed;
• within which kinds of overall survey design mixed modes could be employed.

The following issues were assessed:

• coverage and response rates that can likely be achieved with different modes and mode combinations;
• likely differential error between modes (particularly non-response error and measurement differences) and its causes.

Research projects

Between 2003 and 2012, six studies were undertaken. Of these, three focused solely on assessing the effect of mode on measurement (studies 1, 2, and 3). The other three studies investigated the feasibility and practical challenges of implementing the ESS using a different mode, where all sources of error were affected by the mode design (studies 4, 5, and 6).

• Studies 1 and 2, undertaken in collaboration with Gallup Europe, focused on measurement equivalence. Study 1, conducted in Hungary in 2003, involved a lab test with random allocation of respondents to different modes of data collection (face-to-face, telephone, Internet and paper self-completion).
• Study 2, conducted in Budapest and Lisbon in 2005, was an experiment designed to investigate the causes of measurement differences between face-to-face and telephone interviewing.
• Study 3 was conducted in the UK in 2010, by asking ESS respondents to participate in a short follow-up web survey. The goal was to evaluate measurement differences between face-to-face and web data collection, using a within respondent re-interview design.
• Study 4 assessed the feasibility of using telephone interviews in the ESS, focusing on the effect of varying interview length on respondents' willingness to participate in the survey. The study was carried out in 2006 in five countries: Cyprus, Germany, Hungary, Poland, and Switzerland.
• Studies 5 and 6 tried to evaluate the effects of mixed-mode designs on response rates, representativeness of samples, survey costs, and data quality. Study 5 was conducted in the Netherlands in 2008, parallel to Round 4 of the ESS, and tested two different mixed-mode (web, telephone, face-to-face) data collection designs: a ‘concurrent mode choice design’, and a ‘sequential mode choice design’.
• Study 6 was conducted in Estonia, Sweden and the UK in 2012. Countries were allowed to select the ideal mixed-mode design given the country’s survey environment. Estonia and the UK focused on a web and face-to-face mix, whereas Sweden tested a telephone and face-to-face design.

A summary of the results of the six experiments can be found in a book chapter linked below.

Building on the experience of these studies, ESS ERIC coordinated Work Package 7 of the Synergies for Europe's Research Infrastructures in the Social Sciences (SERISS) project. This investigated opportunities and challenges of high quality online data collection. This established a probability sample based web panel in three countries. WP7 aimed to:

• evaluate the feasibility of establishing the first cross-national probability-based web panel using the achieved sample from existing cross-sectional surveys;
• foreground a methodology for building new and efficient web-based survey infrastructures for Europe based on state of the art procedures and technology;
• develop a blueprint for a comparative probability-based web survey.

More information on this project can be found on our CRONOS page.

Related content

Data on this topic

ESS ERIC reserves the right to make the relevant data sets available upon request.

Publications on this topic