Modes of Data Collection

Background and goals

One of the main aims of the ESS is “to chart stability and change in social structure, conditions and attitudes in Europe”. To make meaningful comparisons of survey estimates of attitudes across countries, it is necessary to implement high standards of scientific rigor (Jowell et al, 2007).

In pursuit of maximum data comparability, essential survey conditions are implemented as similarly as possible across ESS participating countries. For this reason, all countries up until Round 10 (2020-22) have been required to conduct face-to-face data collection.

However, the Coronavirus pandemic made face-to-face data collection difficult to implement in all countries during Round 10 of the survey. As such, special dispension was given for self-completion modes to be implemented in this round only.

In May 2022, the ESS General Assembly endorsed a plan to transition data collection from face-to-face interviews to a ‘web first self-completion’ design. Following a series of reviews and consultations with internal key stakeholders, the General Assembly approved the recommendation from the Core Scientific Team to progress with collecting data primarily through web and paper self-completion surveys.

Details on precisely how and when the change will take place will be decided in early 2023. In the meantime, ESS Round 11 fieldwork (expected to begin in February 2023) will use the existing face-to-face data collection design.

The ESS has always investigated important methodological issues concerning survey data quality in a cross-national context. A programme of methodological research has been built into the project, in order to investigate major issues in surveys such as non-response, the reliability and validity of questions and the feasibility of mixing modes of data collection.

The ESS research into mixed modes provides information that will help to inform decisions regarding:

• whether mixed-mode data collection should be allowed on future rounds of the ESS;
• which modes of data collection should be allowed;
• within which kinds of overall survey design mixed modes could be employed.

The following issues are being assessed:

• coverage and response rates that can likely be achieved with different modes and mode combinations;
• likely differential error between modes (particularly non-response error and measurement differences) and its causes.

Research projects

To date, six studies have been carried out. Of these, three focused solely on assessing the effect of mode on measurement (studies 1, 2, and 3). The other three studies investigated the feasibility and practical challenges of implementing the ESS using a different mode, where all sources of error were affected by the mode design (studies 4, 5, and 6).

• Studies 1 and 2, undertaken in collaboration with Gallup Europe, focused on measurement equivalence. Study 1, conducted in Hungary in 2003, involved a lab test with random allocation of respondents to different modes of data collection (face-to-face, telephone, Internet and paper self-completion).
• Study 2, conducted in Budapest and Lisbon in 2005, was an experiment designed to investigate the causes of measurement differences between face-to-face and telephone interviewing.
• Study 3 was conducted in the UK in 2010, by asking ESS respondents to participate in a short follow-up web survey. The goal was to evaluate measurement differences between face-to-face and web data collection, using a within respondent re-interview design.
• Study 4 assessed the feasibility of using telephone interviews in the ESS, focusing on the effect of varying interview length on respondents' willingness to participate in the survey. The study was carried out in 2006 in five countries: Cyprus, Germany, Hungary, Poland, and Switzerland.
• Studies 5 and 6 tried to evaluate the effects of mixed-mode designs on response rates, representativeness of samples, survey costs, and data quality. Study 5 was conducted in the Netherlands in 2008, parallel to Round 4 of the ESS, and tested two different mixed-mode (web, telephone, face-to-face) data collection designs: a ‘concurrent mode choice design’, and a ‘sequential mode choice design’.
• Study 6 was conducted in Estonia, Sweden and the UK in 2012. Countries were allowed to select the ideal mixed-mode design given the country’s survey environment. Estonia and the UK focused on a web and face-to-face mix, whereas Sweden tested a telephone and face-to-face design.

A summary of the results of the six experiments can be found in the book chapter listed in the sidebar.

Building on the experience of these studies, ESS ERIC coordinated Work Package 7 of the SERISS project. The Work Package investigated opportunities and challenges of high quality online data collection. This established a probability sample based web panel in three countries. WP7 aimed to:

• evaluate the feasibility of establishing the first cross-national probability-based web panel using the achieved sample from existing cross-sectional surveys;
• foreground a methodology for building new and efficient web-based survey infrastructures for Europe based on state of the art procedures and technology;
• develop a blueprint for a comparative probability-based web survey.

More information on this project can be found on the CRONOS page.