Modes of Data Collection

The ESS move to Self-Completion data collection

Background

One of the main aims of the ESS is “to chart stability and change in social structure, conditions and attitudes in Europe”. To make meaningful comparisons of survey estimates of attitudes across countries, it is necessary to implement high standards of scientific rigour (Jowell et al, 2007). In pursuit of maximum data comparability, essential survey conditions are implemented as similarly as possible across ESS participating countries. For this reason, all countries up until Round 10 (2020-22) of the survey were required to conduct face-to-face data collection. 

Even though this decision was made to use this mode, the ESS investigated whether other modes of data collection could be used instead or alongside. Until recently, the ESS had always rejected this on the basis that face-to-face interviews provided the best data quality across all of the countries. 

However, the Coronavirus pandemic made face-to-face data collection difficult to implement in all countries during Round 10. As such, special dispensation was given for self-completion modes to be implemented in this round only. A total of nine countries switched to a self-completion approach, while 22 countries were able to use the usual face-to-face approach. 

In addition to the challenges that face-to-face data collection faced during the pandemic, broader concerns of its sustainability as a high quality survey mode have grown over recent years. For instance, the ESS has seen a steady decline in response rates over the years. Adverse effects of interviewer behaviour on measurement and representation have also been documented (see here).

Prior to and alongside the exceptional implementation of a self-completion approach as official Round 10 fieldwork, the Core Scientific Team (CST) also undertook experimental self-completion data collection activities in order to develop a self-completion approach for the ESS. These experiments and Round 10 experience showed that reasonable response rates could be achieved with comparable sample compositions to face-to-face and acceptable data quality.

In May 2022, the ESS General Assembly endorsed a plan to transition data collection from face-to-face interviews to a ‘self-completion’ design. This followed a recommendation to switch modes by an ESS commissioned strategic review. The CST recommended that the change take place due to diminishing face-to-face interviewer capacity and quality in some countries and improving data quality from self-completion methods.

It has since been decided by the CST, after a recommendation by the Methods Advisory Board (MAB) and Scientific Advisory Board (SAB), that Round 12 (2025/26) fieldwork will be conducted in all participating countries using both approaches. In each country, a random half of the effective sample will be interviewed via face-to-face interviews, with the other half completing web or paper self-administered questionnaires. Round 13 (2027/28) will see data collected solely through web and paper self-completion surveys in all participating countries.

Development of the self-completion approach

Self-completion data collection activities

The ESS has been involved in various research endeavours to develop the best self-completion approach possible across Europe. Self-completion data collection undertaken outside of official ESS rounds has been pivotal for the ESS’s self-completion future. A summary report of all results of these data collection activities is available here. Here is an overview of the implementation of these surveys:

• Between 2003 and 2012, six studies were undertaken that focused on assessing the effect of mode on measurement and investigated the feasibility and practical challenges of implementing the ESS using a different mode (Villar & Fitzgerald, 2017).
• A three-country postal push-to-web survey was undertaken in Austria, Serbia, and Hungary in 2020. This was the first trial of the web and paper self-completion approach the ESS administers today, albeit using a shortened ESS questionnaire. More information is available in a survey methods webinar from April 2021.
• An additional self-completion experiment was conducted in Austria in 2021 that tested a much shortened, slightly shortened, and full ESS questionnaire. It was unclear if it would be possible to adapt the full face-to-face ESS questionnaire to a self-completion web and paper context. Fielding a long ESS questionnaire in self-completion proved to be feasible without suffering a substantial reduction in response rates. This study also tested different incentive schemes. More information on the results of this experiment is available in this presentation from the 5th International ESS Conference. 
• Alongside Round 10 face-to-face fieldwork, two parallel self-completion surveys were administered in Great Britain and Finland. The Great Britain experiment tested different incentive schemes. Details on the results of the survey in Great Britain and its microdata are available on the UKDS website, and the data for the Finland experiment are available on request from the ESS Data Portal. 
• A self-completion survey was implemented in Hungary in 2022 comparable in size to Hungary’s Round 10 face-to-face fieldwork. This survey administered the full ESS questionnaire with the ESS’s self-completion methodological approach and is available for downloading on Hungary’s Round 10 country documentation page on the ESS Data Portal.
• A self-completion survey was implemented in France in 2024 in parallel to their Round 11 face-to-face survey. The data of this self-completion experiment will soon be published by Sciences Po.
• Small-scale self-completion feasibility studies were conducted in Lithuania and Slovakia in 2023 and 2024. The data from these studies will be available on request from the ESS Data Portal in the near future.
• Czechia conducted their Round 11 fieldwork using the ESS’s self-completion approach. Data collection there concluded in August 2024, and data will be released alongside Round 11 face-to-face data as a country-specific file.
• The ESS12 Pilot was conducted using the ESS’s self-completion approach in June - July 2024 in Ireland and Bulgaria. These data will be available on request from the ESS Data Portal in the near future.

Self-completion survey results

Overall, the ESS has implemented a self-completion survey, either as official or supplementary data collection, in 18 of the 28 ESS member countries and one observer country. Of the 11 countries that have not conducted a self-completion survey using an ESS questionnaire, the CST has determined that the implementation of the ESS’s self-completion approach will be feasible in 8 of them – Belgium, Croatia, Estonia, Iceland, the Netherlands, Norway, Slovenia, and Switzerland – based on the experience of comparable surveys in those countries that have implemented a self-completion approach in the past. For the three remaining countries – Italy, Montenegro, and Portugal – the CST is working with NC teams to undertake self-completion feasibility studies prior to their implementation of self-completion approaches at Round 12.

Where the ESS has implemented self-completion surveys, greater homogeneity of response rates has been observed as compared to face-to-face. Most countries achieve response rates between 30 and 40 percent. In the past, face-to-face response rates have varied widely across Europe, ranging from percentages in the low 20s to low 70s.

In general, the ESS’s self-completion approach tends to draw in younger, more educated populations than face-to-face. The Sampling and Weighting Expert Panel (SWEP) produced a report on the effect of switching modes at Round 10 on sample composition. 

Additional research in self-completion methodology

To assist in the transition to self-completion, the CST took advice from four appointed special advisors - Andrew Cleary (Ipsos International Social Research), Peter Lugtig (Utrecht University), Laura Wilson (Office for National Statistics, ONS) and Ranjit Singh (GESIS - Leibniz Institute for the Social Sciences).

• Andrew Cleary produced a report on the use of fieldworker-assisted self-completion data collection (knock-to-nudge), with a particular focus on the implementation of a fieldworker follow-up phase on nonresponding sample units.
• Peter Lugtig produced a report on mode effects with analysis of Round 10 self-completion data and data from the Round 10 parallel self-completion experiments conducted in Finland and Great Britain. This report will be available on this webpage in the near future.
• To assess the impact of potential response scale changes that may be considered when switching to self-completion, Ranjit Singh analysed data from relevant ESS MTMM experiments in previous rounds.
• Laura Wilson worked with CST members in redesigning invitation letters, the web-survey landing page, and the web and paper questionnaires with a respondent-driven focus. A redesigned, mobile-first web questionnaire based on this feedback was employed during the ESS12 Pilot and will be used for Round 12.

The CST has commissioned two user testing studies on the web and paper self-completion questionnaires. A report on the outcomes of user testing for the paper questionnaire is available here, whereas user testing of the web questionnaire is currently ongoing. A report on web user testing will be published on this webpage in the coming months.

Publications on this topic