Modes of Data Collection
Background and goals
One of the main aims of the ESS is “to chart stability and change in social structure, conditions and attitudes in Europe”. To make meaningful comparisons of survey estimates of attitudes across countries, it is necessary to implement high standards of scientific rigor (Jowell et al, 2007). In pursuit of maximum data comparability, essential survey conditions are implemented as similarly as possible across ESS participating countries. For this reason, all ESS countries are required to conduct face-to-face data collection.
At the same time, the ESS presents an outstanding opportunity to investigate important methodological issues concerning survey data quality in a cross-national context. A programme of methodological research has been built into the project, in order to investigate major issues in surveys such as non-response, the reliability and validity of questions and the feasibility of mixing modes of data collection.
The ESS research into mixed modes aims to provide information that will help to inform decisions regarding:
- whether mixed-mode data collection should be allowed on future rounds of the ESS;
- which modes of data collection should be allowed;
- within which kinds of overall survey design mixed modes could be employed.
The following issues are being assessed:
- coverage and response rates that can likely be achieved with different modes and mode combinations;
- likely differential error between modes (particularly non-response error and measurement differences) and its causes.
To date, six studies have been carried out. Of these, three focused solely on assessing the effect of mode on measurement (studies 1, 2, and 3). The other three studies investigated the feasibility and practical challenges of implementing the ESS using a different mode, where all sources of error were affected by the mode design (studies 4, 5, and 6).
Studies 1 and 2, undertaken in collaboration with Gallup Europe, focused on measurement equivalence. Study 1, conducted in Hungary in 2003, involved a lab test with random allocation of respondents to different modes of data collection (face-to-face, telephone, Internet and paper self-completion).
Study 2, conducted in Budapest and Lisbon in 2005, was an experiment designed to investigate the causes of measurement differences between face-to-face and telephone interviewing.
Study 3 was conducted in the UK in 2010, by asking ESS respondents to participate in a short follow-up web survey. The goal was to evaluate measurement differences between face-to-face and web data collection, using a within respondent re-interview design.
Study 4 assessed the feasibility of using telephone interviews in the ESS, focusing on the effect of varying interview length on respondents' willingness to participate in the survey. The study was carried out in 2006 in five countries: Cyprus, Germany, Hungary, Poland, and Switzerland.
Studies 5 and 6 tried to evaluate the effects of mixed-mode designs on response rates, representativeness of samples, survey costs, and data quality. Study 5 was conducted in the Netherlands in 2008, parallel to Round 4 of the ESS, and tested two different mixed-mode (web, telephone, face-to-face) data collection designs: a ‘concurrent mode choice design’, and a ‘sequential mode choice design’.
Study 6 was conducted in Estonia, Sweden and the UK in 2012. Countries were allowed to select the ideal mixed-mode design given the country’s survey environment. Estonia and the UK focused on a web and face-to-face mix, whereas Sweden tested a telephone and face-to-face design.
A summary of the results of the 6 experiments can be found as follows:
Villar, A & Fitzgerald, R (2017) ‘Using mixed modes in survey data research: Results from Six Experiments’ In Breen, M (ed) Values and Identities in Europe. Evidence from the European Social Survey. Routledge. London.
Click here to view an open access version of this paper
Building on the experience of these studies, ESS ERIC is currently coordinating Work Package 7 of the SERISS project. The Work Package investigates opportunities and challenges of high quality data collection using the web, capitalising on existing probability-based face-to-face surveys to establish a probability sample based web panel. WP7 has the following objectives:
- evaluate the feasibility of establishing the first cross-national probability-based web panel using the achieved sample from existing cross-sectional surveys;
- foreground a methodology for building new and efficient web-based survey infrastructures for Europe based on state of the art procedures and technology;
- develop a blueprint for a comparative probability-based web survey.