|
The ESS presents an outstanding opportunity to investigate important methodological issues concerning survey data quality in a cross-national context. A programme of methodological research has been built into the project, in order to investigate major issues in surveys such as non-response, the reliability and validity of questions and the feasibility of mixing modes of data collection.
The ESS research into mixed modes aims to provide information that will help to inform decisions regarding:
- whether mixed-mode data collection should be allowed on future rounds of the ESS;
- which modes of data collection should be allowed;
- within which kinds of overall survey design mixed modes could be employed.
The following issues are being assessed:
-
coverage and response rates that can likely be achieved with different modes and mode combinations;
-
likely differential error between modes (particularly non-response error and measurement error) and its causes.
To date, four phases of experiments have been carried out. Phase I and II, undertaken in collaboration with Gallup Europe, focused on the measurement equivalence. Phase I involved a pilot study conducted in Hungary in 2003, which allowed paired comparisons across all the main modes of data collection (face-to-face, telephone, Internet and paper self-completion). Phase II – carried out in Hungary and Portugal in 2005 – was an experiment designed to investigate the likely impact of a switch to telephone interviewing on data quality.
Phase III was designed to assess the feasibility of telephone interviews as a method of data collection for ESS, and focused on the effect of varying interview length on respondents' willingness to partcipate in the survey. An experimental study was carried out in five countries (Cyprus, Germany, Hungary, Poland, and Switzerland) in 2006/7.
Phase IV was conducted in the Netherlands in 2008, parallel to Round 4 of the ESS, and tested two different mixed-mode data collection designs: a "concurrent model" where repsondents were given a choice to complete the survey by telephone, web, or face-to-face, and a "sequential design" where the different modes were offered to respondents in the order of their costs for the survey agency. Comparison between these two designs, and between each design and the ESS 4th Round data allowed insights into the effect of mixed-mode designs on response rates, representativeness of samples, survey costs, and data quality.
Currently, one further phase of data collection is planned as part of this research programme.
Publications
Reports
Jäckle, A., Roberts, C. & Lynn, P. (2006) "Telephone versus face-to-face interviewing: mode effects on data quality and likely causes. Report on Phase II of the ESS-Gallup Mixed Mode Methodology Project". Colchester, Institute for Social and Economic Research Working Paper 2006-41.
Articles in scientific journals
Jäckle, A., Roberts, C. & Lynn, P. (2010) "Assessing the effect of data collection mode on measurement". International Statistics Review 78 (1): 3-20.
Roberts, C. Jäckle, A., & Lynn, P. (2006) "Causes of Mode Effects: Separating out Interviewer and Stimulus Effects in Comparisons of Face-to-Face and Telephone Surveys". Proceedings of the Survey Research Methods Section. Washington, DC, American Statistical Association (pp. 4221-4228).
|
Mixed Mode Data Collection
