Data Quality Assessment

In order to achieve its research objectives and ensure that ESS data is collected using the highest methodological standards, the ESS undertakes a range of data quality assessment activities.

The ESS Core Scientific Team (CST) implements these activities throughout the survey life cycle and across ESS rounds: these include evaluating the quality and comparability of its measurement instruments, assessing the socio-demographic sample composition using external benchmark data and assessing the process and output quality of the survey.

Measurement quality and comparability

Measurement quality of individual questions

All decisions taken when designing survey questions, such as whether to provide an introduction to respondents, an instruction to interviewers, which type of response options or wording to use to formulate the request for an answer, affect the way respondents react to a specific question, and thereby the measurement quality of the responses to this question. Measurement quality refers to the strength of the relationship between the concept of interest and the observed answers.

The measurement quality of single questions can be estimated using the Multitrait-Multimethod (MTMM) approach, an experimental setting that consists in asking the same respondents three survey questions measuring different concepts of interest (traits) twice using different response scales (methods) each time. A MTMM experiment allows to estimate the reliability, validity and method effects of the questions included in the experiment. The product of reliability and validity is known as the measurement quality.

In ESS Rounds 1 to 7, a two-group split-ballot MTMM design was implemented. To improve the estimation of the MTMM models, a three-group design was implemented starting with ESS Round 8. The topics of the evaluated questions are summarised in this table:

The complete set of ESS questions evaluated through MTMM experiments and their quality information is available from the Survey Quality Predictor (SQP)'s open-source database. For more information on how to obtain the measurement quality estimates from the MTMM experiments, see the SQP user manual:

Measurement quality estimates from the MTMM experiments serve three purposes:

• Aiding question design
• Correcting for measurement errors and thereby increasing the accuracy of substantive conclusions.
• Enriching the meta-analysis underlying the SQP

Survey Quality Predictor (SQP)

SQP was developed to predict the measurement quality of survey questions. Predictions are based on a meta-analysis of a large number of MTMM experiments and the characteristics of the questions which were included in the MTMM experiments. SQP is a free license online software.

In the ESS, SQP is used during the source questionnaire development and translation processes.

Measurement quality of concepts

It is common practice in the social and behavioural sciences to combine the indicators of a concept into a single measure to facilitate its use in further analyses. Those combinations of indicators are referred to as indices, sum-scores, composite scores, or composite measures. While for individual questions it is possible to predict their measurement quality, e.g. using SQP, once the indicators are combined this is no longer possible. Therefore, the ESS provides users with the indices and their quality if scalar equivalence was established. The indices for which the information is available are summarised in this table:

Measurement equivalence

The ESS aims to achieve comparability of the data collected across all countries while minimising total survey error. In the pursuit of this goal, the ESS follows procedures while developing and translating the source questionnaire to ensure that the concepts being measured are equivalent across the participating countries. After the data collection, the CST tests for measurement equivalence, which determines whether the differences found between countries or groups can be attributed to differences or are only caused by the measurement instruments.

In the ESS, we test for three levels of measurement equivalence: configural, metric and scalar equivalence. The concepts for which measurement equivalence was examined are summarised in this table:

Assessment of socio-demographic sample composition

Survey samples should reflect the underlying target population adequately. The comparison of survey results with independent and more accurate information about the population parameters is a well-known method to analyse sample quality. For ESS Rounds 5 to 7, the socio-demographic sample composition in ESS countries was assessed by comparing ESS variable distributions with external benchmark data from the European Union Labour Force Survey (LFS). The analyses pursue two aims. First, they provide an indication of the degree of over-/underrepresentation of certain demographic subgroups in ESS samples. Second, they describe the correlates of over-/underrepresentation, focusing on two basic parameters, namely the response rate achieved and the type of sample used.

Quality report

From ESS Rounds 6 onwards, the Core Scientific Team produced a Quality Report seeking to provide a comprehensive overview of the quality of the ESS. This encompasses almost all elements of the survey life cycle and comprises an assessment of both the process and the output quality of the survey. This report is the basis for country-specific evaluation and advice regarding future ESS rounds. The reports are available below.

Related content

Data on this topic

Download ESS Test data (MTMM) documents

Data from test variables (MTMM)

Users of data are obliged to read the ESS Conditions of use within the Survey Participants page.

ESS Documents on this topic

Publications on this topic