Objectives and expected impact:

JRA3 will concentrate on problems associated with the quality of data collection instruments.  Two main areas in particular need investigation.  The first is testing and attempting to improve the reliability and validity of questions in cross-cultural surveys such as the ESS.  The second is to improve the procedures and protocols used for translation across participating countries, a critical issue for cross-cultural equivalence. 

1.  Reliability and validity of ESS questions

Data from survey research contain both random and systematic errors, which are attributable to a range of factors.  In attitude surveys, for instance, random error is a consequence of mistakes made by the respondent, interviewer and others in recording the answers.  Systematic errors in contrast can arise from ‘faulty' questions or different reactions of respondents to the chosen methods, thus generating biased answers.  In a comparative context, measuring and correcting for errors is exacerbated by the fact that the size of these different error components may vary cross-nationally, resulting in reduced comparability of findings. 

The aim of this part of JRA3 is to estimate the size of these different error components and to propose correction procedures so that a higher degree of equivalence can be achieved across data from different countries.  Not all aspects of data quality are easy to measure or evaluate.  Among the most widely used criteria are reliability, validity, extent of item non-response, relative bias and response effects, (mis)understanding of questions, and problems in the interaction between interviewer and respondent.  A large body of research has been undertaken into the sorts of question which are particularly error-prone in relation to one or more of these criteria, several of which have tested alternative formats and wordings by means of ‘split ballot experiments' (Schuman & Presser 1981; Krosnick & Fabrigar, forthcoming).  Meanwhile, non-experimental studies have investigated the effect of question characteristics on item non-response and bias (Molenaar 1986), and longitudinal studies (with test-retest designs) have evaluated the effects of question design on the reliability of responses (Alwin & Krosnick 1991).  ‘Multi-Trait Multi-Method' (MTMM) studies have in turn evaluated the effects of question design on reliability and validity (Andrews 1984; Költringer, 1995; Scherpenzeel 1995; Scherpenzeel & Saris 1997). 

Most MTMM studies have concentrated on the effect of one factor on the distribution of the variable of interest, but a few have employed meta-analysis of MTMM studies to determine the effects of alternative design choices during the development of questions on reliability and validity (Andrews, 1984; Költringer, 1995; Scherpenzeel 1995; Scherpenzeel & Saris 1997).  Recent meta-analysis covering all available MTMM experiments (Saris and Gallhofer, forthcoming; Saris and van der Veld, 2004) has been used to develop a program for predicting the quality of survey questions, the Survey Quality Predictor (SQP).  Using this program, the question designer is explicit about the choices they have made in developing the survey item, and the program employs codes to estimate the reliability, validity and ‘total quality' of that item.  This approach has been applied during the questionnaire design process of each Round of the ESS and is also being used to evaluate the performance of ESS indicators post hoc. 

The major added complication of cross-cultural surveys is that an estimate of the reliability (random error) and validity (systematic error) of questions is required for each different language.  Otherwise the results cannot properly be compared.  Indeed, we have tentatively begun such an evaluation programme in the second Round of the ESS, which will provide data on the quality of survey questions in more than 20 countries and languages.  We propose to develop this work as part of JRA3, extending the SQP program (including the necessary databases) to predict data quality in different languages and to include questions that have not so far been studied.  An important aim is to develop procedures that improve the comparability of results from different countries.  Such a program is still in development, but the groundwork has been done.  Its further development will be invaluable not just for the ESS but also for other cross-national surveys in Europe and beyond.

2.  Facilitating translation efforts and translation monitoring

Following best practice, the ESS translation guidelines require two draft translations to be made for each language. These draft translations form the basis of the careful appraisal, revision and documentation that make up the procedures ultimately resulting in a final translated version of the source questionnaire.  Team translation efforts of this kind are proving their worth in multiple areas of cross-national survey research.  But they require meticulous attention to detail and multiple revisions and changes in moving from a draft translation to a finished final version.  

The documentation required for such work is substantial, starting with a review of draft translations and finishing with a detailed account of all aspects of the procedure.  This documentation helps to inform subsequent data users about the nature of and reasons behind national decisions.  It also aids the work of future question design teams in their efforts in optimising their source questions.  But in general the uniquely detailed documentation of translation procedures that the ESS employs provide a valuable source of information at all stages of the project about the often complicated relationship between source questions and their multiple translations.

National teams in good cross-national research have numerous roles, for each of which they need efficient and practical tools.  As noted, not only do they have to produce the translations, but they also have to update and document them as they adapt and change to the process of review and arbitration.  Countries sharing a language (there are many in the ESS) are permitted to use the same tool to compare their respective versions but not in advance of performing the initial task themselves.  Unlike in the ESS, commercially-based translators regularly work with computer-assisted translation workbenches (CATW), which track their progress and facilitate the process of documentation.  But social surveys have mostly had to work within more parsimonious environments.  Now, with the development of technology, some of these facilities should soon be more readily available.  And though none of them will meet all the needs of a large multi-lingual, multi-national survey, some components in existing computer-assisted workbenches could greatly facilitate the translation goals of ESS-type projects.

The aim of this part of the JRA3 is therefore to conduct reviews of existing CATW and to identify those areas in which either new developments or adaptations are required so as to produce a custom-built program that suits the needs of such surveys.  The advantage of working within the environment of the ESS is that its existing translations are available to be used in the review and benchmarking of these new developments.

The impact of such a development should not be underestimated. After all, survey research is not alone in the benefits it has accrued in recent decades from the development of computer technology. But translation protocols are perhaps the single area of survey work which have not yet been able to exploit such benefits. A sea-change now seems within reach and the i3 is the ideal mechanism through which to make substantial progress towards that goal.