English title: A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design

Author(s): W.E. Saris - Albert Satorra - Germà Coenders -

Language: English

Type: Journal article

Year: 2004

Abstract

Two distinctly different quantitative approaches are used to evaluate measurement instruments: the split-ballot experiment and the multitrait-multimethod (MTMM) approach. The first approach is typically used to indicate whether variation in the method causes differences in the response distribution; the second approach evaluates the reliability and validity of different methods. The new approach, suggested in this paper, combines the more attractive features of both methods. The strength of the split-ballot experiment is its use of independent random samples from the same population to provide information about differences in response distributions. This is also possible with the new approach, but this approach provides more detailed information about the reasons for the differences. The MTMM approach provides information about reliability and validity on the basis of repeated observation of the same traits using different methods. This information is also provided by the new design. The difference is that the new approach reduces the need for repeated observations of the same trait. Each sample is provided with a different combination of only two methods and the complete model with all methods is estimated as a multiple-group model. This reduces the burden for respondents and also reduces memory and order effects. Alternative designs and estimation methods are discussed, their efficiency is analyzed, and illustrations are provided.

Volume: 34

Issue: 1

From page no: 311

To page no: 347

Refereed: Yes

DOI: 10.1111/J.0081-1750.2004.00155.X

Journal: Sociological methodology