Data Quality Assessment
In order to achieve its research objectives and ensure that ESS data is collected using the highest methodological standards, the ESS undertakes a range of data quality assessment activities.
The ESS Core Scientific Team (CST) implements these activities throughout the survey life cycle and across ESS rounds: these include evaluating the quality and comparability of its measurement instruments, assessing the socio-demographic sample composition using external benchmark data and assessing the process and output quality of the survey.
Measurement quality and comparability
Measurement quality of individual questions
All decisions taken when designing survey questions, such as whether to provide an introduction to respondents, an instruction to interviewers, which type of response options or wording to use to formulate the request for an answer, affect the way respondents react to a specific question, and thereby the measurement quality of the responses to this question. Measurement quality refers to the strength of the relationship between the concept of interest and the observed answers.
The measurement quality of single questions can be estimated using the Multitrait-Multimethod (MTMM) approach, an experimental setting that consists in asking the same respondents three survey questions measuring different concepts of interest (traits) twice using different response scales (methods) each time. A MTMM experiment allows to estimate the reliability, validity and method effects of the questions included in the experiment. The product of reliability and validity is known as the measurement quality.
In ESS Rounds 1 to 7, a two-group split-ballot MTMM design was implemented. To improve the estimation of the MTMM models, a three-group design was implemented starting with ESS Round 8. The topics of the evaluated questions are summarised in this table:
The complete set of ESS questions evaluated through MTMM experiments and their quality information is available from the Survey Quality Predictor (SQP)'s open-source database. For more information on how to obtain the measurement quality estimates from the MTMM experiments, see the SQP user manual.
Measurement quality estimates from the MTMM experiments serve three purposes:
- Aiding question design
- Correcting for measurement errors and thereby increasing the accuracy of substantive conclusions.
- Enriching the meta-analysis underlying the SQP
Survey Quality Predictor (SQP)
SQP was developed to predict the measurement quality of survey questions. Predictions are based on a meta-analysis of a large number of MTMM experiments and the characteristics of the questions which were included in the MTMM experiments. SQP is a free license online software.
In the ESS, SQP is used during the source questionnaire development and translation processes.
Measurement quality of concepts
It is common practice in the social and behavioural sciences to combine the indicators of a concept into a single measure to facilitate its use in further analyses. Those combinations of indicators are referred to as indices, sum-scores, composite scores, or composite measures. While for individual questions it is possible to predict their measurement quality, e.g. using SQP, once the indicators are combined this is no longer possible. Therefore, the ESS provides users with the indices and their quality if scalar equivalence was established. The indices for which the information is available are summarised in this table:
Measurement equivalence
The ESS aims to achieve comparability of the data collected across all countries while minimising total survey error. In the pursuit of this goal, the ESS follows procedures while developing and translating the source questionnaire to ensure that the concepts being measured are equivalent across the participating countries. After the data collection, the CST tests for measurement equivalence, which determines whether the differences found between countries or groups can be attributed to differences or are only caused by the measurement instruments.
In the ESS, we test for three levels of measurement equivalence: configural, metric and scalar equivalence. The concepts for which measurement equivalence was examined are summarised in this table:
Assessment of socio-demographic sample composition
Survey samples should reflect the underlying target population adequately. The comparison of survey results with independent and more accurate information about the population parameters is a well-known method to analyse sample quality. For ESS Rounds 5 to 7, the socio-demographic sample composition in ESS countries was assessed by comparing ESS variable distributions with external benchmark data from the European Union Labour Force Survey (LFS). The analyses pursue two aims. First, they provide an indication of the degree of over-/underrepresentation of certain demographic subgroups in ESS samples. Second, they describe the correlates of over-/underrepresentation, focusing on two basic parameters, namely the response rate achieved and the type of sample used.
Quality report
From ESS Round 6 onwards, the Core Scientific Team produced a Quality Report seeking to provide a comprehensive overview of the quality of the ESS. This encompasses almost all elements of the survey life cycle and comprises an assessment of both the process and the output quality of the survey. This report is the basis for country-specific evaluation and advice regarding future ESS rounds. The reports are available below.
Publications on this topic
Pirralha, A. & Weber, W. (2014). Evaluations of the Measurement of the Concepts 'Political Satisfaction' and 'Quality of State Services', RECSM Working Paper no. 40, Barcelona
Zavala, D. (2012). Evaluation of the Concepts 'Trust in Institutions' and 'Trust in Authorities' (ESS-DACE Deliverable 12.4), RECSM Working Paper no. 29, Barcelona
Saris, W.E. & Gallhofer, I.N. (2011). The Results of the MTMM Experiments in Round 2, RECSM Working Paper no. 23, Barcelona
Saris, W.E., Oberski, D., Revilla, M., Zavala, D., Lilleoja, L., Gallhofer, I., & Gruner, T. (2011). The Development of the Program SQP 2.0 for the Prediction of the Quality of Survey Questions, RECSM Working Paper no. 24, Barcelona
Loosveldt, G. & Beullens, K. (2017). Interviewer Effects on Non-Differentiation and Straightlining in the European Social Survey. Journal of Official Statistics, 33 (2), 409–426
Saris, W.E. & Revilla, M. (2016). Correction for Measurement Errors in Survey Research: Necessary and Possible. Social Indicators Research, 127 (3), 1005–1020
Revilla, M., Saris, W.E., & Krosnick, J.A. (2014). Choosing the Number of Categories in Agree-Disagree Scales. Sociological Methods & Research, 43 (1), 73–97
Revilla, M. (2013). Measurement Invariance and Quality of Composite Scores in a Face-to-Face and a Web Survey. Survey Research Methods, 7 (1), 17–28
Revilla, M. (2010). Quality in Unimode and Mixed-Mode Designs: A Multitrait-Multimethod Approach. Survey Research Methods, 4 (3), 151–164
Saris, W.E., Revilla, M., Krosnick, J.A., & Shaeffer, E.M. (2010). Comparing Questions with Agree/Disagree Response Options to Questions with Construct-Specific Response Options. Survey Research Methods, 4 (1), 61–79.
Coromina, L. & Saris, W.E. (2009). Quality of Media Use Measurement. International Journal of Public Opinion Research, 21 (4), 424–450
Saris, W.E., Satorra, A., & Coenders, G. (2004). A New Approach to Evaluating the Quality of Measurement Instruments: The Split-Ballot MTMM Design. Sociological Methodology, 34 (1), 311–347
Revilla, M., Zavala, D., & Saris, W.E. (2016). Creating a Good Question: How to Use Cumulative Experience. In C. Wolf et al. (eds.), The SAGE Handbook of Survey Methodology. London: SAGE
Saris, W.E. & Gallhofer, I.N. (2014). Design, Evaluation, and Analysis of Questionnaires for Survey Research. Hoboken: Wiley