Conferences

It is increasingly common practice for designers of survey questions to subject them to some form of evaluation or testing during the survey development phase (or even after the survey has been conducted). There are many methods available, both quantitative and qualitative. Quantitative methods can involve statistical modelling, using Item Response Theory (IRT) or predictions of reliability and validity. They can also involve experiments using split-samples to compare alternative versions of questions. Response error can be identified using response latency and para linguistic expressions. Behaviour coding can identify problems not only with response error but also with the adequacy of the interviewers’ script. Qualitative methods can involve the utilisation of cognitive testing methods, based on cognitive and social psychology, to identify problems for respondents in the question and answer process. The use of focus groups, depth interviews, vignettes and card sorts can assist in our understanding of how respondents conceptualise particular social phenomena. There is an extensive armoury of methods available to question designers who wish to evaluate the performance of survey questions. But is it worthwhile testing and evaluating survey questions? Can we improve the performance of survey measures by using these techniques? What does the evidence tell us about whether we have been successful in ‘improving’ survey questions or survey measures as a result of applying these evaluation methods?

We welcome papers based on research that has combined different testing and evaluation methods, which may include, but are not limited, to the following:

1. research designs which combined different methods to evaluate different versions of the same survey question(s) or set of survey measures. What did the evidence tell you about which version was ‘best’ or ‘better’?

2. methodological investigation of the use of different evaluation and testing methods. Were the data obtained using different methods comparable or not? What kinds of data do different methods yield and how useful is this to the survey designer/analyst?

3. multi-method research aim at predicting sources of measurement error.? How ‘good’ are the predictions about the sources of measurement error and was it possible to quantify the extent of this error?