Conferences

ESRA2009:

Conference main page

Overview of sessions

Time table

Warsaw 2009: Presentations and short courses

Using a Questionnaire Appraisal System to Improve Cross-National Questionnaire Design

Session: Questionnaire Developing and Testing

Author:

Sally Widdop; City University London, United Kingdom

Abstract:

Despite the range of pre-testing and development tools that can be used to design questionnaires, errors can often still occur leading to low measurement quality in the final items that are fielded. These errors are more problematic in cross-national surveys because of an increase in the range of things that can ‘go wrong’ and reduce equivalence both within and between countries.

By adapting and building on the Question Appraisal Systems (QAS) developed by Willis and Lessler (1999) and Dean et al. (2007), a QAS is currently being developed for use during the question design process of the European Social Survey (ESS). The QAS is a structured questionnaire that acts as a check-list for assessing potential problems with draft versions of questions.

In every round of the ESS, two 50 item rotating modules are included in the questionnaire. These focus on a specific topic and are designed by multi-national teams following a European-wide competition. Members of the Central Coordinating Team of the ESS and individuals from the successful team are involved in the design and development process for the modules. This combines expert review, country consultation and a detailed set of translation procedures with statistical modelling techniques such as multi-trait, multi-method experiments (MTMM). Based on the results of this process, individual items are adopted, re-designed or discarded.

The QAS being developed for the ESS will be split into several parts, which will be used at different stages of the design process that is already in place. In addition, the cross-cultural elements from QAS-04 designed by Dean et al. will be adapted for cross-national implementation.

This paper will assess the contribution made by the QAS to the question design process used on the ESS and will seek to answer the following key questions:

1. Did using the QAS help to identify problems that were not identified by unaided expert review?
2. Did unaided expert review identify problems that were not identified by the QAS?
3. Was using the QAS helpful in producing solutions to the problems identified compared to the expert review process?
4. Was it helpful to divide the QAS into several parts and to apply these to different stages of the question design process?
5. Was the QAS useful at identifying problems specifically associated with cross-national research?