Conferences

Survey data are unavoidably contaminated with measurement error. We focus on formatting error, the increase in confidential intervals of survey estimates, making it more difficult to detect existing differences. Formatting error happens when respondents do not find an option on a rating scale that perfectly reflects their true value. The difference between the true value and the chosen response option is formatting error.

From a theoretical point of view, continuous visual analogue scales (VASs) have Ð on the individual as well as on the aggregated level Ð an expected formatting error of zero, because there is a perfectly fitting option for every graduation of the true value. An empirical determination of formatting error with VASs is pending and it is unclear, if populations with a low formal education are able to use VASs in a meaningful way.

Formatting error with categorical scale is different for single individual variables and for aggregate data. In the first case, it depends on the number of categories only. In the second case it is additionally influenced by the actual distribution of values in the sample. We simulated differently distributed data (e.g. from uniform distributions, narrow and wide normal distributions, chi-square and exponential distributions) to determine the expected formatting error with categorical scales consisting of 3 to 21 categories.

Study Design

To determine formatting error we chose numeric values as a construct with a linear mental representation that should be understood by every respondent in the same way. In a Web experiment participants had to locate 15 percentages (ranging from 5% to 95%, presented in random order) on VASs anchored with Ò0%Ó and Ò100%Ó. In a between-subjects design respondents were randomly assigned to VASs made of plain horizontal lines or to VASs with a number of vertical equidistant markers (ranging from one marker indicating the middle of the rating scale to 100 equally spaced markers).

Results & Conclusion

Overall (N = 1909), we found a very low mean empirical formatting error of M = -1.24 percentage points (SD = 3.05). Ratings on plain VASs without any marker (n = 167) were worse (M = -2.48, SD = 2.64) and formal education (below college: M = -2.87, SD = 2.93; at least college: M = -1.71, SD = 1.61) made a statistically significant difference: F(1, 166) = 7.56, p < .01, eta2 = .04. VASs with ten markers (n = 181) lead to the smallest formatting error (M = -0.41, SD = 1.55) for respondents with a low education (M = -0.47, SD = 1.63) and respondents with a high formal education (M = -0.30, SD = 1.39), F(1, 180) < 1.

For individual variables, the empirical formatting error for VASs with ten markers is even with respondents with a low formal education lower than the expected formatting error for categorical scales up to 50 options. Overall, the authors strongly recommend considering VASs in computer-based self-administered questionnaires for the sake of more confident survey estimates.