Survey woes

In the “Feedback” section of the holiday New Scientist (p. 88):

The sign-up questionnaire on the Captain Cash money-saving website run by the London-based Sunday tabloid News of the World asks applicants to select from the following: “In a typical month, I buy the NOTW: never/less than once/1-2 times/3-4 times.”

Michael Barraclough is still trying to work out which of the first two options to tick.

You can see what the questionnaire was trying to tap: the average frequency with which someone buys the NOTW; that’s what “in a typical month” is supposed to convey. So the options were supposed to be understood as:

/ some months, I don’t buy it, but some months I buy one copy;
/ in a month, I buy one or two copies;
/ in a month, I buy three or four copies.

This is still problematic (for reasons I’ll discuss in a moment), but the first thing to note is that ordinary people are not at all good at estimating such frequencies; the task requires averaging over a number of months, and people are unlikely to hold such data in memory. They can report what they did in the most recent month, or what they intend to do in a typical month, but that’s not what the survey wants to find out; it’s supposed to gauge readership. Readership can be estimated by indirect and somewhat complex methods, but self-report surveys are the cheap and easy way to go.

Suppose, however, that we reformulate the questionnaire in terms of averages (that is, means):

/ the average number of times a month I buy the NOTW is greater than 0 but less than 1;
/ the average number of times is greater than 1 but less than 3;
/ the average number of times is 3 or greater.

There are other possible reformulations, but this one seems to me to come close to the intentions of the original questionnaire. In particular, the second clause (an average greater than 0 but less than 1) looks like a good fit to the original second clause (less than once in a typical month), and not so good a fit to my first reformulation above (no copies in some months, one copy in some months) — because the second formulation allows for possibilities other than averaging some number of 0s with some number of 1s (three 0-copy months with one 2-copy month, for instance.

Another complexity is that though most calendar months have four Sundays, some have five, and very occasionally there’s a February with only three [added: not really so; see Kivi Shapiro’s comment below], so that the averages will depend on which particular months are in the mix. (Of course, you could try reformulating things with month referring to four-week periods beginning with a Sunday, but ordinary people just won’t cope with that.) [added: for a possible solution, see Chris Hansen’s comment below]

These are small points, involving minor effects, and quite possibly my first reformulation (which is at least comprehensible to ordinary people) will do for practical purposes.

There is a larger point here, about how difficult it is to construct surveys that tap what you want them to.

3 Responses to “Survey woes”

1. Chris Says:

In another life I was head of the typing department at the Harris Poll, responsible for typing questionnaires, statistical tables, and reports. The way I would have worded this question is this:

How often do you buy the NOTW?

1. Every week
2. Three out of every four weeks.
3. Twice out of every four weeks.
4. Once every four weeks
5. Less often than once every four weeks
6. I have never bought the NOTW.

This is still suboptimal in some ways, having the disadvantage that the average number of copies bought by respondents can’t really be determined with any reliability, as there is no numerical value that one can assign to answer #5 although most analysts would assign a value of 0.5 to it (indicating buying once every two months) just for the sake of the numbers.

The best way to find out how to word the question is to ask the client (ie, NOTW) what they want to find out from this question. If they are interested in segmenting the sample into, say, Regular readers, Semi-regular readers, and Occasional readers, then this question would give them the information they need (1-2 Regular, 3-4 Semi-regular, 5 Occasional, and 6 Non-Readers). It also has the advantage of not being tied to months, and is clearer to the respondent.

The major difficulty in writing questionnaires is making the question clear and unambiguous while also satisfying the client, who is usually not familiar enough with market research to understand what you are doing. When I was working at the Harris Poll, a client who shall remain nameless asked us to administer a questionnaire to its customers. One question went on for about 20 lines. It requested a Yes/No/Not Sure response and was administered over the phone. Harris tried to get the client to simplify the question, but s/he was adamant that the question should be asked as submitted. They were then quite surprised when about 75% of the respondents answered “Not Sure” but we weren’t–the question was so long that the respondents had forgotten what it was by the time the answer was required.

One other interesting factoid appeared in a study we did for a liquor company. The final report bounced back and forth from various levels of the company’s management with requested changes. When it got to the CEO’s office, it came back to us with one requested change: wherever the word “drunk” appeared, as in “How often have you drunk gin in the past month?”, it should be replaced by the word “consumed”.

2. Kivi Shapiro Says:

Three-Sunday Februaries really are very occasional. The only one that’s been observed so far was in 1700, and that only in Denmark. There have been several 2-Sunday Februaries, though: in 1753 Sweden, in 1918 Russia, and in 1923 Greece.

3. arnoldzwicky Says:

To Kivi Shapiro: ouch. Just so; I miscalculated. Note added in the text.