Good examples of poor survey questions

Example 1: From a local council postal survey

The uncleaned data showed a large percentage of parents answering “no” to Q10 and continuing on to mark childcare in Q11! Fortunately this had a clear pattern. These parents marked the “Informal” category. These parents clearly did not think that family, friends and neighbours counted as childcare. This is a fault in the design of the simple question about childcare use.

Q10 Do you use childcare?		Y/N
Q11 If yes, what type of childcare to you use?
	Informal, e.g. Family / Friends / Neighbours	1
	Childminder, Self-employed carers based in their own homes	2
	Home Based Child Carer, Employed by parents and based in parents home	3
	Nanny / Aupair, Employed by parents and based in parents home	4
	Crèche, Occasional care for parents to access work, training or one off events	5
	Pre-school Play Group, Sessions of play and/or education for children aged 2-5yrs	6
	Day Nursery, To provide care and education for children aged 6 weeks to 5 yrs	7
	Breakfast / Before School Club, Safe place to play before school starts for children aged 3-14yrs	8
	After School Club, Safe place to play after school finishes for children aged 3-14yrs	9
	Holiday Play Scheme / Club, Safe place to play during school holidays for children aged 3-14yrs	10
	Specialist care for ages 15, 16, 17 - children with a disability only	11
	Other, please state	12

Example 2:

From my 2011 AAPOR presentation based on ESRC SDMI Mixed Modes and Measurement Error data (grant number RES-175-25-0007) with colleagues Michelle Gray and Margaret Blake (National Centre for Social Research) and Steven Hope (University College London). Other core project team members include: Gerry Nicolaas (National Centre for Social Research) and Peter Lynn and Annette Jäckle, (Institute for Social and Economic Research). Below respondents have been randomly assigned to two conditions.

8 Category Versions	3 Category Versions
FM75. Which of these best describes your home? Would you say a . . . (READ OUT) . . . Detached house 1 Semi-detached house 2 Terraced house 3 Bungalow 4 Flat in a block of flats 5 Flat in a house 6 Maisonette 7 Or other? 8	FM75. Which of these best describes your home? Would you say a . . . (READ OUT) . . . House 1 Flat or maisonette 2 Or other ? 3

When the 8 category version is collapsed into the same 3 categories as the 3 category version, you don't get the same results. Cognitive interviewing suggested that 'type of dwelling' was a confusing concept when this would not have been expected:

• What the [Blip] difference is there between a maisonette and a flat and a block of flats, a flat and a house? (Male, postgraduate degree, employed, high income, White British
• Regarding a maisonette. Household member: I'll call it a duplex, yeah. Respondent: Well, it's what they call it in the South. (Male, postgraduate degree, employed, high income, White British)
• R answered 'flat', but the interviewer observed it as semi-detached house. R said it had to be very large to be called a house (Female, higher education below degree level, employed, medium income, other ethnicity).

Example 3: From Housing Association postal questionnaire

2. How happy are you with the area you live in?

Very satisfied
Fairly satisfied
Neither satisfied nor dissatisfied
Fairly unsatisfied
Very unsatisfied
Don't know

• Mismatch between stem and answer categories - Happiness vs. Satisfaction
• Uses “dissatisfied” and “unsatisfied”
• "Area" is ambiguous
• "You" is ambiguous - Singular (meaning just the respondent) or plural (meaning the whole household)
• A reference period would help
• Too general to be useful
• Etc, etc.