A population survey
A geographical survey: answer the tiny questionnaire and then analyse all the collected responses...
Age
14 to 18
Challenge level
Are you good at estimating? We have written a survey inviting you to estimate the answer to a question, and then compare your estimate with all the other people who have had a go.
Before reading on, please complete this very short questionnaire* on your own to contribute your estimate (and please just make a reasonable guess if you don't "know" the answers):
(Only fill it in once!)
Once you have filled in the questionnaire, click "Show" to see the results so far:
This resource is part of the collection Statistics - Maths of Real Life
Before reading on, please complete this very short questionnaire* on your own to contribute your estimate (and please just make a reasonable guess if you don't "know" the answers):
(Only fill it in once!)
Once you have filled in the questionnaire, click "Show" to see the results so far:
There are actually three slightly different questionnaires!
You filled in one of these chosen at random by the computer.
All three questionnaires ask: "What do you think the population of the Philippines is (in millions)?" However, two of the questionnaires ask an additional "primer question" first.
On one questionnaire, the primer question reads:
and on the other, it reads:
What effect do you think the primer questions might have on people's estimates?
Click "Show" to see graphs of these different survey responses:
You filled in one of these chosen at random by the computer.
All three questionnaires ask: "What do you think the population of the Philippines is (in millions)?" However, two of the questionnaires ask an additional "primer question" first.
On one questionnaire, the primer question reads:
"Do you think the population of the Philippines is more than 30 million?"
and on the other, it reads:
"Do you think the population of the Philippines is more than 70 million?"
What effect do you think the primer questions might have on people's estimates?
Click "Show" to see graphs of these different survey responses:
The results are shown in this comparative box plot (updated approximately every hour), where there is one box plot for each questionnaire.
In this graph, the "n =" values show the number of responses. The red dashed lines mark 30 and 70 million, while the pale blue line shows the approximate actual population of the Philippines (about 105 million as of 2018). The graph is cut off at 150 million (so larger responses are not visible, though they are used when calculating the box plots).
We can also distinguish between those respondents who answered "yes" to the primer question and those who answered "no", resulting in the following comparative box plot:
You can also see the raw data in this Google doc (which is continuously updated); the three sheets are for the three different questionnaires.
What is the impact of asking a "primer question"?
Why might this be the case? What does this suggest about designing questionnaires?
Image
In this graph, the "n =" values show the number of responses. The red dashed lines mark 30 and 70 million, while the pale blue line shows the approximate actual population of the Philippines (about 105 million as of 2018). The graph is cut off at 150 million (so larger responses are not visible, though they are used when calculating the box plots).
We can also distinguish between those respondents who answered "yes" to the primer question and those who answered "no", resulting in the following comparative box plot:
Image
You can also see the raw data in this Google doc (which is continuously updated); the three sheets are for the three different questionnaires.
What is the impact of asking a "primer question"?
Why might this be the case? What does this suggest about designing questionnaires?
You can download the R code used to generate these box plots: philippines.R; you can use R (or RStudio), a spreadsheet or other software to explore the data further if you wish.
* This anonymous survey is run through Google forms. Google only provides NRICH with the date and time that the questionnaire was completed and the responses to the actual questions asked. No other data is collected.
With thanks to Michael Posner of the Center for Statistics Education at Villanova University for suggesting this problem.
With thanks to Michael Posner of the Center for Statistics Education at Villanova University for suggesting this problem.
This resource is part of the collection Statistics - Maths of Real Life
The primer question influences people's views of what sort of size population the Philippines is likely to have. Asking whether people believe it is more than 30 million suggests that the correct answer is about 30 million, so it is likely that people will estimate an answer close to 30 million, and similarly for the 70 million question.
It is also likely to be the case that the spread of estimates will be much smaller with the primer question than without it. Whether this is the case or not will become clear over time as more people respond to the survey!
If we want to know people's "genuine" views or ideas, then, we must be very careful not to ask primer questions beforehand. Any question which hints towards a certain type of response might well influence the results of later questions.
On the other hand, primer questions can be very desirable if the survey's actual purpose is to influence people. For example, if ACME Widgets wanted to influence people to shop in its stores, it could send out people to do surveys on a high street, asking them questions such as, "Which of the following would you be most likely to visit ACME Widgets to buy? (A) ..., (B) ..., (C) ..., (D) something else" before asking them "Which store are you most likely to visit to buy X?" This is quite obvious, though, and people might well realise that it's a marketing ploy, so the primer question may be more subtle than this. However, it is important to be aware that this sort of survey is not actually a survey but a marketing tool, and there are laws relating to marketing which must be followed in such a case.
It is also likely to be the case that the spread of estimates will be much smaller with the primer question than without it. Whether this is the case or not will become clear over time as more people respond to the survey!
If we want to know people's "genuine" views or ideas, then, we must be very careful not to ask primer questions beforehand. Any question which hints towards a certain type of response might well influence the results of later questions.
On the other hand, primer questions can be very desirable if the survey's actual purpose is to influence people. For example, if ACME Widgets wanted to influence people to shop in its stores, it could send out people to do surveys on a high street, asking them questions such as, "Which of the following would you be most likely to visit ACME Widgets to buy? (A) ..., (B) ..., (C) ..., (D) something else" before asking them "Which store are you most likely to visit to buy X?" This is quite obvious, though, and people might well realise that it's a marketing ploy, so the primer question may be more subtle than this. However, it is important to be aware that this sort of survey is not actually a survey but a marketing tool, and there are laws relating to marketing which must be followed in such a case.
Spoiler alert: If you want to do the survey on the problem page yourself, please do so before reading further, as your answers may be influenced by the content of these notes.
To read the Teachers' Resources, click "Show".
Why do this problem?
One aspect of questionnaire writing that is often overlooked is the effect of primer questions, that is, questions which have an impact on respondents' answers to later questions. Sometimes this is used to deliberately manipulate people's views, but usually it simply biases the results of the survey in an unintended way.
This problem provides a minimal example of this phenomenon: a one or two question questionnaire.
Possible approach
One approach is to ask all of your students to visit this problem before the lesson or at the start of the lesson, click on the questionnaire link and answer the survey. This has the additional benefit of providing more data for other users of this problem to see. You could, if you wish, ask your students to copy down the question or questions asked and their responses so that you can discuss the class's responses in addition to the globally collected responses. If the class do answer the questionnaires at the start of a lesson in a computer room or using tablets, they will be able to see their results immediately on the Google doc (though it is not possible to isolate their responses precisely), but the box plots are only updated hourly.
An alternative approach is to gather the class data with pencil and paper. One way is give everyone a primer question. Give half the class one of the primer questions and the other half of the class the other primer question, and ask them to note their answer. (For example, this could be printed on differently coloured pieces of paper, or half the class could close their eyes while the question is asked to the other half by displaying it on a projector.) Then ask the whole class the second question and for them to write down their answers. Then the class can collect their responses and work together to analyse them. Another way is to split the class into three, giving primer questions to two of the groups and a relatively irrelevant question to the third group, for example "What are people who come from the Philippines called?", "Which continent is the Philippines in?" or "What is the capital of the Philippines?"
Once the data is collected, the class can be told about the primer questions, and asked what they expect the numerical answers to the population question to be: how would they differ for each group? After a short discussion, their collected data can be plotted in some way, or students can be shown the box plots from the data collected on the site. To what extent does the data support their earlier thoughts or challenge it?
Moving beyond this specific case, students could then be asked why survey designers might be careful to avoid this effect, or why they might deliberately want to use it.
Key questions
- Why do we have to think carefully about the whole sequence of questions that we ask in a questionnaire?
- What effects might be caused by having related questions on a questionnaire?