How ‘question order’ affects the electorate’s response
Consider this: You are director of operations for a PAC. A poll just came out that suggests the electorate significantly increased their support for your cause. Now you want to send a message and hammer home that message. You used Aristotle 360’s Query Builder to assemble a list of individuals that meet your selected demographic criteria. The perfect message is created and, thanks to 360, a tailored email is ready to send. One question remains, how do you know the poll is an accurate representation of the electorate’s view?
One duty of the director of operations is to traverse the delicate public perception of their cause. Political technology enables a message to be adjusted in real-time. An experienced PAC program knows how to evaluate a poll and see through partisan objectives, which are, in the case of polls, mostly media driven. Most polls are structured in a form called ‘question order’, which means there is logical reasoning for asking one question before another. (Pew has an exceptional description of question order here.)
There is plenty of information noted in survey textbooks, psychology scholarship and opinion research papers in regard to ‘order(ing) effect’. Researchers, good researchers, go out of their way to make sure that the order in which a question is asked limits the impact on the subsequent question.
A question’s verbiage is significantly important. (E.g., President Obama’s time in office has been pretty awful, why would you say that is?) With a question like this, you would expect to see answers like: “because he’s terrible” or “he isn’t American.” This question makes one think that I am ‘leading’ to give the president negative marks. However, more important than the verbiage of a particular question are the questions that precede it.
President Obama was handed the worst economic situation in the history of the world, how do you think he has handled it.
During Obama’s time in Office, every oil field in the world dried up, with that said, how do you think Obama has handled this adversity?
Now, after knowing the two preceding questions, you may expect different results to the question: President Obama’s time in office has been pretty awful, why would you say that is? “Because he inherited the worst economic situation in the history of the world”, “there’s no more oil”. . . I would almost rather know the question which proceeded than the verbiage of the question.
Consider this recent poll. The first question reads as follows:
Q1. Do you approve or disapprove of the way Barack Obama is handling his job as president?
- Approve (%): 44 (Total) 12 (Rep) 76 (Dem) 39 (Ind)
- Disapprove (%): 46 (Total) 84 (Rep) 16 (Dem) 47 (Ind)
- Don’t know/No answer (%): 10 (Total) 4 (Rep) 8 (Dem) 14 (Ind)
The second question then asks:
Q2. Do you feel things in this country are generally going in the right direction or do you feel things have pretty seriously gotten off on the wrong track?
- Right direction (%): 37 (Total) 14 (Rep) 57 (Dem) 36 (Ind)
- Wrong track (%): 56 (Total) 85 (Rep) 33 (Dem) 56 (Ind)
- Don’t know/No answer (%): 7 (Total) 2 (Rep) 10 (Dem) 7 (Ind)
Now, after knowing (1) the preceding question and (2) scholarly literature on ordering effect, we can make some educated assumptions. Now I’m not an economist, but I think it’s safe to assume that people typically associate a ‘right’ direction question having to do, in part, with the president’s performance. However, this doesn’t mean Q2 is an invalid question. It just needs to be examined in the context of Q1 — not in and of itself.
Let’s say these questions were flipped, and ‘right’ direction came before presidential approval. Logic would seem to reason, since fewer people responded ‘right track’ than ‘approval’ of the president, that if the question order were flipped, you may see a falloff of a few points on presidential approval.
Bottom Line: Taking into consideration question order may give you a slight advantage when analyzing polls. Whether you are a campaign manager, PAC director or running a grassroots team, you should always acknowledge the psychological effects of ‘order(ing) effect’. In doing so, you can then prepare strategies and create messaging that are a more sound representation of the electorates actual beliefs.
Brett Loyd
Director, PAC & Grassroots Consulting
