I have worked in public opinion polling in the past, and indeed have polled on Scottish independence before, so a few people have asked me what I think about the question that has just been released by the Scottish National Party that is soon to be used in a referendum on Scottish independence:
Do you agree that Scotland should be an independent country? (Yes/No)
I'll go out on a limb here and say it's not the worst. And I thought it would be worst. But it's not great either.
Lets start with a good point. Alex Salmond has said the question is ''short, straightforward and clear'' and he's right, it is, I think anybody would be able to understand it. As we are talking about a referendum, it is necessary to plan for every adult in the country to be able to read and understand the question and I think that has been achieved. The question seems to comply with the following element of the Market Research Society code of conduct (our professional quality standard):
B.14 Members must take reasonable steps to ensure that the design and content of the data collection process or instrument is appropriate for the audience being researched.
However, and most importantly to me, as a member of the Market Research Society I wouldn't write this question myself and use it in a questionnaire. The main reason is that it is not a balanced question. The MRS code of conduct states that:
B.14 Members must take reasonable steps to ensure that Respondents are not led towards a particular point of view.
The question in no way meets this quality standard. Because it asks us whether we agree that Scotland should be an independent country, we are led towards the agree response. The way that the question is framed assumes a particular value position. It is saying 'Do you agree (with us) that Scotland should be an independent country? Eliminating bias is questionnaire design 101, and not doing so is really not acceptable. Questions should be written to be balanced by assuming no value position, for example:
- Do you agree or disagree that Scotland should be an independent country?
- Should Scotland be an independent country?
But lets consider the content of the question and intended use of the results. Thinking about what the SNP want to achieve this is a good question for the SNP to ask, in that it is probably the best way for them to drum up a bit of national pride. They ask us whether Scotland should be an independent country. Of course many will simply agree, but at the margins this wording has the potential to elicit an idealistic agree response from anyone with any leaning towards patriotism (and there are plenty of them in Scotland) particularly when the ballot form is tarted up with ethnic imagery such as saltires (sorry, I mean the Scottish Government logo). We are not asked to judge when independence should happen, or whether it will work. We are just asked to say if, in an ideal world, it should be the case. And that's a legitimate question.
There's a caveat to that though. We all know that a 'yes' vote will lead to independence, so those in Scotland who do agree that Scotland should be an independent country but don't agree that it will work in practice will be forced to lie with a no vote. So given the potential use of the result it is unclear whether the question meets the following MRS quality standard:
B.14 Members must take reasonable steps to ensure that Respondents are able to provide information in a way that reflects the view they want to express
Linked to this, when the SNP get the results they will take the percentage yes vote and if it is high enough they will use it to start the process of independence. But is that appropriate based on the question? Not really. The answer to this question is not hugely practical, it just tells you what the nation thinks might be good if we lived in a vacuum. Don't forget this is only one of a big pile of subtly different questions that the SNP could have asked. I bet if they asked Should Scotland become an independent nation next year? or Could Scotland function as an independent nation? they'd get an entirely different response. It is therefore unclear whether the question meets the following MRS quality standard:
B.14 Members must take reasonable steps to ensure that responses are capable of being interpreted in an unambiguous way
By the way, this is the original referendum question that the SNP were proposing in 2007. Much more practical and balanced. Interesting that five years on they feel the need to go the other way...
The Scottish Parliament has decided to consult people in Scotland on the Scottish Government's proposal to negotiate with the Government of the United Kingdom to achieve independence for Scotland:
Put a cross (X) in the appropriate box
I AGREE that the Scottish Government should negotiate a settlement with
the Government of the United Kingdom so that Scotland becomes an
independent state.
OR
I DO NOT AGREE that the Scottish Government should negotiate a
settlement with the Government of the United Kingdom so that Scotland
becomes an independent state.
So is the current question the right question? Ach, well that depends on what you are trying to achieve. Would I have asked it that way as a professional researcher? No. Is the proposed approach ethical? Questionable. But if you are trying to drum up support for Scottish independence it is pretty well framed, so good on the SNP for being a bit sneaky there - I have a certain respect for their cunning. And if the no vote wins, that implies a no to all the other more ethical less idealistic questions too. So this is them giving it their best shot...
Ruth Stevenson runs Ruthless Research. This article first appeared on her blog.