Surveys are not realistic portrayals of reality, as has been proven once again in the past few months by the unexpected victories of Donald Trump and the Brexiteers. The goal of surveys, which also cover public opinion polls, is to represent reality as accurately as possible.
No matter how important it is to get a good sample to represent the population as accurately as possible (and most criticism on public opinion polls seem to focus on this aspect), the measuring tool is equally as important.
Only by using a reliable and accurate measuring tool will it be possible for a researcher to make well-founded statements regarding reality. Many errors can be made when designing a survey. To illustrate common mistakes, below we will evaluate the survey the Universiteitskrant used recently for the ‘Big Black Pete Survey’.
Crude language
Actually getting respondents to fill in a survey proves difficult. You should not show contempt by using disrespectful and crude language. Friendly and respectful language use is important not only when greeting respondents, but also in how questions are formulated.
For example, the use of a word to ask the respondent a more complex question is an indication for a respondent that you are not taking him/her seriously. In case of the formulation of the questions below, question 2 could be reformulated to: ‘At which faculty are you a student or employee?’
Are you in favour of or against the Black Petes? Which type of Black Pete is intended here? And why ‘the’ Black Petes? Subtle differences in how questions are formulated can have major consequences for the obtained results. This is why preliminary research to determine how respondents exactly interpret a specific term is vital.
The Dutch version completely lacks any clues that indicate to the respondent what version of Black Pete is intended. The formulation in English seems to be a bit clearer mainly due to the fact that ‘traditional depiction’ is used, but even then, the term ‘traditional’ can be interpreted in many different ways, because traditions are often subject to change.
Too long
The formulation for question 6 is too long, but its translation into English is also incorrect. The complexity of the formulation makes it more probable that respondents satisfice more often. Satisficing is an answering strategy used by a respondent when he/she wishes to complete a survey with as little effort as possible.
Complex formulations of questions take extra effort to read and interpret, which discourages people who fill in the survey with minimal intrinsic motivation from giving an optimal answer.
If a survey is done in person, then there is a possibility of socially desired behaviour. In that case, an interviewer has more insight into the provided answers. If the questions are not answered anonymously, then respondents often give the answer they feel is socially desired.
In case of the Black Pete Survey, being in favour of changing Black Pete could be viewed as a socially desired answer, for example. If an interviewer was present when this survey was done, then it is likely that his/her presence influenced the obtained results.
Mistakes
Apart from the oft-made mistakes mentioned above, there are a number of other mistakes researchers make when designing a survey. That is why you should always contact an expert when you are designing one. An expert could, for example, recommend that you do a pre-test for the survey.
During this pre-test, different methods can be used for determining whether the formulation of questions is correct, whether the conceptualisation of used terminology can be interpreted by respondents, whether the answer alternatives fit the questions. No matter the effort and expertise a designer puts in a survey, in the end, only respondents can assess whether a survey works for them or not.
Yfke Ongena (y.p.ongena@rug.nl), university instructor (Survey Research Methodology)
Teunis Dokter (t.dokter@rug.nl), former chairperson Lijst Sterk in the University Council