Online learning and survey tools have made it very easy for an instructor to poll and ask students questions. Institutions also send out polls and surveys (the university, faculty, department, units, etc.). It is so easy!

In the social sciences, there are courses on how to create a good field instrument. There are additional courses on how to interpret the data. There are qualitative and quantitative techniques. There are advanced courses and there are many thick texts on how to do field research using surveys and such. It is not easy to do well. If one reviews many of the courses, the content focuses on what you would expect; the testing of hypotheses, analysing quantitative ‘answers’. While there is some, there is little guidance on how to deal with self-reporting and with perceptions.

To illustrate some of the possible biases and issues with timing and phrasing, we suggest you look at Kahneman et al. (2021). It provides a good summary for how answers and decisions can be affected by a variety of factors.

It would be rare for individuals in certain disciplines such as STEM (e.g., Engineering, Math, Science) to have appropriate and substantive training and experience in qualitative, field driven, empirical research. They would not likely have any training in how to phrase questions, design scales, design in checks and balances, the use of proxies, avoid bias, etc. Unless one is very lucky, it is not possible to design a quality field instrument without learning the trade; we heard one individual claim ‘I did a lot of research Saturday morning on how to create a survey and this is what we are going with’. Hmmm.

Suggestions and observations…

• The design, deployment, analysis, and interpretation of field instruments involve cognitive skills and it is important to learn, practice, and develop the necessary skills to the desired level. This suggests that one should be cautious about the early tools designed and executed. It is very unlikely that the first attempts will be of high quality.
• For respondents to take a survey seriously, they must understand the value of doing the survey and there should be some kind of direct, personal (or group, organizational) value they can see. They need to feel that answering the survey and answering it well is worth the effort.
• The results of the survey or poll should be actionable. What are you going to do with the results? Do the questions give you that clarity?
• Respondents need to feel that their responses are valued and will be listened to. There will be expectations of ‘use’ and potential change or impact arising from filling in the survey.
• Respondents need to have shared definitions and baseline comparators. Otherwise, you will have apples compared to oranges.
• We prefer odd numbered Likert scales (e.g., 5, 7, or 9) unless there is a clear dividing point. If there is any chance of a middle case, use an odd number and do not try to do a false ‘make a choice’. We usually use 5, but have been known to use 9 when warranted.

• Self-reporting is a problem and it is important to build in additional checks, proxies, and balances to test for validity.
• Perceptions based questions are a problem and it is important to build in addition checks, proxies, and balances to test for validity.
• Proxies can be based on secondary actions or situations. For example, instead of asking them if they think of themselves as an innovator, you ask them a series of questions based on ‘how many times in the last year did you do xyz’ and use these types of questions to be more quantitative and less perceptual. While it is not possible to do this in all cases, it is possible in most.
• Scales should be consistent; high values on the right, low on the left.
• Sometimes, it is important to state a position or point (or question) in rather ‘academic’ or ‘abstract’ phrasing and then provide an example or context to allow the reader to understand the point in context; especially when different parts of the respondent pool might have different baselines or points of reference.
• Never use inflammatory language in a scale, or cute adjectives; keep things neutral.
• Avoid inflammatory language in the questions (including flippant, sarcastic, ironic sentiments).
• Avoid using biasing adjectives in questions like ‘how good do you…’; it is better to position the question or point in a neutral way, and then refer them to a scale. Including qualifying adjectives introduces bias. Generally, a no-no.
• It is sometimes good to have open, general questions such as ‘what are things that should continue or increase in the course’, ‘what should be reduced or removed’, and ‘what should be added to the course’. It is important to encourage students to explain ‘why’. We prefer to avoid saying ‘the top 3’, and just let them write as much or as little as they wish. If you say ‘3’, you are creating an expectation for ‘3’ even if the student can only think of one, and the student might have more than three points to make. We also prefer to ask some specific questions in addition the general, open ended ones.
• If possible, help the students with creating a baseline or comparative benchmark; that will often improve response quality. If you are asking the students to compare something subjectively, can they do this? Are you going to get good data? Can the students actually answer the question? First years? Seniors? In fact, does the same survey make sense for first years and seniors?
• There can be temporal locality bias. This is when the timing matters. For example, asking for a course critique after a midterm, when the results come out, or after some kind of issue arising with the class.
• Timing also matters when the questions are asking about short term or long term learning. For example, in certain types of courses, the learning outcomes are not realized or perceived until the very end of term, or after the term. A survey in week nine or ten of a course would not yield good data in these cases.
• Survey fatigue can be a factor. Too many, too often can annoy and reduce the responses and response quality.

• Be transparent and explain how the scales are interpreted and used. For example, we know of one course evaluation that asked the students to use A-E as their ratings. The students thought E was 50% while the departments and faculty used E as meaning 0. There is a BIG difference. The students thought the scales were in increments of 10, not 20 and when they thought they were giving an instructor an 80%, it was not interpreted as such.
• It is possible to innocently introduce subtle bias depending on the context. Certain questions used across a campus might find respondent bias based on faculty. For example, how age and gender expectations play a role in some disciplines and not others. An innocent question can create unwanted and unexpected bias. A one-size-fits-all tool rarely works across a heterogeneous population when there are possible biases within the demographic groupings.
• You have to think about the quality of the respondent and if the respondent can give you high quality, valued answers. For example, if a student does not come to class or does the minimum amount of effort just to ‘pass’, is their voice to be considered equal to another student who actually comes to class, preps for class, does the homework, goes to the tutorial, etc.? Not all respondents are created equal and if it is not possible to sort this out in the sample, it is important to remember this when analysing, interpreting, and using the results.
• Certain surveys will require educating the respondents (e.g., students) and the individuals receiving the results (e.g., instructors, department, faculty, university).
• The power and quality of the instrument needs to be thought about; it is not wise to use a strong method in a weak situation, or vice versa. Depending on what you are trying to do, there are standards for things like the number of respondents in order for the analysis to be considered robust and appropriate. If the instrument may have potential impact on one’s employment, financial security, or reputation, then the highest expectations and standards should be in play.
• Sloppy and amateurish attempts with far-reaching consequences should not be considered acceptable.

Further reading

• Kahneman, D., Sibony, O. and C.R. Sunstein (2021) Noise: A Flaw in Human Judgment, Little, Brown, Spark.
• Parsons, C. (2007). Web-Based Surveys: Best Practices Based on the Research Literature. Visitor Studies, 10, 1, pp. 13-33.
• Artino Jr., A.R., La Rochelle, J.S., Dezee, K.J., and H. Gehlbach (2014). Developing questionnaires for educational research: AMEE Guide No. 87, Medical Teacher, 36, 6, pp. 463-474.