10.10 Student Perception Surveys Of Teaching

There are times you might want to survey or poll the students. There are also the cases where the institution or others are surveying or polling. For example, student perception surveys come to mind.

Lots can be written about survey design, how to phrase a question, what scales to use. We are not going to repeat all of that. You will not learn how to write a high quality survey or poll in a few hours. Not possible. However, we will share with you some common issues and points to think about; when you are creating one, filling one in, using the results from one.

Most important, do the questions lead to something actionable? The analysis should result in something actionable; does it? Is the question and answer set specific enough to do this? And, actionable by whome? The instructor, department, faculty, institution?

Often the questions might be beyond the average student. For example, answering about understanding and reflections about the process requires a certain level of self-awareness of what is meant by understanding and being very aware of the process, and not just turning up, going through the motions. There is also a maturity and expectation setting from other situations. Consider the following. Some students think learning is having something to memorize and recite. They confuse cognitive skill development with facts and recipes. They also confuse effort with quality and results. The current generation requires and expects significant different types of interactions than pre-millennials. If you do not answer their question, you are rude. If you do not give them the answer, you are not doing your job. All of these and more issues can be hidden or under the radar when asking about perceptions about what has happened and how they feel about it.

The timing of the instrument is critical as is the implicit implication possibly imbedded in a question. Responses can be biased by recent events, but the survey may be asking about things yet to come, not yet realized. For example, in some cases, the students should not be aware of the teaching methods till the end of the term during a debrief. If constructive failure is used and deep learning is used with deliberate practice, there are some things you want the student to know during the process and other things that should be shared during the debrief. The quality of the course and what is actually 'learned' might not be apparent to the student until after the final project or assessment is done, or even later when the learning is actually revealed in a subsequent course or after graduation. The context of the question is important.

Kahneman et al. (2021) provides a great summary of possible biases and issues when decisions are being made. Surveys, especially perception surveys, need to be crafted and used keeping these biases and possible sources of error in mind.

There is something called the Net Promoter Score that is used in business to pick up customer loyalty. This can adapted and used a perception survey if one goal is to find out how much the students valued the course. Did they value it enough to recommend the course to others, or would they take it again?

Why is a question being asked or survey done?

• Establish a baseline
• You might be doing it for future reference; for analyzing relative impacts of planned or unplanned interventions.
• Weakness focus
• Someone thinks that there is a perceived problem and thinks a survey will help understand the issues, the problem.
• Is it a problem?
• How big of problem is it?
• Is it still a problem? For how long?
• Is the problem getting worse? How much?
• Who are the stakeholders impacted by the problem and by how much?
• Has there been a problem intervention performed? How did that go?
• Strength or verification focus
• Someone wants to do a survey after the fact. We wanted to do ‘more’, broader and deeper. Did we?
• Is the effect what we were expecting?
• Are there unexpected results?
• Is it past the initial honeymoon period?
• Is it being sustained?
• Are there potential trends arising with sustainability, growth?
• What do we need to do more of it? Even better?

Some other thoughts...

• Is the question understandable by the student?
• Does the student understand what is being asked?
• At what level of understanding can the average student expected to be at? Comprehension, application, analysis, synthesis, evaluation?
• What is the shared understanding and definition of the what the question is talking about? What is the variance within the class?
• Does the student know what the answers will be used for? The impact of their answers?
• Does the question matter to the student?
• If a question is perceived to be irrelevant, ‘stupid’, ‘dumb?’, or ‘really?’, it is possible that the whole instrument is dismissed as a waste of time. It will not be taken seriously.
• If a question is not seen as possibly making a ‘difference’ or ‘actionable’, it is also possible that the instrument will not be taken seriously, a waste of time.
• Is there a shared baseline for comparison?
• If responses are being averaged across students, does averaging make sense? Do comparisons make sense? Perception instruments often make sense at the individual level, but might not across respondents. Were the students using the same baselines? Same criteria? Same expectations?
• What were the ‘expectations’ at the beginning of the process being evaluated? Were there common expectations, common goals, desires? What is the variability?
• Bottom line, do the students really, REALLY know how to evaluate? You are getting answers and perceptions. But are they useful? Valid?
• Biases?
• Social Comparison theory? Non-conformists? Outliers? Outperformers? Tribal behaviour?
• Race, gender, age, language skills, etc.?

• Imbedded in the question? The language? The implications?
• Order of questions is extremely important in perception surveys; where the ‘taste’ from preceding questions can linger and focus the thinking of those following.
• Kahneman et al. (2021). This is a must read if you are doing any of this stuff, or if you are worried about the quality of decision making in general. Lots of good points on biases!
• Student knowledge and expertise?
• What skills and expertise the student needs to be able to give a high quality answer?
• Can the average student give an ‘average’ quality answer to the question?
• Can the student discriminate a response on a scale?
• Does the student understand the scale correctly and what the scale implies? For example, in the one survey, many students thought ‘E’ was a 50 and the scale went from 50 to 100, not realizing that the scale as used by the faculty was 0-100.
• Variability in the total population?
• Can any of the responses vary by age/position of student? E.g., junior vs senior student.
• Can any of the responses vary by demographics? E.g., students in a core, program course versus an elective in the program, versus an elective in the department shared by two or more programs? Versus an elective in a different faculty?
• Do any of the criteria vary by ‘when’ the survey is executed? E.g., amount realized, amount to be realized, temporal effects, recent events biasing baselines, ‘emotions’.
• Constraints in instrument structure?
• If there are limits to the number of questions, is each question yielding the ‘same’ power of clarity and information? Universal to the majority of students? Is there survey fatique?
• Is there is constraint as to when the instrument can be executed?
• Is there a constraint on who can be surveyed? How often?
• Repetition within instrument?
• If a survey is not limited in size, it sometimes wise to add questions which ask the same thing two or more ways to check for response consistency and integrity.

• If the survey is limited, repetition should be avoided if there are ‘better’ questions to ask.
• If the instructor has access to the information via other forms or proxies, there is limited value in a limited survey to ask redundant questions.
• Is compression possible?
• Often in perception surveys, it can be boiled down to one ‘event’ or ‘situation’ that can taint all of the responses within a threshold.
• One or two questions hitting a core issue can sometimes be representative of a whole bank of questions trying to tease out factors. What mark do you expect to get in this course? Are you happy with this mark? What mark do you think you should get in this course given the effort you are putting in? For example, student perception surveys can be heavily weighted by the last assessment, the individual’s performance, friends’ performance, and the class performance. Are you happy with your last mark can often be used as a proxy for picking up possible bias and for judging general happiness.
• Independence of action by respondent?
• How much does the individual’s own actions contribute or relate to the question’s outcomes?
• That is, is the respondent partially (or more) responsible for the perception being asked about? For example, if a student is not reading the material, doing the suggested practice questions, not attending class, etc., they might have a bias and be part of the problem when they do not like certain things about the course and how they are turning out.
• It is also possible that these types of questions can be picked up by those seeking ‘ways out’, a reason for doing nothing, and thus ignoring other, possibly very valuable, feedback. 

Further reading

• Kahneman, D., Sibony, O. and C.R. Sunstein (2021). Noise: A Flaw in Human Judgment, Little, Brown, Spark.
• Parsons, C. (2007). Web-Based Surveys: Best Practices Based on the Research Literature. Visitor Studies, 10, 1, pp. 13-33.
• Artino Jr., A.R., La Rochelle, J.S., Dezee, K.J., and H. Gehlbach (2014). Developing questionnaires for educational research: AMEE Guide No. 87, Medical Teacher, 36, 6, pp. 463-474.