There are two aspects to course quality: maintaining and sustaining a certain level of quality, and reaching the desired level of quality in the first place.

To maintain quality in either a service or production situation, there are well established concepts we can apply. See the American Society for Quality (ASQ) site for many ideas, tools, and methods to use. For example, there should be regular, repeatable sampling to identify any trends in the process. Have clear definitions and targets for outcomes and know how to quantitatively or qualitatively measure them. Understand the qualities that make up the quality. Do a root cause analysis for areas of less-than-desired quality. There should be a clear understanding what can be ‘controlled’ and what cannot be, and it is important to understand the types and sources of variance. There are many systematic and well-thought out methods for understanding and improving quality in processes such as teaching. In summary, using a quality management lens helps to identify the qualities of a course that matter, how to set targets for the qualities, how to achieve the targets in a systematic way, and how to measure them. Although you might not have studied quality management as part of your academic training, there is no time like the present to learn some of the key methods and ideas about how to manage quality in your profession!

There are also important principles that help when introducing change and one principle is monitoring the periods just before the change, during the change, immediately after the change, and after the change has stabilized. This monitoring might require specialized tools and methods, and frequent sampling; and might be different in each of the four periods.

If you as an instructor are trying to teach a course, you should know what you want your target effect to be, BUT there are several contributing factors that lead to this overall effect. What exactly is leading to a quality course or target? The starting point for any quality analysis or control program is knowing what you are supposed to deliver. What the qualities are that define an effective and efficient course and what targets do you, your TAs, your students, the department, faculty, institution, accreditation bodies, and society have for these qualities. It is likely that all of these parties have a different view of what qualities matter and what the expected quality level should be. It is also likely that most of these parties have never really thought about it too much. In any discipline, we should know what we mean by 'qualities', what the targets are, how to measure a quality, and how to interpret the results; that is, we should use a scientific lens when approaching teaching and the quality of teaching.

Observations...

• Start somewhere, preferably from the top down (e.g., institutional definitions, then faculty, etc.). In reality, you might have to start with your own course in a quasi-vacuum.
• With a mentor or peer, discuss what the qualities of an effective and efficient course would be. This will likely cascade into the qualities of an effective and efficient instructor, student, Teaching Assistant, etc..
• Group and organize the qualities into major groupings and themes. The Ishikawa/Fishbone diagram and the KJ method (affinity diagram). are a good places to start. Another useful resource is the description about root cause analysis.
• For each major theme or quality, review the points discussed and ensure that you have points that identify what makes the major quality effective and efficient (e.g., the subcomponents or sub-qualities). These major and secondary qualities are what you attempt to measure and analyze.
• Some of the qualities and sub-qualities can be measured quantitatively and others will be qualitative. Some can be measured directly and some will need a proxy or indirect indicator.
• For understanding, you can attempt to measure all aspects – perhaps not all every year, but there should be a regular tracking and reflection of how things are going.
• For evaluating the actual quality and holding the instructor accountable, the instructor should be held accountable for what the instructor has control over. It is unfair and inappropriate to for an instructor to be accountable for something out of their control. Thus, the department must understand what the instructor should be held accountable for.
• Use non-parametric statistics or methods for ‘small’ samples if you are going to do any numeric analysis and ensure that all of the necessary assumptions of the method are satisfied.
• Listen to all stakeholders and obtain their perceptions and opinions. This is important – do not dismiss or exclude any participant. However, be aware of the possible biases and inherent issues. See the point below!
• If opinions and perceptions are being used for measuring quality, be aware of the assumptions and biases of the source – including the qualifications and credibility of the source. Do they know how to assess or evaluate? Do they understand what they are evaluating? Are they qualified to provide an opinion worth listening to? Do they even know what to assess or what learning means? Do you know what the baseline is that the source is using for any comparison? Are the source’s expectations and baselines reasonable? Listen to all, but believe no one until you can triangulate or have secondary sources of evidence.

Co-dependencies & The Necessary and Hopefully Sufficient

• Knowledge can usually be ‘tested’ immediately and is a cleaner quality to measure. There are issues, but it is easier and cleaner than probing cognitive skills.
• Cognitive skills can be probed to some degree during a term, but if there is real learning of a deep level, the students might not be aware of what they are actually learning; in other words, they are not usually self-aware and do not recognize the relatively slow and incremental changes in cognitive skill. Knowledge is more definitive and easier for the students to understand; they memorized something and hence they learned something. They are not necessarily the best judges of their own abilities to think; comprehend, apply, analyze, synthesize, or evaluate. Many of these are subtle skills and while it might be possible for the instructor to see and measure a change (e.g., problems are progressive during the term in small ways), the students might be oblivious and think that they have learned nothing. Even if the instructor is transparent and discusses this phenomenon with the class, the students are not likely to recognize any improvement or change in their own thinking; unless the instructor sits down one on one with each student and debriefs. This is another reason why student evaluations of a course/instructor are ineffective measurements.
• Without the clear outcomes and definitions/targets for the course qualities, it is hard to say if a course has quality or not. If the institution expects certain knowledge to be learned, this should be clear. If the institution expects certain skills to be learned, these too should be clear. If the institution expects the students to ‘like’ an instructor, this should also be clear. It should be clear how much of these contribute to the institution’s definition of course quality. For example, if the traditional student course evaluation is used, it should be clear what the evaluation actually measures (which is not an objective measure of what was learned) and how much this should be weighted by the department.
• Our own suggestion is that the usual student perception survey be used for no more than 15% of an instructor's teaching quality. It should not be dismissed totally, but it should not used as the definitive statement either. There are many problems with student evaluations and they must be used wisely. That being said, we have seen many departments use them as a primary assessment of teaching and the expected occurs - the instructors teach to the evaluation instrument. There are other mechanisms that can be used to help assess teaching; portfolios, course reviews, and in-class observation come to mind. Nothing is perfect, but if done seriously and with expertise, they will provide insights into the instructor's skill and development. One size does not fit all!
• The institution’s objectives should also be applied with wisdom and balance; e.g., using the insights from the Bloom report on what students will like or dislike when the various levels of learning are probed. A knowledge base course can have higher expectations of student satisfaction while it will be harder for a course that pushes students out of their comfort zone on one or more cognitive skills.
• We want to make a specific note about the possible biases in student perception evaluations. We will not repeat all of the points, but Kahneman (2011) is a good place to start to understand the decision processes that might be behind a student's evaluation. Kahneman et al. (2021), provides an excellent overview of biases that can create problems with response quality. Prejudgment, anchoring, lack of comparator baselines, and what is called occasion noise all come to mind.
• It is important to know what is actually being measured, how it should be interpreted, and if the measurement should be discounted because of low sample size, pre-existing student knowledge and skill levels or pre-existing behavioral issues. That is, any measurement should be valid and it is important to know of any dominating factors or biases in the measurement. For example, does it make sense to record or publish a course evaluation when 6% of the students completed the evaluation? What should the thresholds be for ‘quality’ assessments? How should poor assessments be used? Shared?

Tips for Beginners

• Avoid the trap of making too many changes at once. It may take several offerings of a course to get it where you want it to be and that’s ok. If you want to measure the increase in quality brought by the changes you need to know which change is having the impact.

Further reading

• Academic Quality Work, Tomorrow’s Professor Postings, Stanford University. https://tomorrowsprofessor.sites.stanford.edu/posting/891
• Kahneman, D. (2011) Thinking, Fast and Slow, Farrar, Straus and Giroux.
• Kahneman, D., Sibony, O. and C.R. Sunstein (2021) Noise: A Flaw in Human Judgment, Little, Brown, Spark.