Course evaluations are an emotional topic. In this section we will try to take an Engineering approach to the problem inspired by what Engineers know about process evaluations. A course is essentially a process and the fields of Industrial Engineering and Management Sciences have been evaluating processes for over a century.

• A process has an objective and in the case of a course, we have the learning outcomes or objectives. The learning outcomes should be aligned to Bloom’s cognitive complexity taxonomy as this helps identify how the outcomes can be assessed. For example, the lowest level of ‘knowledge’ used in comprehension can be easily assessed while the intellectual ability layers are problematic for many reasons.
• Learning outcomes should address what is the learning objective (the what), and how this is defined within the context of the course (immediate proof of mastery), and how this is defined beyond the course (retention and application).

A process has inputs (students), the actual activities associated with the process and materials used in the transformation of the raw inputs (learning activities and learning materials) and some form of mechanism that guides the transformation (the instructor).

Processes are often evaluated on two dimensions: effectiveness and efficiency. Effectiveness focuses on the ‘what’ and efficiency on the ‘how’. In addition to the primary objectives associated with learning outcomes, the institution can set additional expectations for both efficiency and effectiveness. A holistic approach to a process at how the incoming inputs are dealt with, how the transformation is performed, and how the outcomes are assessed. Rarely is any assessment in a process evaluation black and white; there are degrees of grey everywhere.

A complete process evaluation also assesses the learning outcomes; are they aligned with the program curriculum, and are they reasonable and feasible given the ‘normal’ knowledge, experience, and learning skills of the students taking the course. Having ill-conceived expectations for a course creates a situation for failure; before the course begins.

A process evaluation listens to all stakeholders and looks at all of the key transformation elements, but is aware of possible biases and issues with the stakeholder feedback. For example, what is the institution expecting? What are the students expecting? Are the expectations aligned with reality?

If a process evaluation is done repeatedly, one can set incremental goals for improvement. This assumes that you know what you have, what you think should have, and know how to bridge the gap.



A process evaluation is aware of spikes and unusual factors and deals with sustainability issues. 

Thoughts…

• Clear expectations are set for what is measured and how the measurement will be weighed. If the students are learning what they are supposed to learn, how much does this contribute to overall course quality? If the students are supposed to like the instructor, how much does this contribute to course quality?
• Multiple peer observations/assessments should be conducted when a course is being developed/re-developed or when a junior instructor takes over a course. At tenure and promotion points, a full evaluation can also be considered, and then every 5-6 years after tenure/full.
• When a course is having a full evaluation, peer observations/assessments should be considered mandatory. There should be multiple observations; the first lecture, one or two samples during the term, and as the course wraps up.
• The peer reviews should be conducted by a team of credible observers. This implies that one observer is someone who knows pedagogy, perhaps another observer who knows the specific course content, and perhaps someone from the department with a broader view of teaching expectations, program flow, and curriculum.
• The course package or portfolio needs to be reviewed for clear learning outcomes that will be conveyed to the students and to ensure that the learning outcomes are reasonable and feasible.
• The course package needs to be reviewed for student load and effort. Certain courses can be abusive to the students and set excessive time and effort expectations.
• On a yearly basis, between full reviews, quantitative assessment results can be combined with student feedback and a rough indicator of course quality determined. Note: the student feedback needs to be explicit and aligned with the stated goals for the course (e.g., the likeability factor).
• If the assessments are radically changed, they should be reviewed; e.g., to ensure that easy assessments are not being used to fudge student feedback, that the load factors are reasonable.

Co-dependencies & The Necessary and Hopefully Sufficient

• A process evaluation often gathers qualitative and quantitative evidence about these elements and potential biases must be consciously addressed.
• Quantitative metrics must be assessed. Do the metrics actually measure what they pertain to measure? Are the quantitative metrics actually associated with the stated learning outcomes? What are the possible measurement errors? Are there any biases?
• Qualitative metrics (perceptions, beliefs, self-reporting, opinions) can be useful when the institution’s expectations for a course include objectives such as ‘the students must like the instructor’.
• Assessors must be assessed as well. What can the assessors assess? What are the biases and possible assessment errors associated with the assessors? What is the baseline that the assessor is using for the assessment? What does the assessor know about assessing? Can the assessor reasonably and feasibly provide a high-fidelity assessment?
• The assessment of a course should be composed of clear expectations with respect to outcomes; the process needs to be designed to deliver the expected results. This should be clear with regard to weighting (e.g., quantitative measurement of learning – 60%, instructor’s likeability factor – 10%, etc.). Unfortunately, institutions are now in battles for popularity and students must be attracted based on factors other than how much they will actually learn. The course evaluations and corresponding instructor evaluations must deal with this reality and if learning must be sacrificed for ‘fun’, ‘enjoyment’, and ‘edutainment’, then so be it. This should not be buried and should not be swept under the rug. Instructors should know exactly the compromises that must be made as the institution is ultimately responsible for what a student learns and not the individual instructor.
• Issues with self-reporting, biases, and measurement error need to be addressed.
• Courses need to be evaluated ‘in situ’. A junior foundational course is inherently different from a senior course that is skill based and focused on open-problem solving; an elective course in a major will be viewed different from a core course in the major; an elective course not in a major will be viewed differently. There will be different expectations.
• Expectations must be tailored to the course; knowledge or skill based, junior vs senior. The more skill based and more rigorous the course is with respect to deliberate practice and learning through failures, the less likely an instructor will be ‘liked’ during the term. The more knowledge based with a reliance on memory and recipe following, with few surprises and little uncertainty; the more the students will like the course. A course that resets the comfort zone must be assessed differently.
• The course environment is also important; the instructor is limited by the facilities to some degree (see. Zandvliet 2014).

Further reading

• Hounsell, D. (2009). Evaluating courses and teaching, Chapter 14, pp. 198-213. A Handbook for Teaching and Learning in Higher Education Enhancing Academic Practice, Third edition, Edited by H. Fry, S. Ketteridge and S. Marshall, Routledge.
• Hildebrand, M., Wilson, R.C. and E.R. Dienst (1971). Evaluating University Teaching, Handbook, Center for Research and Development in Higher Education, UC Berkeley.
• Thomas, J.E., and C.R. Graham (2017). Common practices forevaluating post-secondary online instructors. Online Journal ofDistance Learning Administration, 20, 4.
• Zandvliet, D.B. (2014). PLACES and SPACE: Case Studies in the evaluation of post-secondary, place-based learning environments, Studies in Educational Evaluation, 41, pp. 18-28.