There is perhaps no better way to describe the spirit and intent of a program review. A review should be open-minded and be willing to challenge any and all assumptions guiding the program design and delivery; getting, losing, keeping, and possibly casting away.

A program exists to educate students over a number of terms and years in a concentration way that results in a collection of ‘learning outcomes’ that can include a variety of aspects. The outcomes can be theoretical and abstract concepts, applied skills and knowledge, outcomes that relate to the how and the what, and can also include complementary skills and knowledge that will assist the student upon graduation. This is why students enlist in the program, and why society feels that the program is worthy. Each program will have a different mix of outcomes, with some programs being more theoretical, conceptual, applied, general, specialized, etc., and it is the mix that provides the program identity and what the stakeholders value. A program will not exist if it is only the instructors who value it, students must value it and society must also value students with these outcomes.

Reviewing a program requires a number of cognitive skills, including intimate knowledge about the current program and how to conduct a review. Conducting reviews and doing evaluations are not often part of an instructor’s regular training and education. They are experts in their own domain but this expertise does not necessarily transfer to other areas such as process evaluations and reviews.

Our summary observations...

• Trying to do a program review via a 1-2 day retreat with a large number of faculty members is not likely to yield a quality result. It might be a feel-good activity but problem-solving research suggests that such events are not the place to actually solve problems with a few minutes of thinking. You will come up with ideas and things to action and move forward on, but the foundation is likely to be rather swampy or smell of taurus excretus. Professionally done reviews outside of higher education can take weeks or months to perform by trained professionals; not 1-2 days by people with the best of intentions but who are usually amateurs at conducting a review at the end of the day.
• Major problems require major thinking and time. Retreats and monthly ‘whole committee’ meetings are not where problem solving should be attempted. Kahneman’s System One thinking will dominate and there will be little System Two thinking (Kahneman 2011). Quality problem-solving is not often a group sport (it is a small team of qualified individuals who will do a better job in almost all cases).
• Retreats and open forums are subject to many of the biases and issues noted by Kahneman et al. (2021). They should not be used for making decisions, especially at the end of the day.
• Have everyone submit a 1-2 page brief on the program as they see it; dealing with the who, what, where, when, why, and how. What are the strengths, weaknesses? What should the outcomes be?

• It is important to understand the assumptions driving the decisions and we suggest that these be documented and shared.
• It is better to strike a subcommittee with the appropriate representation, do a thorough review, and then present the results to the faculty members for discussion and debate.
• The results can be shared progressively as the review is done and need not be done via the big bang theory. For example, start with the vision, mission statement, the strategic and get that in place. Then move onto the next level of design.
• The review team must also be open to appropriate, quality suggestions and changes backed by evidence. If necessary, the review team might have to go back and validate assumptions and seek more insights from the stakeholders. A defensive posture will not be helpful.
• It is important to start with the raison d'être, 生き甲斐 (ikigai), the reason for being, the reason for existence. If you do not revisit this as part of the review, you are making many assumptions about what is needed, what has been delivered, but and what should be delivered. Hubris and over-confidence can lead reviewers off the cliff. You will do what we see students often do; great answer to a question that was not asked.
• This is a review and not a new program design exercise. This implies that the program has been running, had desired/stated outcomes, outcomes that should have been measurable; outcomes for while students were in the program and outcomes for students upon graduation.
• The review should work with alumni to understand what happened and what is happening. Did the program deliver the desired outcomes, and were the outcomes appropriate for the students during the program and upon graduation.
• What have students done with the knowledge and skills baked into the program? How many of the students have used the knowledge, concepts, methods, thought processes, theoretical, abstract, and applied immediately after graduation and 3-5 years out? When, where, how, why, what, who? The frequency and impact of the outcomes will either validate or invalidate existing assumptions and design criteria.
• The review should start with the desired outcomes, understand the current outcomes and do a gap analysis.
• The gap analysis will inform the review about the opportunities for change: the getting, the keeping, the losing, and the casting away.

• If the reviewers assume that they know what is ‘needed’ in the program based on limited knowledge (e.g., 'I talk with several alumni', biased exit surveys, etc.), the review will not be effective. The review should be evidence driven and criteria based. Not opinion driven and done by whimsy and popularity ‘likes’. Be aware of what is actually measured in an exit survey and what is not. For example, the students might be happy with the final degree because of the job opportunities but not actually value the courses or processes.
• Reviews need to deal with legacy, vested interests, personalities and opinions. If you do not teach xyz, they will learn nothing. You cannot drop ‘MY’ course. Abc has been taught for decades. Other programs are teaching asdf, we must too. What will I teach if we change or drop jkl? Making changes will be akin to sandbox politics in the playground.
• There are many ways to stifle and undercut innovation and change. Some people will be afraid of being the first, creating a precedent. Others might be afraid of the extra work and impact a change might have on their own activities. Many reasons to keep the status quo. Many will be over confident, exercise their loud voices, and supporting cliques.
• The review team and senior management in the department/faculty must have the appropriate analysis and negotiation skills, or bring in a facilitator if significant changes are necessary.
• Start with the end outcomes and backward chain through the program, cascading the required learning outcomes, term by term, course by course. Every course should know its place within the vertical and horizontal scheme. If something exists that is not in the driving outcomes, either the driving outcomes might need revisiting, or the course (or course topic) needs to be seriously considered for elimination or adaption.
• Every course and major learning outcome needs to be rationalized and justified based on the driving program level outcomes. Courses should not exist just because an instructor thinks that they are fun to teach, matches their research interest, or that they think that it should be taught. If innovation is being done, there are different ways to deal with the pilots, controlled risk, and appropriate oversight (e.g., the course review level). The personal level innovation type topics should be avoided in a program level review.
• Part of the review needs to be a plan for change, with pacing and appropriate instructor support for the changes. The plan needs to address what learning and preparation is necessary, how to do pilots for anything truly innovative, the resources and effort needed, etc. The simple phrase of ‘make it so’ will not be sufficient.
• The number of concurrent changes needs to be controlled and a common mistake is trying to do too many changes, too quickly, and too many at the same time. Instructors will need time to adjust, as will the infrastructure that supports the courses and outcomes. For major program changes, it might be a 3-5 year plan. C’est la vie.

Further reading

• Kahneman, D. (2011) Thinking, Fast and Slow, Farrar, Straus and Giroux.
• Kahneman, D., Sibony, O. and C.R. Sunstein (2021) Noise: A Flaw in Human Judgment, Little, Brown, Spark.