11.3 Process Evaluation

Process Evaluation SoT-11-3-ProcessEvaluation

A high quality process evaluation is worth the effort. It will take time, effort, and someone with skill and experience.
You might not 'see' the processesl; you might be looking and not seeing. That does not change the fact that most things in higher education are indeed processes; even if you do not call them processes or want to think of them as processes.
As professsionals, we should be professional about how we evaluate our domain.

How’s it going?

Many things in life can be viewed as a process. For example, how something is used is a process, and the steps along the journey is a process. Processes typically have outcomes at the end of the process, possible outcomes during the process, tasks and activities by which the outcomes are achieved, and internal, minor partial outcomes from one step acting as input to a subsequent step. Processes can have temporal dimensions, dependencies between steps, parallel and concurrent activities, or be laid out in a sequential manner. There are often constraints controlling the initiation of the step, what happens in the step, and when the step is considered to be done.

The students are in a process from first year until graduation; at the program level. There is also a process within each course. There are administration processes that must be followed. Instructors have a process in mind for what they and the TAs do during the term, in a lecture. There is a process used for marking. There is a process when you review a course or a program, or have a department meeting. Lots and lots of processes.

There are many tools available to help one design a process, to review a process, or to evaluate a process. There is a body of knowledge dating back centuries on these tools, research, best practices, courses, and many mistakes to be possibly made. There is rarely one right way to do these tasks, or only one tool that can be used. They all involve specific cognitive skills which require understanding and deliberate practice if one wants to move from ad hoc and amateur approaches to those which are effective and efficient. We urge caution when you first attempt a systematic and conscious approach to the design, review, or evaluation of a process and be prepared to do some small practice activities before you attempt them on something critical or serious like a program review, teaching evaluations, or course design.

We regularly use these tools when thinking about academic programs, course designs, course delivery, student learning, and how to evaluate teaching. We have also used these tools in a wide range of domains such as healthcare, law enforcement, search and rescue, manufacturing, retail, finance, 3.and the hospitality industry to name a few. This note focuses on the evaluation of a process, and there are separate notes on reviewing and designing.

Key observations…

As with any cognitive skill one can progress from the naïve and novice levels over many years and many experiences involving deliberate practice to levels of performance and thinking which exhibit expert levels of skill and mastery.
Many of the tools appear simple and trivial to use, but in reality, there are many subtle facets to comprehend and understand if they are to be applied or used in analysis.
Evaluations are inevitably a mixed methods situation where there is often data to be gathered and analysed, interviews, observational activities, and things like surveys. The evaluators require expertise in all of these methods.
A process evaluation can be done to simply document what is going on without any concluding observations, recommendations, or editorial opinions. It is important to understand what people are expecting from the evaluation.
To effectively and efficiently evaluate a process it is important for the individual or evaluation team to understand the general domain to be studied, the best practices in the domain, the common practices in the domain, the tools used for evaluation and to have experience with them. The evaulator needs to be able to know what could be versus what is, and they need to know the general norms and expectations. The evaluator needs to know the import of how things are said, and to also know what is not being said. This is why the evaluation cognitive skill is at the highest cognitive skill level.

It is important to do background analysis before the evaluation to understand the specific culture and situation that is being evaluated.
It is necessary to know what the constraints are guiding the process, and the authority and decision-making of various players involved with the process; namely what do people have control over which they should be held accountable for, and what they do not have control over.
It is important to have some kind of benchmark or model when starting an evaluation.
The actual evaluation looks at the usual who, what, when, where, why, and how. It looks at the implicit and explicit assumptions which are being made. Any constraints and their elasticity are also looked at; constraints that force things to be within a range of options, and constraints that force things not to be in a range of options. Relationships between the steps within the process and any co-dependencies must also be investigated. The activities and time periods before the process starts is important to include. As are the activities and time periods after the process. All of this is guided by the context of the process and a holistic view of the stakeholders involved in the process before, during, and after.
Process evaluations can themselves be evaluated and it is important to understand the targets and desired outcomes from an evaluation. The depth, breadth, and quality of an evaluation will dictate the level and skill required of the evaluators. It will also dictate the possible methods, elapsed time, and total effort required to do the evaluation.
It is possible to do a process evaluation against established targets for the how and the what; either the terminating outcome under your control or partial sub processes along the journey.
Part of the homework is to develop a clear understanding of the history of the process under review and the original objectives and desired outcomes, and how these objectives and desired outcomes might have evolved over time.
Another form of process evaluation is done using knowledge about best practices and what is considered acceptable normal practice as comparative benchmarks. This is sometimes necessary when the objectives and outcomes for the process are not clear, or when there might be a political agenda or other reasons why something is being looked at.
It is not uncommon for a process evaluation to be done for reasons other than trying to understand or help the actual process. Sometimes, there is a hidden agenda and the evaluation is being done in a Machiavellian way to support this agenda.
During a process evaluation, it is important to listen to everyone, but believe no one unless there is evidence to support the statements made. This evidence can be quantitative data, triangulation of perceptions and subjective observations, or through observation.
We know from cognitive science that individuals have a hard time explaining what they actually do, and in certain cases may not be aware of what they actually do. It is for this reason that the best practice for process evaluations involve an ethnographic component where the process is actually observed over a longitudinal time horizon.
In certain situations, it can take many weeks (perhaps 4-6) to develop the credibility and relationships necessary to get the necessary knowledge that will inform you as to the real process and what is going on. Before this, you will be receiving political answers and knowledge that is superficial and not very helpful.
There can be individuals who are very uncomfortable and unhappy with a process review because they have a vested interest in the process. They might have been involved in the original design, or the current process is convenient for them and supports their own agendas.

We often start off with a KJ, Ishikawa/Fishbone exercise that set up a greenfield or best practice view of the process and desired outcomes to help guide what is actually looked at during the evaluation. This can be considered a form of a maturity model that is used for the benchmark.
The Ishikawa spines help you organize your evaluation as to methods to be used, what needs to be evaluated, and gives you insights about what should be paid attention to during interviews, data analysis, or focused upon in surveys.
We also like to work with the local expert, someone with credibility, to help interpret the signals and observations which arise from the evaluation. The local expert may not be a real expert in the context of cognitive science, but they will be the relative expert compared to others in the situation. The expert might also have biases that you have to be aware of during the evaluation.
The local expert is also important when writing up the final observations and recommendations so that you can avoid making stupid observations, and infeasible and unreasonable suggestions.
It is very important to avoid getting caught up in any political situations.
Do not share or voice any of your own opinions and thoughts until you actually know something and have thought it through. It is easy to destroy your credibility with early rush-to-judgement observations and suggestions.
A process evaluation might have an initial phase that sets the scope for the full study, or summarizes the situation. These initial phases and the reporting can be used to check your knowledge, your vocabulary, and help make sure that you are on the same page as the people who will be using the results of your evaluation.
If the stakes are high, be self-aware of potential overconfidence, and faux belief in one’s ability to actually evaluate the process under analysis. If you do not have a demonstrated track record doing a systematic evaluation in a similar situation, it is wise to seek help and advice from someone who has.

11.3 Process Evaluation

How’s it going?

Key observations…

Further reading