This note looks at the macro level of learning outcome assessment - at the class level, year-over-year. Are the students going where you want them to go?

It goes without saying that all serious endeavours should have some kind of a defined outcome that can be used to ascertain if the endeavour was successful; else why are you doing it. That being said, many people do things, serious things, without much thought or thinking it through. In our studies of organizational planning, we have often observed individuals that take more time and effort to plan their personal vacations or grocery shopping then they take when making multi-million-dollar decisions or decisions that will affect many people for a long time; especially when they have no skin in the game.

Outcome-based assessment in education has a long history and there are many resources available that help an instructor understand the principles and suggested best practice. There is a list at the end of this note.

Our observations…

• The outcomes can be organized and aligned with both of the Bloom taxonomies - the complexity and/or the process. It is a starting point.
• While outcomes can be expressed with academic terminology, the outcomes shared with the students should also be in a language they understand.
• Do not assume students will actually read a course outline or pay attention to any stated learning outcomes in the outline. It is necessary to discuss them at the beginning of term, raise them occasionally during the term, encourage the students to reflect upon them during the term, and summarize them at the end of term.
• There can be a one-to-one matching of learning outcomes discussed with the students but it is also possible that there are meta or conceptual outcomes which are assessed from a instructional and program level which are not relevant when it comes to discussing a specific course with students. A general discussion on learning outcomes from a student perspective can be found in a separate note.
• The material in a course outline, especially the learning outcomes, are there for the students' benefit and need to be expressed in a way that the students will understand them, not necessarily how a later review or accreditation process will read them. That should be handled with other material associated with the course. The macro level assessment for a program should not artificially force grammar and phrases down to the course level.
• On assessments, it is possible to have questions where the students have to reflect and use their metacognition skills regarding the learning outcomes and the journey thus far in the course. These types of questions can be mined for insights.
• The learning outcomes should be phrased in terms of what foundational concepts have been learned in the course, or what skills have been initiated or further developed in the course. Learning outcomes should not be at the fine granular level of a specific method or set of facts.
• The learning outcomes and the students’ mastery, should be reviewed by the instructor at the end of term and at the beginning of the next offering. If there is a gap between the desired outcome level and the students' demonstrated mastery, this should be addressed with the appropriate course improvements. This should be done component-by-component!
• If the learning outcomes are foundational, as in many junior courses, the learning outcomes should have clear and measurable criteria. For example, 80% of the students will be able to remember and explain 80% of the foundational concepts.
• If the learning outcomes are at Bloom’s higher levels which are focused on the cognitive skills beyond that of simple comprehension, i.e., applying, analysing, synthesizing, and evaluating, then the learning outcomes need to identify the type of cognitive skill that is being developed in the course and the level of skill being sought (e.g., novice, junior).
• General, macro level outcome-based analyses are not intended to be executed necessarily every term, or even every year, to the two and three decimal place accuracy. They are intended to be a rough trend analysis done over a long term horizon.

• The analysis of outcome-based assessments relies on the laws of large numbers and is not intended to deal with every exception or rare case. It is easy to get caught up in the minutia and do nothing because of this case not being handled or that case not being handled, and forget that the concept is an aggregation trend approach.
• This aggregation philosophy also suggests that outcome-based assessments should probably be at two levels of analysis; a general area of learning, and two to four subtopics.
• It is possible to create a simple tracking tool (e.g., in Excel) that can identify where a learning outcome is assessed during the term, a weight for each indicator, student grades associated with each assessment (total or at a specific question level), and automatically score the term for the level of mastery achieved in each outcome.
• When looking at the analysis, it is important not to make the common mistake of trying to average averages. It is necessary to always use the granular data because there can be a different number of students and a different number of assessments used for different outcomes during the term. Everything needs to be normalized.
• In theory, the learning outcomes are cascaded from the program’s desired outcomes. Starting with the skills and knowledge that graduates are expected to have, the learning outcomes should be defined using a backwards chaining concept where every course and every learning outcome can be linked to the program’s desired learning outcomes. If the learning outcome cannot be linked to the program level outcomes, why is it in the course or curriculum?
• Outcome-based assessments should not exist only for accreditation processes. They should be used for every offering as an active tool for course improvement.
• A holistic and integrated view to outcome-based assessments can be used to determine the success of vertical and horizontal integration. For example, an outcome of learning can be viewed as how the concepts and skills are used in a subsequent course or upon graduation. And the outcome-based assessment of a course can have components that assess the retention and use of what is taught in one course versus where it resurfaces. In this perspective, the outcomes from a course are not necessarily the immediate and direct learning outcomes associated with what a student does during the course.
• It can be tempting to try to develop standardized learning outcomes and standardized indicators for the purpose of accreditation, but this is not recommended. General categories can be used, as per the two Bloom taxonomies, but the learning outcome framework used for outcome assessment should fit the course and not the other way around. Especially, if you expect students to relate to them.

Further reading

• Aligning Outcomes, Assessments, and Instruction. Centre for Teaching Excellence, University of Waterloo. https://uwaterloo.ca/centre-for-teaching-excellence/teaching-resources/teaching-tips/assessing-students/aligning-outcomes-assessments-and-instruction