16.3 Peer Based Comparisons & Subjective Rubrics

This is not peers reviewing peers, it is how to compare students against each other when working with qualitative assessments. It is peer based and NOT peer review. Peer review is a different topic.

In some disciplines there is an inherent quantitative element that makes it easy to say if an answer is correct or not. There is science, numbers and mathematics everywhere and scientific and mathematical skill are the dominate assessments. However, there are other skills required in the real world when individuals graduate or in studio/design courses. Where they must do problem solving of a complex nature, the problem solving is done in situ and within organizational structures and situations, they must be able to read and write business and industry communications, they must be able to work with others, and they must be able to practice life-long learning. It is difficult to assess such things in a quantitative way and most instructors in STEM type subjects shy away from qualitative aspects.

Even with quantitative situations where there are multiple possible choices, the problem can be loosely stated and allow the student to pick and justify their choice. The student should be able to explain the assumptions, criteria and demonstrate their level of comprehension beyond the simple facts. The justification is likely to be somewhat qualitative in a complex problem and the justification can be assessed. There is also the variance in how the answer is derived, the process behind the answer. If the 'how' is a learning outcome, it is likely that peer comparisons will be necessary.

And then there are the qualitative topics and situations where it is all about deductive, inductive reasoning, building up causality, justifying a summative result. Rarely is there one right way, or one right answer and it can be appropriate to use peer comparisons in these situations.

The students will expect and demand to know why they got a certain mark. They want the marking to be fair. They want to know what they did right and what they did wrong. They want to know why they got a 'B' when their buddy got an 'A'. Unfortunately, with qualitative topics, there is are often shades of grey and few absolutes; sometimes, it is just better or worse, not right or wrong.

One approach we have used is using the class itself as the baseline (combined with an eye to minimally accepted standards or expectations). It is our experience over the decades that there will be the 15-20% of the students who really nailed it compared to the others and these are the 85+ anchor group. It is possible to get a 100% in this model; if the response is that much better than the horde, then it does not have to be perfect (perfect does not exist in this process), just has to be that much better.

The trick to peer comparisons is the rubric, and how the rubric is constructed and used. Our concept of a rubric is not the common one of a simple matrix with marks/categories across the top and perhaps 5-10 topics on the vertical. The rubric style we have developed over the years addresses the problems associated with marking qualitative submissions when the students are not comfortable or familiar with qualitative marking.

Observations…

• Social Comparison theory is important to understand if you are going to do peer evaluations.
• Possible bias is important to understand if you are going to do peer evaluations (Kahneman et al. 2021).
• Design the assessments and topics to explore the student’s mastery of the intellectual abilities encapsulated in Bloom’s taxonomy; the comprehension, the ability to apply, analyse, etc.
• The first time an assessment is used, avoid creating a rubric in advance and for project or assignment work, definitely do not give out a rubric in advance.
• Once the assessment is done, you make a quick but ‘good enuf’ pass through the submissions to identify the good, bad, and the ugly. This is not a difficult task but does take perhaps an hour or two to do. As you are performing the sniff test, things will stand out: what was in the good responses, and what was in the others (or not). You note the common mistakes.
• You then create a rubric that is based on the sniff test and the points noted during the pass. If the assessment is similar to one in the past and a rubric is being re-used, it is updated. It is important to create or adjust the rubric based on actual class execution.
• There are four parts to the rubric:
• The rubric starts off with an interpretation of the question. The key phrases or clues that were needed to comprehend the situation and what is being asked. You open up and discuss the thinking process. For example, the ‘how do you figure out how to’ bit of the ‘how do you figure out how many students a whale can eat’, and not the ‘how many students a whale can eat’. You need to help the students learn how to decode the question and assessment.
• The second part of the rubric is a point list of what the A’s did; what was found in the good submissions. This can be 5-10 points for a short assessment, or it can be a page or two for a major term project. This allows the students to know what they were being compared against.
• The third part of the rubric is explaining the basic marking, e.g., how much was the justification bit worth, and the key aspects that were looked for when judging their justification.
• The fourth part of the rubric is a list (as long as it needs to be) of the common mistakes made by the class. Sometimes you might need to put a scale on the point (if there are degrees of mistakes; e.g., degree of handwaving and points made without justification). The key words can be highlighted in the rubric for each student or the scale checked off.
• The actual assessments are not marked upon. The rubrics are either printed off and marked on, or the soft-copy rubrics are marked up. If during the detailed marking the rubric needs adjustment, it is done and things are re-balanced if they must be.
• The marking is done in favor of the students. We give them the benefit of doubt, and being somewhat gentle when interpreting the rubric. The spirit and intent is used. The purpose is to test the comprehension and understanding of the student, and the marker cannot be pedantic.

Co-dependencies & The Necessary and Hopefully Sufficient

• The process is based on a basic Industrial Engineering principle. Spend the time on the setup and then the time per piece will be effective and efficient. This technique depends on the time dedicated to get the rubric ‘close enuf’. There will always be minor issues and things to fix on the next rubric. Do not worry about getting the rubric 100%. The overall marking effort is actually reduced; the time to mark and the time to interact with students. The feedback is also better. You have to have patience, the willingness, and discipline to take the initial time to set up the rubric and not just jump in and start marking.
• It is necessary to have the instructor actively engaged with the assessment process. Some instructors create the assessment and then have the TAs mark 100% of the submissions using general guidelines or an example ‘golden’ answer sheet. In these cases, the instructor never really knows what the students did or what mistakes were made by the students or why. It is not possible for the instructor to create or guide the creation of an acceptable qualitative rubric without being involved. They must look at the assessments and know what is going on!
• The quality of the TAs is important if there are major changes being made to the assessments. The first time a rubric is created and used, there will be a longer process as it is reviewed and adjusted by the marking team.
• You have to give enough examples and points about what the 'A's did and did not do. You have to give enough detail about the marking scheme and sufficient points for the common mistakes. Without sufficient points, the students will remain in darkness and lack the clarity and understanding they need to learn from the assessment exercise. There will be long lines during office hours if you do not provide sufficient feedback.
• We have often found that the student TAs are harsh markers, are going after blood, and are sticking to every minor point in a very black and white way. This will not work and the TAs need to mark with spirit and intent, and not the letter of the law. This is why the instructor must test and review the TAs marking before the whole assessment is processed. We ask the TAs to select an example of each grade level and then discuss the choice with them.
• Just having one qualitative assessment in a term will not really work. They will need the first one to get a feel of what is expected. They will need a trial run to understand the process and trust that the system will be fair and balanced. They need to know that they will get credit for what they have learned. In one course we teach, where there are many qualitative assessments, by the end of term the students get to create rubrics and apply them to other team projects (as an exercise) and also develop a rubric for what makes a good rubric. This is possible, even in a freshman class. This helps them with the high cognitive tasks of evaluation and analysis and actually understanding how to do criteria-based analysis.
• It is necessary to have all four parts in the rubric; the expectations and decoding of the assessment objectives, what the 'A's did with suitable examples, the key aspects which will (or will not) get marks, and the list of common mistakes which are checked off on an individual basis. The mistake list is also important; instead of simply seeing what they did wrong, they also get to see the larger set of common mistakes which were possible to make.
• Often the list of possible mistakes and shortcomings is rather long and even with a light marking hand, the marks are reasonable for the average student; not too high, not too low; because of the relative ranking and comparator approach. If the stated, course policy is to 'remark' a complete submission upon petition, we have found that almost all students are happy with the marking, feel that the results are fair, and they do not seek a remarking. They have been given the necessary feedback and they might even think that they 'got away with a few things'. If there is an actual error made in the marking, the errors are addressed, but the holistic approach to the problem prevents students petitioning for a mark, 1/2 mark, debating nuances, that kind of thing. We have been using this approach for three decades and the students find it fair and accept the relative marking. They like that they are not held to an artificial gold standard, and that the assessing team takes the time to really understand the assessments and critique them.

Further reading

• Kahneman, D., Sibony, O. and C.R. Sunstein (2021) Noise: A Flaw in Human Judgment, Little, Brown, Spark.