16.2 Online Assessments

Online Assessments SoT-16-2-OnlineAssessments

Best to assume that additional resources will be used by some of the students.
Try to design assessments that cannot be googled.
Try to design assessments that have an individual touch to them, perhaps linked to what the student did during the term on a project or class activity; something that can be traced and verified.

I can’t watch them!

Assessing students online is tricky. Students inclined to cross the line will, and others who are stressed out and panicking, might. It is also possible that students do not know how to identify improper behaviour and think that certain activities are fine and acceptable. An example of the latter case is when a student might describe the test to another student before their friend takes the test.

Surf about the percentage of college/university students 'cheating'. Often the figures are self-reported and the results are conservative. We have been told by very bright and ethical high school students that 'every cheats a little on online tests'.'

At the end of the day, there are five basic approaches for online assessments,

there are the draconian, big brother approaches of creating situations where the student is tightly monitored or controlled
arrangements are made for remote proctoring
some instructors use an honour pledge
creating personal, “viva voce” assessments; done 'live' in some way as a verbal exam
carefully designing the assessments acknowledging that you have little real control and if students are going to cheat they will be creative and find a way to do it

Thoughts about the first four methods...

There are companies that will provide you with software that will monitor the student behaviour while on their laptop. Some also offer the ability to use WebCams so that you can watch the students do the assessment. Some of the online teaching tools will allow you to create time tests or restrict the student from question to question. We believe that all of these methods create unnecessary stress and anxiety and can prevent the student from demonstrating their mastery.

Take time-controlled tests and controlled sequence tests for example. Best practice tells the students to read the whole test first, think about it, plan their strategy for the questions to be answered, move on to the next question if they are stumped and then return, and leave time at the end of the test or quiz to review all of their answers. The best practice is based on how people solve problems, and work things through. Assessments that restrict the order of question taking are definitely not best practice and can create stress for the student. Time tests may or may not be a problem, it depends how much time the students are given. The assessment time should allow the average student to get an average mark without stress and without anxiety. Timed tests also have to take into account technical difficulties which might occur during the test.
Some institutions can arrange for remote proctoring, but this does require funding and can create issues when students are widely distributed. The whole process can also create stress and anxiety.
There are instructors having great success with honour pledges, and discussing integrity with their classes. This seems to rely on the individual instructor, their class relationship, and is likely hard to replicate and sustain. It is probably a good thing to try in all cases anyway, but it cannot be assumed that it can work for all instructors in all courses. If you use them, best to include some checks and balances to verify that the pledge is indeed working.
“viva voce” or verbal examinations can be one way that an instructor can examine each student and ascertain their mastery of the material. When done at the graduate school level, there is also usually a written component (thesis) and a multiple hour long exam to ensure that the student was the author of the work, understands what they have written as it relates to the field, the strengths and weaknesses of the work, and how the research contributes to the field. Voce examinations have a long and rich history. In small classes, a similar approach can be taken where a student does a written submission, and the examiner reviews the submission with the student asking supplementary questions to ensure that the submission was the student’s own work, and representative of their knowledge. The verbal allows the examiner to adjust the grading if necessary. If the assessment is relatively weighty, the written portion may require one to two hours of effort by the student and the verbal can easily be ½ hour to an hour long. A short dialogue is not effective and cannot probe the material to the appropriate depth and breadth. It also takes skill on the examiner’s part to execute the test without causing stress and anxiety. While this method can work, it is not scalable and it is hard to replicate.

The fifth way...

Another approach is to create somewhat individualized and custom assessments for each student in such a way that the students can demonstrate their mastery, and it makes it difficult for students to systematically cheat in a sustained way. We have to assume that many students may cross the line occasionally but very few will actually do it consistently on all assessments. We should focus on the students where the cheating will matter in the long run.

The following are some thoughts about how to design assessments which are open book, open surfing, and potentially open friend…

If the technology allows for it, a bank of tests with random selection of questions can be used.
Simple true/false, or unstructured multiple-choice questions should be avoided. While it takes time, structured multiple-choice questions can be designed to pick up flaws in the basic knowledge, comprehension, ability to apply and analyse, and to synthesize and evaluate. The set of questions can be designed so that a theme is repeated and the students consistency can be checked. The structured multiple-choice questions should also have a textual portion: why the choice was made.
If the answers are in random order for the students to select, it is harder for another student to simply say the answer is B.
Noted above, but worth repeating: for multiple-choice questions, we also always include a short written component that checks the students understanding for why the answer was chosen. This is important for both classroom and online assessments.
Considering cognitive science and what we know about the acquisition and development of skills, most of the assessments should focus on comprehension and the ability to apply the ideas with the decreasing emphasis on analysing, synthesis, and evaluation. Analysing, synthesis, and evaluation at any level of expertise is beyond a junior or intermediate student and most seniors. In senior seminar or studio courses, it is possible to probe the deeper learning.
An online assessment can be a mix of simple questions and one or two longer questions which probe the students’ ability to comprehend and apply the ideas. These can be half a page to a page long and focus on the foundational core concepts that support the learning outcomes.
The questions can be customized and individualized in many ways. For example, the student creates the example and explains it to a more junior student or a child. A fantasy situation with vampires, zombies, etc. can be used and the student has to apply and transfer the concepts to a different situation.
Comprehension can also be tested in more quantitative and STEM areas, by presenting an analysis which may or may not be correct in asking the student to analyse the answer and explain what went wrong and how it could be corrected. It is unlikely for duplicate answers to arise consistently throughout a test.
Comprehension and applying can also be tested on an online assessment when you present a problem, and ask a student to explain how it would be solved.
Yes, all of the suggestions require more effort on the part of the instructors and the markers, but with practice such questions and assessments can be dealt with effectively and efficiently.

16.2 Online Assessments

I can’t watch them!

Thoughts about the first four methods...

The fifth way...

Further reading