16.4 Multiple Choice Assessments

Generally speaking, we are not a big fan of the way multiple choice assessments and the way they are used. There can be good, even great multiple choice assessments, but in our experience, they are rare. The dodo bird comes to mind.

They take quite a bit of effort and skill to create. In fact, in Bloom’s report (1956) the topic is discussed at length. They give a great deal of guidance about how to create a multiple choice question and how to set up questions to test for comprehension, and deeper levels of knowledge. It is not easy as each choice is supposed to be designed to expose a student’s mistake in one of the core concepts and the set of choices to expose errors in the taxonomy (e.g., understanding, comprehension), in addition to one right answer. It is still the best reference we have found that explains how to do this right!

That is, a wrong ‘choice’ would indicate that a certain, flawed process was followed. A different flawed processes would end up with a different wrong choice. This requires the instructor to think through how a student might end up if they did not comprehend etc. This is very time consuming and takes lots of experience and skill to do. Then, you have to replicate to avoid the random effect. Try doing this. Think about doing this for a bank of questions. Every assessment? Changing the banks every year? Tired yet? Ready to give up? Easier to just makes answers a little tricky with wording to confuse the student, or have complicated math to do that will allow for simple math mistakes.

Some thoughts…

• Multiple choice assessments have the best chance of success for Bloom’s most basic level of knowledge, and the typical end-of-chapter type questions where there is no serious, deep thought or cognitive process to work through. The tests will assess basic memory and definition, but not anything that would be considered mastery.
• To assess mastery, it is necessary to understand the how and the what, not just the what. Unless you see the process, you do not know how they got to the solution. You just do not know.
• You can have multiple questions testing the same basic knowledge and correlate the questions to see if guessing was involved, but you would need sufficient questions to do this level of validation (e.g., what are the odds of guessing the right answer five times for the same type of question?).
• If you are going to use multiple choice, allow for an additional sentence or two asking the student to justify or explain how they got the answer. What equations were used, what partial results arose. Why a certain concept helped them reach the answer (e.g., in non-math courses), etc. It is possible to advise the students understand how to answer such questions so that they are comfortable with the additional level of cognitive probing and that they may be able to demonstrate ‘what they know’ vs. one single correct answer with the process 100% hidden.

• We do appreciate and understand how convenient and cost effective such tests are for the instructor and ‘the system’. However, considering the overall contribution made by student tuition etc. per term, it does not sound like a balanced equation to us.
• We believe that sufficient resources should be available and used to perform appropriate assessments. If this means more tutors and grad student teaching assistants, so be it. If this means less money and time for research, so be it (sorry). If teaching is a critical pillar for your institution, faculty, and department, treat it as such.
• To deal with the above point when multiple choice has to be used, we have started to suggest that one-third to one-half of the assessment be multiple choice (with properly designed choices), and that the remainder of the assessment be written or full answer questions which expose the process and allow for partial marks. In this way, there is some efficiency and convenience for the instructor, but the student still has a chance at a fair opportunity to demonstrate their mastery.
• The goal should not be the easiest and most convenient for the marker. A fair assessment of the student’s true mastery of the desired learning outcome should be the goal.

Further reading

• Bloom, B.S. (Ed.), Engelhart, M.D., Furst, E.J., Hill, W.H., & Krathwohl, D.R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive domain. David McKay Company.