14.10 Design (Progress) Reviews

It seems that term projects, team based, are becoming more and more popular as assessment tools. We think that they are great. But, we also think that many instructors do not think too much about what a term project means. Especially one that involves designing or significant problem problem solving and independent thought. It is very easy, so easy, to craft a project and just say 'do it' and then mark the final submission. Or, have one or two simple interim deliverables. Designing a good project is not easy. It has to address the learning outcomes for all students on a team, and the how should be considered as important as the final what.

Term long projects, team or individual, will benefit from several design/progress reviews. The projects are often worth 25% or more in a term and represent a significant effort on the student's part. A rough guide for a project would be 2 hours of expected effort per mark, similar to what is probably put into a normal assessment, studying etc. It is unreasonable and unfair to expect/demand more!

If you want some fishbone fun and practice, do a cause and effect diagram/analysis for what creates an effective and efficient project in your course, aligned with the course learning outcomes. What is the criteria or mechanisms you will put in place to monitor and understand the causes and effects as they roll out during the term? This process will provide you insights into the types of decisions you will have to make. For example, how much freedom do you give the students on each aspect, why? Each of your decisions should be tied to learning outcomes, priorities, and what the students really need to do. 'It is a good topic!' Why? 'This is what I want the students to do!' Why? 'This is the feedback I will give!' Why? 'This is what the students can decide for themselves!' Why? If it is not on the critical path, aligned/attached to a cause and effect, why do it? It just creates noise and additional load/effort/stress. Designing a project and how the students will be guided implicitly or explicitly requires conscious thought and is itself a design task, a project. It should not be taken lightly. In our own experience, it can take many days and many weeks to design a term project and think it through. Never has it been minutes or hours. Term projects also benefit from the buddy system and review. Your buddy can challenge each and every decision being made about the project, what should be in it, assumptions about what students can and will do, what risk management is required, and so on.

You should also do a fishbone exercise on what you think an effective and efficient design review is. Just saying. This is one example of what it means to be a professional and to have a professional attitude to the task. Professionals should do due diligence and do things with intent and justification. If you are not thinking it through, how are you any better than the average person on the street at a task. Eh? And just because you did projects as a student or marked them as a TA does not mean that you know how to design one. It is also wise to be cautious when just using a packaged case or project from another source, you should analyse it and think about it; is it a good project/case for your class? What criteria are you using? How much do you really understand about the case/project and what the students have to do? If you are using a third party project or case, it is even more important for you to have gone through the project/case yourself, being a student.

Getting back to reviews. If possible, the instructor and TAs should participate in the design reviews. It is a good way for both to know who is doing what, when, and how well. This intel about the project is probably a decent proxie for other things as well.

For term projects, we strongly recommend that the instructor or a TA has actually done the project in advance, has some appreciation for the decisions to be made, possible choices. This will work for certain types of projects where many of the requirements are provided. If the students have a lot of decision making (not recommended in first year btw), as in a senior year course, this is not doable. In this case, the individuals doing the design reviews should have extensive design experience in the general domain space so that they understand the subtle issues and decisions being made (and know what decisions could have been made). If you have not designed x or something in the same scope/area as x, how can you review it? If you cannot provide sound, expert advice, why did you let the students do it? If you assign it, you better be able to review the how and the what to the level required by the learning objectives.

The learning outcomes for the project should be VERY clear; what do you want the students to really learn and demonstrate via the project. This is mandatory if you want to provide an effective and efficient design review.

For some design projects, the goal is the design process, learning how to design. In other projects, it is to apply course concepts, assuming that the students know how to design and do a project. In yet other cases, it can be combo.

You should not assume that students know how to design and know how to approach a project. Someone along the way needs to teach them this stuff and others need to reinforce the learning through deliberate practice, feedback, and reflection. You should not assume that the students will know how to prepare for a status update, design review, or how to respond to feedback.

Here are some thoughts about design reviews for term long projects (and some about projects in general)...

• There is a difference between a status update meeting and a design review. A design review is longer, more intense and dives into the design process and the decisions being made (even it is more about applying course concepts vs learning design). The status update meetings are different and have different goals.
• There should be a design review about two weeks into the show, with a deliverable that covers planning, team organization, roles, and such. This will help them get their act together. It will provide you with insights about leadership, roles, task distribution and such.
• This plan should be reviewed by the instructor for learning outcomes: does the plan allow for all of the students to learn all of the learning outcomes? As the term proceeds, this is also a factor. Are all of the students on a team engaged in all of the necessary activities?
• If you are engaged in any tracking effort to understand if the learning outcomes were actually learned by the students, team projects can bias the results unless you, as the instructor, structure and manage the situation. One of the four students might have learned an outcome and this is erroneously associated with all of the team members. Whoopee! 80% of the students learned x and can do it to a 80% level, while in reality, it is really only 20% of the students who have a clue.
• Depending on the project, the first review is similar to a Product Opportunity or Preliminary Requirements (if they have degrees of freedom in that space). This is the 'what is the problem, why is it a problem, how big of a problem is it', and what are the main features/functions that will address the problem, and why are they appropriate, feasible, reasonable.
• If they are basically given the requirements, the first design review will be about their interpretation and perhaps detailed out. If everything was detailed out, the first review will be about possible solutions or major design decisions.
• If you are using the constructive failure strategy, you will not give much advice, 'fix' things for them. You might suggest 'think about x', but you keep the advice to a minimum. You want Darwinism to work its magic. Use your best judgement. Make a decision, justify it, discuss options, explain your rationalization for one decision over another. These are typical things we tell the students. We try to avoid 'good', 'right', etc. You will deal with these decisions in the next meeting. You follow this pattern throughout the term - reflect on the past decisions and in some cases allow them to do a reset, fix things, and then move forward.
• In the early design reviews, it is about following instructions and the process, not the what per se. It is learning to design, to think through the situation and the actual outcome is an artifact, not the goal. This is true for the whole activity actually; what are you hoping the students will learn by doing the project. They are students, they are learning, they are not experts yet, they are spending a relatively small number of hours on the task, what do you expect?
• For term long projects, we like to have a team update weekly or bi-weekly (depending on the desired outcomes). This says who did what, what is planned for the next week (or two), how long it is supposed to take, who is supposed to do what. A general 'all did x' is unacceptable, it needs to be broken down. We also ask for them to reflect on deviations from plan. What went better than planned, worse than planned, why, and recovery plan. In parallel, we often do weekly reflections and the students are asked about what they did with respect to the course/project and what they have learned in general as a 'student' for the week. This often provides secondary intel on who actually did what and when; if you ever need to know.
• The second review should be two to three weeks after the first.

• The third review perhaps three weeks after the second. The reviews can be more frequent if the project warrants it.
• As the design reviews progress, it is good to reflect back on previous decisions and choices made. Still happy? Still makes sense? What have you learned about that decision? Would you do something different in hindsight? If you do not revisit all of the key decisions in the past, for the good, bad, ugly, the students will not learn.
• If the students have dug themselves a hole and they have 'learned' the lesson, you can give them options for fixing things. This needs to be done fairly across the class though. Sometimes, we give each team one 'coupon' to redeem, allowing them to change a previous decision. Sometimes two coupons are needed; it depends on the project and decisions that you let the students make.
• It is sometimes good to have a buddy, a third party who can act as the client or outside reviewer for the project. A wild card. The students will have a game plan for the instructor and the course in general. The third party can mix things up, reveal decision making in the moment.
• For all major decisions and for many of the minor ones, the students should be able to present and discuss the decision process they used; how did they arrive at their decision. What was the criteria? Did they have a systematic way to think through the process, the cause and effect relationships? How to compare options?
• Different students should be randomly picked to answer questions and to present any presentations. This helps the team work together, ensuring that everyone knows the minimum about the whole project, is involved.
• There should be at least two major reflection points, team and individual. What have they learned so far, and what would they have done differently. We time these for about 1/2 way through and as part of the final deliverable. This helps the students and also helps the instructor understand what is being learned, and who is actually doing what. These also feed into the design reviews; helping you with your questions and probes.
• Just assigning a term project without suitable deliverables and feedback during the term about the what and how is far from best practice. It borders on negligence. As does the simple creation or use of a project/case that you have not actually done in advance. You are the instructor and you need to know what the student is going to do, what they are faced with. This goes for questions and all assignments actually. We first encountered this in high school with teachers assigning work they did not know and could not do. We have seen it ever since by instructors who are not 'good'. The good instructors know what is entailed and know what the students will do, and can do the task themselves; they have intimate, personal knowledge of what they are expecting the students to do.
• We have seen two trends with term projects that instructors need to deal with. First, there is the selective task allocation where one student does all of x, another does all of y. If the intent is for all of the students to learn the targetted learning outcomes, you need to manage this (some of the above points address this). Second, if the students are in a cohort, we have students split up the projects. One student does the project in class A, another student does the project in class B, etc. This is really bad, but the students think it is really smart. And, by the way, having the students sign pledges and attest to contributions, does not always work, and probably doesn't in most cases. What is considered 'cheating' is way different than it was a decade ago, two decades ago. As one student commented to me about online courses, 'everyone cheats'. Perhaps not everyone, but it is believed that everyone does, that it is ok, and that they will likely get away with it, and if they are caught, a mere 'I am sorry' will suffice without any further penalty or cost.

Further reading

• Skinner, D., Mckeage, K., Seymour, R.M., Donahue, D.W., and T. Christensen (1999). Implementing an Interdisciplinary Marketing/Engineering Course Project: Project Format, Preliminary Evaluation, and Critical Factor Review. Journal of marketing education, 21, 3, pp. 217-231.
• Hurst, A., and O.G. Nespoli (2019). Comparing Instructor and Student Verbal Feedback in Design Reviews of a Capstone Design Course: Differences in Topic and Function. International Journal of Engineering Education, 35, 1(A), pp. 221–231.