I’m back from taking a few days away for the holidays and to get the new semester underway. Before the break, I had posted a couple of articles about specifications (“specs”) grading and how I planned on using it in my classes. That was before I had sat down to hammer out the specifics. Now that I’ve done some hammering, I thought I’d give some updates, specifically on the way specs grading has worked itself out in the Discrete Structures for Computer Science course.
A lot of what I am going to write here is a repeat of what I wrote earlier on December 22, but between that previous article and the actual start of classes two weeks ago, I made some significant changes. So bear with me if it sounds like I’ve said a lot of this before – I have, but this is the final version.
I’d taught this course before a couple of times using a traditional grading system. The course is one of my favorites to teach, but the traditional grading always seemed like a poor fit. Since the course is required for the CS major but is not itself a prerequisite for any further course, there’s a strong temptation to treat it like a hoop, even among the best students in the course. It was a course where the grades very often gave a lot of false positives (high grades from students who didn’t really demonstrate consistent mastery – like strong students who slum through and get “B”s by getting only 80% on everything) and false negatives (low grades from students who weren’t trying hard enough, or who got caught up in other things). For that reason alone, I felt specs grading was a good fit for the course, since it emphasizes students being in control of the grades they earn and on providing concrete evidence of mastery. I also felt that CS people would understand specs grading better than others, with the emphasis of competence-based learning in CS and computing certifications.
So here is how the assessment was set up in the course.
Before writing out the syllabus, I went through the entire course and the textbook, section by section, and identified all the learning outcomes I felt were appropriate for the course. As with any learning objectives, I wanted to phrase these as action verbs that could be easily measured. I ended up with 68 of these. However, just like when I write Guided Practice assignments, I realized that some of these objectives were what you might call “basic” and others were “advanced”. So I separated these into two large lists, one list of 35 “basic” objectives and another of 33 “advanced” objectives.
Then I realized something else – that among the advanced objectives, there were some that were more important than others for true understanding of the course material. The full list of advanced objectives is something I would expect an “A” level student to master. But the subset of really-important advanced learning objectives is something that all students (or, at least those at a “C” level or higher) should master. I ended up identifying 20 of these “core” learning objectives – objectives for which any student who earns a “C” or higher in the course ought to show multiple instances of mastery.
I ended up calling the “basic” objectives Concept Check or CC objectives; the “advanced” objectives Module or M objectives; and the subset of really important advanced objectives Core Module or CORE-M objectives. Here is a document of all those objectives; there is a list of objectives by topic, and the same list again remixed by type.
The weird nomenclature for the objectives comes from the way that I chose to assess those objectives.
- CC objectives (the simple, low-level objectives) will be measured through Concept Checks, which are short objective quizzes given in class. These objectives are simple enough so that the questions on the quizzes can be fairly assessed through stating definitions, multiple choice, true/false, or simple calculation questions where a right answer can be reasonably be interpreted as mastery, and a wrong answer means no mastery. Each concept check assesses 3–5 CC objectives, and there is one item per objective.
- M objectives (the harder, higher-level objectives) are measured through Learning Modules. These are analogous to Linda Nilson’s “learning bundles”. Learning Modules are thematic homework sets to be done outside of class and include items that go deeper than the objective items on concept checks; some of these include creative items like programming or proofs, and all of them include an item asking students to engage in some metacognition, reflecting on their work process and how they could improve.
- CORE-M objectives (the subset of most important advanced objectives) are measured not only through Learning Modules but also through timed assessment. I have this set up as follows. There are four class meetings during the semester set aside specifically for timed assessment on the CORE-M objectives. Prior to those timed assessment periods, students will look through the list of CORE-M objectives and decide which objectives they believe they are ready to be tested on. I will make out one problem to work for each CORE-M objective that students requested and then bring a stack of copies of each problem to the assessment period. Then, students come and get the problems they wanted to work, they work them, and then they hand in the ones they want me to grade. Students can also ask to retake any CC objective that they didn’t pass in the concept checks from class – and, importantly, they can retake any timed CORE-M problem they didn’t pass in a previous timed assessment period, and they can take any CORE-M problem from earlier in the course if they weren’t ready to be assessed on it before.
There are two other items in the class – Guided Practice (since this is a flipped class) and an Application Project.
True to the specs grading ethos, nothing gets partial credit. The concept checks are graded on the basis of right/wrong answers only; if a student gets a right answer on an item for a particular objective, the student gets a Pass mark on that objective; otherwise the student gets No Pass. In fact there is not a single item in the course that has a numerical value attached to it at all.
Grading on the Learning Modules and timed CORE-M problems is also done Pass/No Pass, but it’s more involved because of the advanced nature of the questions. For these, we have a detailed set of specifications that categorize Passing work. To mark student work on Learning Modules and CORE-M problems, I read the work carefully, stick to the specifications, and based on the specs and on my best professional judgment, I decide whether the whole of the work is Pass or No Pass.
The class has three main topics in it – relations, graphs, and trees – and I decided to have three Learning Modules for each topic. There are also two non-content Learning Modules: A “Getting Started” learning module that involves students working through an online quiz over the syllabus, writing a mathematical biography, and setting goals for the semester; and a “Tech Competency” module which has student set up a SageMath Cloud account and use it to write some basic Python code and write up a Markdown document.
So that’s the assessment structure for the course. We have several Concept Checks to assess basic skills; Learning Modules to assess higher-order thinking; and timed problems to give double-coverage of assessment on the most important higher-order skills. Plus Guided Practice to help students prepare for class work and gain fluency with basic ideas on their own before class, and an Application Project at the end.
The students’ grades are determined, simply, by the number of items that they pass. Here’s the main table from the course syllabus:
There are also rules for earning plus/minus grades because, frankly, my Dean forced me to include them. I asked whether I could opt out of giving plus/minus grades and was told that if the university has plus/minus grades on the books, I have to provide a means for earning them. I would have liked not giving plus/minus grades, but whatever. You can read the whole syllabus here.
Under this grading scheme, for a student to attain what I consider to be a minimal baseline competency in the subject, he/she has to pass a little over 70% of the CC objectives, 60% of the CORE-M timed problems, the Getting Started and Tech Competency modules along with 67% of the other modules, and prepare successfully for class 75% of the time. And notice that only the students who are aiming for an A or B grade have to do the Application Project; so only the most highly motivated students are going to be working on it. (Conversely, the highest grade you can get in the class without the project is a C+.)
If anything, I think that the baseline for a C here is too soft. Perhaps setting the CORE-M and Learning Module cutoffs at 70% would have been more appropriate; we’ll see. However the Learning Modules are going to be pretty challenging and rigorously assessed -- a “Pass” is what I would normally consider to be B+ level work -- so passing only 6/9 of them might be appropriate C level work.
Here are some things that I like very much about this all turned out.
First: We are replacing partial credit with feedback plus revision. If a student turns in work that is subpar, I no longer have to agonize over how many points to give it. If it’s truly not good enough to meet our specifications, I mark it No Pass and give it back to the student with detailed feedback on what was subpar and how to fix it. This seems to encourage students not to give up on a problem – if they had issues with a topic or problem earlier in the course, as long as they have the will and the means to try again, they can try again. To me this seems much more humane, and much closer to actual professional practice, than partial credit, which seems rather to encourage a fixed mindset in students (once they get a grade on an item, that’s how good they are on it, regardless of how much they learn from that point forward).
Second: It forces students to pay attention to what they are doing and give their best work. If you submit a Learning Module but it’s in the wrong format, or you leave a problem off because you’d rather not work on it, etc. you will get a No Pass. You can’t get a B in the class by earning 80% on everything and not demonstrating real mastery on anything.
Third: It puts students in charge of their grades. It seems like in traditional grading systems, grades just sort of happen to students. They come into our classes and have the mindset of I’m going to work hard and hope that I get an A. Instead, my students started off the semester in the Getting Started module by going through this goal-setting exercise and answered the question: What grade do you want to earn in this class? and followed that up by listing all the requirements for that grade. Students know, at all times, exactly what they need to do to meet their goal. It’s very intentional and I think that’s a refreshing change.
Fourth: Specs grading has already changed the narrative about student work in the class. The conversations I’ve had with students have not been about points but about math – occasionally, about work habits and how to manage tasks and projects in order to get one’s work done. But these are good conversations, not loser conversations about how many points one has to earn on the final exam to get a B for the class. I am hopeful that it will change the way students think about themselves too – not as passive bystanders but as intentional actors in their education.
It’s still early and as I continue to roll this system out, I’m sure I will find bugs and loopholes. But as of week 3, I can say that students are not rebelling over the system; in fact quite a few students were enthusiastic about it. Many of them liked the fact that once you have met the requirements for a course grade, nothing can lower that grade – so there is an air of safety in the class. Some liked the de-emphasis on timed testing – I like that too. Many students are still puzzling this out and are reserving judgment, which is fair. The main idea is to continue to listen to students and act in their best interests, which is what I believe specs grading is allowing me to do in a way that traditional grading did not.
Want to continue the conversation? Follow me on Twitter (@RobertTalbert) or on Google+, and share this article on your networks using the social media buttons above.
