Last week’s flare-up over Khan Academy was interesting on a number of levels, one of which is that we got a new look at some of the arguments used in KA’s favor. Perhaps one of the most prominent defenses against KA criticism is: Khan Academy is free and really helps a lot of people. You can’t argue with the “free” part. On the other hand, the part about “helping” is potentially a very strong argument in KA’s favor —but there are two big problems with the way in which this is being presented by KA people.
First, the evidence is almost entirely anecdotal. Look through the Pacific Research Institute whitepaper, for example, and the evidence presented in KA’s favor is anecdotes upon anecdotes — possibly compelling, but isolated and therefore no more convincing than the critics. The reason that anecdotes are not convincing is because for every anecdote that illustrates KA’s helpfulness, we have no idea how many equal and opposite anecdotes might be out there where students weren’t helped — or worse, where students got “help” that did more harm than good.
And that leads to the second problem: What is “help”, anyway? If a student comes to me who is stuck on a problem and I tell them exactly what to do in order to finish the problem, is that “helping”? It will certainly leave the student with a feeling of having been helped, but what happens when the student is faced with the next problem, or a problem unlike the one I “helped” them with? All I’ve really done here is relieve the student of responsibility, which is a relief, but nobody who is serious about education can call that “help”. In the end, there’s only one kind of “help” that matters: Assistance that leads students to complete subject mastery and independence. Most KA supporters, if you read their blog comments or whitepapers, don’t usually try to distinguish between real and fake help, or actual help and the mere feeling of having been helped.
If I were a KA supporter, I would be pretty concerned about this. KA has some big ambitions and pulls in serious money. There are a lot of skeptics and a lot of school systems that might not be skeptical but need further convincing before taking the sort of plunge that places like the Pacific Research Institute are suggesting. What I would want to have on my side is empirical evidence that Khan Academy videos systematically support deep learning and complete subject mastery among a wide range of students. Data, in other words, instead of success stories.
So I would like to propose the following study. It isn’t perfect, but a smart person could pick up the idea, repair it and then run with it. And if it works out, we’d know a lot more about how helpful Khan Academy truly is.
First: Gather a group of (let’s say) 300 students who are all taking the same math or science course — for example, a random sample of students in high school or university all taking AP Physics. Administer an appropriate concept inventory, like the Force Concept Inventory, at the beginning of the study. Why a concept inventory? Keep reading.
Second: Randomly assign those students into three classes for a semester or a whole year. Class A gets their class taught to them using a traditional lecture/group work format. Class B has their class taught using an flipped classroom structure, using Khan Academy videos as the materials that replace the classroom lecture. Class C has their class taught flipped as well — but using a variety of media and sources, maybe including Khan Academy but not limited to KA and not limited to video only.
Third: At the end of the study, give the concept inventory again as a post test, and see who performs the best. That would mean the largest normalized gain on concept inventory scores.
If Khan Academy truly helps people — with real help, not fake help-feelings — we would see Class B significantly outperform Class A, just like we saw in Richard Hake’s epic 6000-student study of interactive engagement pedaogies [PDF]. If Khan Academy hlps students more than any other resource out there, then we’d see similar results versus Class C, the flipped-but-not-just-KA group. If that happened, it would be a huge win for KA supporters and would be a flagship talking point for adopting Khan Academy as a school curriculum.
On the other hand, it might not turn out this way. If there aren’t significant differences between Classes A and B — the traditional group and the KA group — then the best you can say is that KA doesn’t hurt students, but that’s hardly a ringing endorsement. Some people might really flourish with KA videos, but an equal number of people might really flourish without them in a traditional classroom. If Classes B and C show significant differences in conceptual gains from the traditional lecture class but not between each other, then we should stop saying that “Khan Academy” is what’s helping students and instead talk about the flipped class and how it is helping students. If Class C does better than Class B, that would seem to say we should embrace the flipped classroom but KA is not the best resources, and then the question becomes one of how best to curate or create the most effective resources.
Why concept inventories, and not mechanical skills tests? Well, both could conceivably be given, although Eric Mazur’s work shows that students who improve in conceptual knowledge show significant improvement in procedural knowledge as well even though they get less direct instructional time on procedural problems [PDF]. But it’s important not to just give mechanical skills tests as a way of deciding whether students have been helped, because “help” points toward complete, not partial, subject mastery. A person who can execute calculations by hand flawlessly has not necessarily mastered mathematics. Is mechanical fluency necessary for complete mastery? Sure. But you can make videos that address issues of mechanics that also develop a student’s conceptual understanding. Does KA do this? That’s what the study is for. We should insist on more than just mechanical fluency when deciding if a student has been helped.
As I said, this isn’t perfect, but for all of us involved with the emerging flipped class model and how it’s implemented, the old saying has never been more true that the plural of anecdote is not data.
Image: http://pixabay.com/static/uploads/photo/2012/11/19/07/51/help-button-66608_640.jpg
