In this news item from the Washington post (h/t to @ValerieStrauss) we learn that Khan Academy is using “contractors” to check the accuracy of some of its videos. The report is prompted by an email exchange between the piece’s author and Sal Khan himself regarding the accuracy of one of the physics videos. In Khan’s response, he says:
We have deleted the video [a physics video that had an error in it]. We are trying our best to keep up with any errors on the site (both through feedback from users and peer-review from educators). I checked into why we didn’t notice this one earlier, either your friend or someone else did point this out in the comments but they did not surface to the top (we currently have contractors with math/science backgrounds reviewing much of the math material and the comments to find other issues like this). We do need to get better at making sure that this type of thing doesn’t happen.
Read the whole piece, which has a lot of detail on the physics problem in question and some more comments from Khan on their system for accuracy-checking.
Some quick thoughts on this:
1. Give credit to Khan Academy for having some kind of system in place for catching and repairing errors in the content. While we hope that every video from Khan Academy might be mistake-free upon release, the fact about making content is that you will make mistakes at some point. Goodness knows I’ve had to replace or remove my own videos more than once based on errors that users spotted. You try to be right the first time, but you also have to have the capability of correcting yourself in case you aren’t right. It’s good that KA is using content experts to do this sort of thing.
2. I think this situation highlights a major advantage of the flipped classroom. Namely, when you curate all your lecture content online in a reusable format like video, it is fairly straightforward to find errors, suggest changes, and then make those changes or remove/replace the video after the fact. Over time, most of these errors can be ironed out for the next round of students using them in a class and the factual errors contained in them should approach 0. Whereas, in a traditional lecture class, once the lecture is done, it’s done; and it’s a lot harder to go back and fix any mistakes that might be made because there’s simply no record of the lecture unless it was recorded (at which point we are curating the material and can flip the class with it).
3. All that being said – It does make me wonder what sort of quality control for factual correctness is taking place before the videos are posted. Khan mentioned in the WaPo piece that KA “recently [hired] physics, chemistry and biology experts to peer-review content”, which suggests some kind of review taking place before posting videos, but it’s hard to be sure when this peer review is taking place and what it involves. I’d be interested in details on that.
4. It certainly took quite a long time, and a fair amount of uproar from the education community, before any such system was put into place. In my view, the moment Khan Academy started accepting grant money (for example from the Gates Foundation), a rigorous pre- and post-production peer review system should have been put into place. Because at that point, you are no longer a fly-by-night outfit making videos for your niece but a professional organization receiving money to produce something of quality and make it available for students around the world. The content may remain free, but that is no excuse for sloppiness or errors.
5. Is there anybody checking the pedagogical approach to the videos as well, or the organization of the content areas, in addition to the actual domain knowledge that’s being represented? That is, we have a small group of content experts checking for factual correctness, but is there a similar group of pedagogical specialists going through to make suggestions as to how to improve the actual teaching of the content? If not, why not?
