When it comes to measuring the success of an education program, the bottom line is often the completion rate. How many students are finishing their studies and walking away with a credential?
But that is not the right way to judge massive open online courses, according to researchers at Harvard University and the Massachusetts Institute of Technology. “Course certification rates are misleading and counterproductive indicators of the impact and potential of open online courses,” write the researchers in the first of a series of working papers on MOOCs offered by the two universities. (The Harvard papers can be found here, the MIT papers here.)
Released on Tuesday, the papers make good on a pledge by Harvard and MIT in 2012, when the universities teamed up to create edX, a nonprofit provider of massive open online courses. At the time, the presidents of the two universities said their foray into online instruction would include a major research project aimed at learning more about online courses, especially the kind that they and other exclusive universities had started making available free.
The papers released on Tuesday draw on data from 17 MOOCs offered by Harvard and MIT in 2012 and 2013. A number of academics have begun studying aspects of the MOOC phenomenon, but few academic papers have been published so far.
The first of the working papers, which was written jointly by researchers at both universities, provides an overview of the data from those 17 MOOCs. Some findings:
- 841,687 people registered for the 17 MOOCs from Harvard and MIT.
- 5 percent of all registrants earned a certificate of completion.
- 35 percent never viewed any of the course materials.
- 54 percent of those who “explored” at least half of the course content earned a certificate of completion.
- 66 percent of all registrants already held a bachelor’s degree or higher.
- 74 percent of those who earned a certificate of completion held a bachelor’s degree or higher.
- 29 percent of all registrants were female.
- 3 percent of all registrants were from underdeveloped countries.
Some of these findings reinforce what others have already observed about MOOCs: Few of those who sign up for a course end up completing it. Most MOOC students already hold traditional degrees. Students who sign up for MOOCs are overwhelmingly male.
But looking at percentages such as the ones listed above is a bad way to try to understand MOOCs, the researchers told The Chronicle in an interview.
Completion rates make sense as a metric for assessing conventional college courses, said Andrew Dean Ho, an associate professor in Harvard’s Graduate School of Education and director of the university’s MOOC research. In a conventional course, the goals are generally consistent and well understood: Students want to complete the course and, eventually, earn a credential. The instructors want the same thing.
A MOOC is more of a blank canvas, said Mr. Ho. Some students who register for MOOCs have no intention of completing, and some instructors do not emphasize completion as a priority. Success and failure take many forms.
“It’s reaching a completely different set of students, with different intentions, perhaps, and different ways of seeing the instructors and the content of the course,” said Isaac Chuang, a professor of physics, electrical engineering, and computer science at MIT.
In future studies, the researchers hope to classify registrants according to their reasons for taking a MOOC, “so we can judge the impact of these courses in terms of what students expected to get out of them,” Mr. Ho said.
In the meantime, the Harvard and MIT researchers said they hoped the new studies would help people understand that technology and scale are not the only things that distinguish MOOCs from other kinds of higher education.
“People are projecting their own desires onto MOOCs,” said Mr. Ho, “and then holding them accountable for criteria that the instructors and institutions and, most importantly, students don’t hold for themselves.”
Correction (1/22/2014, 10:58 a.m.): Because of a technological error, this article originally showed the wrong byline, Jennifer Howard’s. It has been corrected; the writer is Steve Kolowich.
