Candace M. Thille helped kick-start the move to bring big data to college teaching.
She has founded the Open Learning Initiative at Carnegie Mellon University, won millions of dollars in grants, and been a fixture on the lecture circuit about the power of so-called adaptive learning, where data-powered algorithms serve up content keyed to what a student is ready to learn next. Publishers, venture-capital investors, and foundations have followed her lead. They’ve poured hundreds of millions into new companies and new products vying to score big contracts with colleges, sometimes promising to be the “robot tutor” for struggling students.
It seems like a classic business success story.
These days, though, Ms. Thille has begun to have darker thoughts about an industry she helped spark.
She still believes that adaptive learning will become an increasingly important tool in teaching. But she fears that rapid commercialization is exactly the wrong way to foster innovation at this early stage. What’s more, she thinks professors and higher-education leaders are making a dangerous mistake by letting companies take the lead in shaping the learning-analytics market.
Ms. Thille, who moved to Stanford University’s Graduate School of Education in the fall of 2013, has only recently begun to go public with that critique, voicing it to a few small audiences. But as she shared during extensive interviews with The Chronicle over the past few weeks, it’s a message she hopes college leaders and professors will heed, if only because she’s a messenger who understands, despite all the hype, both how “crude” and simplistic many of the products are today, and how educationally valuable they could one day become.
Her concerns boil down to these:
- Colleges should have more control over this field. Like it or not, she argues, using data to predict student needs and deliver the right material at the right time will become essential. “And a core tenet of any business is that you don’t outsource your core business process,” she notes.
- Companies aren’t as well equipped to develop and test new teaching algorithms as colleges are. She argues that colleges are the ideal living laboratories for any teaching system because they are home both to the research on learning and the actual teaching. As she puts it, “You have a very quick feedback loop, where the research informs the practice and the practice informs the research.”
- When companies lead the development of learning software, the decisions those systems make are hidden from professors and colleges. Ms. Thille says companies that won’t share their processes are essentially saying, “Just trust the black box.” To most academics, she says, “That’s alchemy, that’s not science.”
The proprietary “black boxes” are the algorithms that might automatically serve up, say, an extra lesson on quadratic equations when a student’s responses to a quiz indicate that she didn’t quite grasp the concept. Every algorithm, and every decision about what data it will weigh, is also ultimately a pedagogical judgment call. For the companies selling adaptive software, “that’s where the gold is,” says Ms. Thille.
She contends that the demands of the market — with venture capitalists expecting returns 30 times what they’ve invested and companies facing pressure to deliver products priced so they don’t scare away customers — will inhibit innovation rather than foster it, even if the companies have the best intentions.
Ms. Thille (it’s pronounced “Till”), who is in her late 50s and spent 18 years in the private sector before joining academe, is a person who chooses her words carefully. She recognizes that her critique of the burgeoning learning-analytics marketplace is also fundamentally a critique of the commercialization model of the investor-fueled ed-tech sector, and in some sense, of her own Stanford community.
But she is also convinced that colleges need to find ways to raise the money for research and development of learning software so that companies don’t end up owning the classroom delivery system of the future. Colleges could come together to build such systems, or perhaps the federal government could step in, as it did with the Darpa research that led to the Internet. Society has stepped up before for matters of importance, notes Ms. Thille. “I would claim that this is one of them.”
Big Claims
Ms. Thille’s argument could easily be dismissed as naïve, or even self-interested — she does, after all, head up a research lab that lives by grants. But it meshes with a broader national conversation now surfacing among academics and other experts over the growing role that data and algorithms play in higher education.
Data-driven technologies already touch many corners of the student experience. “Student success” tools send text messages reminding students to see an adviser. Course-suggestion engines at some colleges can recommend that a student switch majors, from pre-med to history, if he did poorly in Biology 101. But educators are especially focused on the ways data engines are used in teaching.
Companies such as Knewton, whose chief executive boasted to an NPR reporter that the software was like “a robot tutor in the sky that can semi-read your mind,” epitomize the problem, says George Siemens, an internationally known theorist on digital technology and the professor who co-taught the very first MOOC, in 2008.
“They make very bold claims, but they aren’t involved in the research community at all. That means we can’t validate their algorithms. We can’t validate the results that they say they’re getting,” he says. “That’s a system that doesn’t serve the future of education well at all.”
They make very bold claims, but they aren’t involved in the research community at all. That means we can’t validate their algorithms.
He compares the state of learning platforms to the earliest days of networked computing, when it wasn’t clear whether the open Internet or a paid service like America Online would become dominant. If the learning-analytics model goes the way AOL hoped to go, says Mr. Siemens, “we don’t have the ability to create a rich ecosystem on top of that platform.”
The chief executive of Knewton, Jose Ferreira, declined to comment for this article.
A small community of academic researchers and others are beginning to press on this issue. Mr. Siemens hopes others will too. “Students should care an awful lot because they’re the ones being sorted algorithmically,” he adds. Among those concerned is Ms. Thille’s successor at Carnegie Mellon, Norman Bier. Without greater transparency for the way tools direct students and dashboards signal their progress to professors, “we face a real danger of hurting our students,” he says. The systems need to be more open, he says, so professors can understand and trust them without having to get an advanced degree in computer science.
But the trend is going in the other direction. More and more colleges are turning to the commercial market for their adaptive-learning products, often in response to vigorous marketing by companies. That includes Acrobatiq, a company that was spun off by Carnegie Mellon after Ms. Thille left. The university still owns a stake in it.
Acrobatiq advertises that its products are “powered by Carnegie Mellon” and “based on cognitive science and educational theory” of the Open Learning Initiative, even though none of the company’s products use OLI-developed software.
Eric Frank, Acrobatiq’s CEO, counters that companies can innovate, and notes that his company focuses on the important role of making sure its products work for a broad range of professors. “We spend less time refining our predictive models and algorithms and more time trying to help faculty to use them,” he says.
As more and more colleges work with more and more companies, he acknowledges, researchers will lose out on opportunities to collect large sets of data on how students learn and use them to advance the field because the data will be “bifurcated into these little fiefdoms.” Acrobatiq’s contracts, he notes, do ensure that colleges own the data. But he notes that the learning-analytics industry still lacks the kinds of standards and protocols that would make it easy to extract, organize, and share such data among institutions for research purposes.
Even some foundations and education associations can operate in ways that undermine the momentum for open learning analytics. For example, in inviting universities to take part in a new $4.6-million Bill & Melinda Gates Foundation grant for the members of a new Personalized Learning Consortium, the Association of Public and Land-Grant Universities specified that applicants could use only 19 specific products approved by the foundation. All of them are owned by companies. One of them is Acrobatiq; the Gates foundation is also an investor in that company.
The foundation specified those “approved suppliers” for the quality of their products and because they “appear likely to be robust and sustainable for the future” as businesses, according to the request for proposals for the grant.
Travis Reindl, a spokesman for the Gates foundation, said last week that the list was not “the be-all and end-all of the possible providers” for the grantees and that other products could be added even after the winning institutions are selected. at the end of May. Asked specifically about whether open courses from OLI or those Ms. Thille is now creating at Stanford would be allowed, he said, “APLU will provide the updated provider list to the selected institutions. The list is still being developed.”
As for Acrobatiq, he said the foundation was “not in any circumstances” trying to favor that company. “There is no thumb on the scale” for Acrobatiq, he said.
Deeper Learning
Ms. Thille, who now co-directs a team of 15 master’s and Ph.D. students in the sleek, high-ceilinged Lytics Lab, on prime real estate on the Stanford campus, can’t pinpoint an “aha!” moment when she began to feel a sense of urgency for her concerns. While she maintains many cordial relationships with the ed-tech crowd in her Silicon Valley neighborhood and around the country, she was influenced in part by seeing the flood of investments and acquisitions in the market: Knewton alone has received more than $157 million, including $10 million this month, according to CrunchBase.
One of her Ph.D. students was another influence. Last spring he came up with a wholly new approach to creating predictive-algorithmic models for deciding what content a student should see next in an adaptive course. She thinks his “deep learning” approach runs circles around Bayesian Knowledge Tracing, the approach used by OLI, her Stanford lab, and many of the commercial learning-analytics products.
“As a researcher I was thrilled,” says Ms. Thille, beaming at the thought of a graduate student who could “beat my stuff” and make it better. “But you can imagine if you were a company whose whole proprietary value proposition was built on, and built around” the existing model. Companies, she says, would find it pretty difficult to say, “OK, that’s not good anymore, now I’ll switch over to a completely different method.”
That deep-learning method may prove not to be ideal either, says Ms. Thille, but it illustrates just how early-stage the field of learning analytics really is. For that reason, she says, colleges should be cautious in their dealings with companies. “Go ahead, experiment with some of these systems,” she advises. “Just don’t get locked in for a long-term contract and develop your whole approach around a particular system, because it’s going to change, or it should change.”
Back when she was at Carnegie Mellon, Ms. Thille initially preached the value of adaptive courses not so much as a tool for students but as one for professors, a way for them to gain greater insights into how their students were making their way through the material. Later, studies found at least some of the methods were also helping students complete courses more quickly and successfully than standard courses were.
The courses developed at OLI were important, she says, because they were created collectively by professors at all kinds of institutions with all kinds of students. “We need the robustness of that perspective and that context to design an environment that can meet the needs of the diversity of students.”
But until she joined the faculty at Stanford, Ms. Thille had never actually taught a course — which only fueled some of the skepticism of her work from professors around the country who saw (and may still see) adaptive courses as an attempt to replace live professors.
Ms. Thille used to say publicly that any professor who could be replaced by an adaptive course deserved to be. She insists she always meant that as a compliment to the insights and skills that professors bring to teaching that an adaptive course could never replace. But she no longer uses that phrase because too many people took it the wrong way. She says she still believes it, though.
Ms. Thille received her doctorate from the University of Pennsylvania in 2013. And today she does regularly teach courses — four a year. That’s only reinforced her faith in the value of adaptive courses to instructors.
In her “Introduction to Data Analysis and Interpretation” course, where she uses OLI statistics courseware, she sees exactly what the students understand, and “based on that, I decide what we’re going to do in class the next day.”
In another course, “Engineering Education and Online Learning,” where she doesn’t use OLI courseware, “I’ll tell you, it is frustrating. I feel like I’m flying blind,” she says. “I have no idea what meaning they’re making of it.”
As part of her latest work, she’s also creating a way for professors to see why an algorithm does what it does, and to select different “skill-modeling” approaches if they prefer them. The professor wouldn’t have to be “some big data geek” to see what’s going on, she says. And even if professors aren’t interested in digging into that level of detail, they could have the confidence of knowing that the system had been “peer-reviewed in an academic way.”
Commercial products, of course, offer some insights to professors, too, but not in the same detail. And some professors who use them say their opacity can be frustrating.
We need to understand how students learn, and we’re not getting that.
Irene R. Foster, an assistant professor of economics at George Washington University who says she loves technology and new products, uses a McGraw-Hill product called Aleks to help her students brush up on the algebra they need for the course.
“We’re better off with it than without it,” she says, but she finds its analytics engine a bit crude. “I don’t really know how smart it is,” she says. When students miss problems, it just generates more questions on the same topic. “I’m not sure if it can fully capture why they are missing the problems.” And in some cases, she says, the software doesn’t distinguish whether the student doesn’t understand the mathematics concept or simply messed up the underlying arithmetic. “We need to understand how students learn, and we’re not getting that.”
R.G. Wilmot Lampros, chief product officer for Aleks, says the ideas that undergird the product, referred to as Knowledge Space Theory, were developed by professors at the University of California at Irvine and are in the public domain. It’s “there for anybody to vet,” he says. But McGraw-Hill has no more plans to make the product’s analytics algorithms public than Google would for its latest search algorithm.
“I know that there are a few results that our customers have found counterintuitive,” he says, but the company’s own analyses of its algebra products have found they are 97 percent accurate in predicting when a student is ready to learn the next topic.
As for Ms. Thille’s broader critique, Mr. Lampros is not persuaded. “It’s a complaint about capitalism,” he says. The original theoretical work behind Aleks was financed by the National Science Foundation, but after that ended, he says, “it would have been dead without business revenues.”
On the Horizon
Ms. Thille stops short of decrying capitalism. But she does say that letting the market alone shape the future of learning analytics would be a mistake.
Colleges should be investing in learning analytics in the same way that they invest in maintaining their buildings, she says.
Some moves on the horizon could, in fact, give a boost to the movement to open up learning algorithms.
In June, Carnegie Mellon announced that it would openly license the OLI software, along with several other key pieces of software that are collectively based on more than $30 million in research. The university said it hoped that the technologies could serve as the foundation for a large-scale, open-source collaboration with other universities interested in forming a community of users and researchers in the field of learning analytics. Mr. Bier says they should be available by the summer.
Mr. Siemens says he and several other experts in learning analytics plan to seek NSF funding for a major project in open-analytics research.
Ms. Thille, meanwhile, is taking a page from the collaborative approaches she’s used in creating OLI courses to develop some new financing models for open-analytics research.
One idea, she says, could be a new technology platform to allow crowdsourced contributions from universities, enabling even “micro but meaningful participation” by professors who may lack the expertise or the time to play a major role but still want to contribute to the effort.
With the vocabulary of a data scientist, not a PR pro, Ms. Thille paints the choices starkly. “To commodify it at this point in time, when we’re still doing very active research in it, almost assures that we’ll get less innovative, suboptimal products,” she argues. But if colleges themselves took up the learning-analytics challenge to “co-construct” learning activities, institutions could both “support teaching and learning and help us understand better how learning happens.”
Goldie Blumenstyk writes about the intersection of business and higher education. Check out www.goldieblumenstyk.com for information on her new book about the higher-education crisis; follow her on Twitter @GoldieStandard; or email her at goldie@chronicle.com.
Join the conversation about this article on the Re:Learning Facebook page.
