Which Ed-Tech Tools Truly Work? New Project Aims to Tell Why No One Seems Eager to Find Out

Every year schools and colleges spend billions of tuition and tax dollars on glitzy digital teaching tools and other educational-technology products. Yet the institutions rarely demand rigorous evidence that those products are effective or take it upon themselves to conduct such research.

Every year ed-tech companies develop and sell shiny new products built more around marketing promises than proven efficacy. Those businesses — and their investors — don’t see a financial payoff in spending their time or their limited financial resources on academic studies.

And every year philanthropists provide millions in grants to educational institutions to encourage the use of ed-tech products without really ever knowing which products truly work well.

“It’s a circle of gridlock,” says Bart Epstein, chief executive of an ed-tech incubator affiliated with the University of Virginia. Everyone says efficacy research should matter, he says. But “universally, everyone thinks it’s not their fault or not their problem” that the research isn’t a bigger part of the purchasing equation.

Now the 18-month-old incubator, newly renamed as Jefferson Education at the University of Virginia, has begun a project aimed at cutting through the excuses and breaking open that logjam.

Jefferson Education has enlisted more than 100 people — professors, entrepreneurs, school and college administrators, policy makers, and foundation leaders — to spend the next year taking a deep dive into the political, financial, and structural barriers that keep companies and their customers from conducting and using efficacy research when creating or buying ed-tech products.

The participants, all volunteers except for 10 professors who will be paid to staff each of 10 groups assigned to particular topics, will survey and interview venture capitalists and university provosts, talk with teachers and professors, and spend time with leaders of ed-tech start-ups to better understand the pressures those constituencies face when developing and buying products. Mr. Epstein says he hopes some of the groups will promise interviewees anonymity, to foster honest answers.

“Behind closed doors,” he says, school and college officials often tell him they’re “terrified” they bought the wrong thing. No provosts would want to say publicly that they have purchased a mishmash of products and “we don’t know if it will work,” he says. “No one wants to look weak.”

Among the questions Mr. Epstein expects them to explore: When a college department is deciding, say, which new writing-support tool to buy, “does the research drive the process, or is it tagged onto the end?”

Likewise, when a company does undertake its own research, is it doing so to demonstrate the true efficacy of its products? Or is it primarily seeking to build a “moat” against a smaller company that might actually have better products?

Mr. Epstein, who was himself an ed-tech entrepreneur before helping found Jefferson Education, says well-financed companies sometimes “do a fancy-looking study” that doesn’t say much but makes it hard for competitors to counter.

More often, he says, companies skip the research altogether. The way the system has evolved, he says, doing so won’t help fuel sales.

Participants in Jefferson Education’s project — whose motto is ostende mihi testimonium, Latin for “show me the evidence” — will present their findings, along with suggestions for change, at an Academic Symposium on Education Technology Efficacy, next May in Washington, D.C.

“We want to help the system value the research more,” says Mr. Epstein, who is also a research associate professor at UVa’s Curry School of Education. To get to that point, “there needs to be a lot of people who think about and care about this issue.”

The symposium’s other sponsors are the Curry School and Digital Promise, a nonprofit organization in Washington that promotes research on educational technology and its use.

‘A Glaring Policy Problem’

Fiona Hollands, who will direct the group studying the role of research in higher-education decision making, says she hopes the project will change current practice. Decision making about educational technology, in elementary and secondary schools and higher education, “is rarely rational,” she says.

Ms. Hollands, associate director of the Center for Benefit-Cost Studies of Education at Columbia University’s Teachers College, cites the way so many colleges rushed toward MOOCs, “because everyone else was doing it,” as a recent, glaring example of that pattern. “The lack of attention to pre-existing research on online learning was just shocking,” she says.

Other groups will focus on such topics as: How do investors and entrepreneurs view efficacy research? How do institutions balance the importance of such research against the “potentially competing preferences of students, teachers, parents, and other end users,” particularly when the ease of use of a product may seem like a more compelling factor? What should the role and goal of the federal government and state agencies be in funding ed-tech research?

Right now, says Mr. Epstein, thousands of schools and colleges wish research was being done. But they lack the resources to do it themselves. “That is a glaring policy problem” that could perhaps be rectified, he says, if a statewide organization could coordinate studies based on several institutions’ shared priorities.

Mr. Epstein hopes to dig deep on that issue himself. He’s helping to lead the group that will explore ways for schools, colleges, and states to crowdsource certain research topics to produce evaluations more comprehensive and rigorous than the less-useful and sometimes duplicative studies now being conducted.

The symposium will also aim its sights at schools of education. Mr. Epstein says few have adapted to the increasingly important role that ed-tech products play in schools and colleges. Colleges of education are not training “procurement ninjas” who are skilled in helping choose the best products based on evidence, he says.

Within the academy itself, he says, “efficacy research is not usually the path to tenure, and that needs to change.” Studies that seek to replicate prior research or to assess how a technology is being rolled out in a school or college classroom are often crucially important steps to effective use an ed-tech product, Mr. Epstein notes. But they aren’t typically considered sexy topics for journal publishers.

Robert C. Pianta, dean of the Curry School and a leader of a group looking at “institutional competence in evaluating efficacy research,” acknowledges that such research hasn’t been a high priority for scholars in the field or for most schools of education, although he says Curry’s curriculum is starting to recognize the need to better understand it.

Most ed-tech studies that are now undertaken at schools of education tend to be performed as consulting projects, an approach that allows the companies that sponsor them to treat the output as proprietary information that may never get published. But Mr. Pianta says he hopes his group can help raise the stature and scholarship standards of ed-tech research. If a professor works with an institution to help it rigorously evaluate its use of technology, “that sounds to me like scholarship,” he says.

Mr. Pianta notes that when the incubator was established, as the Jefferson Education Accelerator in 2015, its mission was narrower. It was established to invest in and advise young-but-established companies poised for growth. Given its ties to UVa, the organization said it would work only with companies willing to prove their worthiness through research.

The organization still works with six to eight such companies each year. But Mr. Epstein says he hopes this effort will have an impact not only on those companies but on the entire ed-tech sector.

Goldie Blumenstyk writes about the intersection of business and higher education. Check out www.goldieblumenstyk.com for information on her new book about the higher-education crisis; follow her on Twitter @GoldieStandard; or email her at goldie@chronicle.com.

Join the conversation about this article on the Re:Learning Facebook page.