I’m sure that the folks at Robin Hood, a foundation in New York City focused on fighting poverty, had good intentions when they invited organizations to compete for up to $5 million by creating an app that would increase completion rates among students at the City University of New York by at least 15 percentage points above the rate for a control group.
Yet once the competition ended, the foundation did not announce the results of the randomized controlled trials with anything near the publicity it had courted in announcing the contest in 2014, or in naming three finalists a year later. Those finalists were Beyond 12, an organization that promotes student success through online coaching and whose founder I recently interviewed for the Innovation That Matters podcast series, and another nonprofit called Kinvolved, which is known primarily for its work in K-12 settings fostering student engagement by using text messages to encourage class attendance. The third finalist, the company EAB, ultimately decided not to compete.
The experiment ended in August 2018. But Robin Hood never issued a news release announcing the outcome. None of the $5 million was ever awarded.
In September 2019, the foundation did post on its website a two-paragraph statement on “prize results,” explaining that neither of the finalists’ interventions had raised graduation rates among students in remedial courses “by an amount statistically significant compared to the control group.” But how would anyone know to look there to find that out?
And then, that’s it? Unfortunately, despite an earlier commitment, the foundation has not made public the complete third-party evaluation of the work. A Robin Hood spokesman said that commitment didn’t necessarily include publicizing a proprietary document. “The primary purpose of the report was to facilitate an internal decision in determining a winner of the prize competition, or in this case, to declare there was no winner,” the spokesman said.
I’ve seen the 59-page evaluation, dated June 2019, under the condition that I can summarize the findings but not publish the document. I’m glad to have that chance, although I’m pretty sure that education researchers better trained to dive into the data (and all those regression analyses) could identify some deeper and potentially useful takeaways if they could see the evaluation, too. But will they?
For now, several conclusions from the report, by Abt Associates, did stand out to me. The Beyond 12 and Kinvolved apps weren’t as effective as they could be, it says, because:
- Students didn’t use the apps as frequently as the organizations had hoped.
- Under the contest’s timetable, the apps couldn’t be tested ahead of time and then tailored to meet students’ preferences or needs.
- It might have been misguided to expect that a single-purpose app could have a significant impact on a diverse population of students.
In other words, maybe the contest wasn’t designed so thoughtfully to begin with. Maybe it was too steeped in silver-bullet thinking. That’s my conclusion, informed to a degree by what I heard from Alexandra Bernadotte of Beyond 12 and Miriam Altman, co-founder of Kinvolved.
While both said they benefited from being finalists, they also hinted at some frustration with the limitations they were under. Kinvolved was accustomed to engaging with administrators at the schools using its products, but with the app it developed for the College Success Prize, Altman said her team wasn’t allowed to connect with administrators at CUNY. The foundation wanted to measure the impact of the tool in isolation, she told me.
As for Beyond 12, its digital-coaching app typically relies on personal interventions when called for. But those weren’t allowed in the CUNY experiment either. Still, Bernadotte said she was gratified the contest proved that “for the students who are most at risk of dropping out, an app alone is not enough.”
That raises another issue: Even though the contest didn’t demonstrate that a single app could meaningfully raise completion rates — and ultimately promote social mobility — it does seem to have succeeded in reinforcing principles that are foundational to the contestants’ approaches.
The foundation said it does not consider the contest or its outcome a disappointment. “As a venture philanthropy, Robin Hood does not wed itself to any one theory of change,” Kevin Thompson, managing director of communications, said in a statement. “We realize that not every idea will be successful, but we owe it to the more than 1.5 million New Yorkers who live in poverty daily to vet and test as many possible solutions to poverty as possible.” And, he added, the results “should not be misconstrued as a reflection on the finalists themselves,” especially given the contest’s “audacious goal.”
The foundation remains “optimistic that technological innovations have a place in making a difference in educational outcomes,” Thompson said, and it will continue to support those ideas, along with other, more-comprehensive approaches.
Six years ago, when I wrote about prizes as a way to promote big goals, I noted the potential for a contest to put too much attention on finding new ideas, rather than highlighting approaches that may actually work but have yet to catch on. Now I realize that there are additional risks, like a timetable and conditions for participants. And perhaps not wanting to embarrass the finalists was the reason the foundation didn’t formally announce the results. Turnover at the foundation could have also been a factor. But Robin Hood officials did say from the start that there might not be a winner. As audacious as its goal was, its follow-through feels like a bit of a letdown — and a missed opportunity for others to learn from the process.
Got a tip you’d like to share or a question you’d like me to answer? Let me know, at goldie@chronicle.com. If you have been forwarded this newsletter and would like to see past issues, find them here. To receive your own copy, free, register here. If you want to follow me on Twitter, @GoldieStandard is my handle.
