When information-technology and higher-education leaders reunited in Philadelphia in October 2021 for the first in-person Educause conference in two years, buzzwords floated around the panel rooms:
“Data analytics.” “Data-informed” action. “Innovation.”
Indeed, college leaders nationwide — many of them still shouldering deep state-budget cuts that followed the 2008 recession — are increasingly looking to harness data, especially about their students, to make strategic and cost-effective decisions. They’re also facing increasing pressure from the public and social activists to close equity gaps for marginalized students and improve student-success metrics such as retention and graduation rates.
This reality, experts say, has sparked a shift in the past decade to “predictive” and “prescriptive” data use. Previously, colleges collected students’ graduation rates, demographics, financial-aid, and other data largely because this data was required to operate and meet federal reporting requirements. Now, they’re also using data for predictive analytics, adopting early-alert systems that flag advisers when students show signs of dropping out. Faculty members can now evaluate and adjust their learning materials by tracking how often, and for how long, students engage with them in learning-management systems. ID-card-swipe data can determine which facilities students are using most often.
Done well, analyzing and acting on student data can boost student performance and deliver returns on investment. Georgia State University’s adoption of predictive analytics in advising, for example, has helped raise its graduation rate eight percentage points since 2010, with each percentage point adding $3.18 million in revenue annually. Amarillo College, a community college in rural Texas, has leaned on online student surveys to inform its distribution of emergency aid, nearly doubling its graduation and transfer rate in five years.
Still, the use of data to drive decisions is a fairly new shift for many in higher education, and colleges are weighing their ability to know more than ever about their students against the ethical and privacy-related risks that come with amassing more and more information — including through vendor partnerships.
“We have a lot more” data, but “that doesn’t mean it’s better,” says Loralyn Taylor, associate provost for institutional effectiveness and analytics at Ohio University. “It actually makes it harder for institutions to separate the wheat from the chaff in terms of what’s actually useful.”
Flashy tech alone won’t ensure success. Experts say student-data collection and analysis done in the absence of a supportive campus culture, data-defining governance boards, faculty data-literacy training, and transparency, among other things, can actually do more harm to students than good.
“Every school has to have a comprehensive plan for how they are going to change” to do this work right, says Allison Calhoun-Brown, senior vice president for student success and chief enrollment officer at Georgia State. “The innovation is not the technology. The innovation is the change that accompanies the technology.”
Risks to Consider
With predictive analytics, the ethical dilemma to keep in mind is that they may not capture students’ unique circumstances and mind-sets. “The model, by definition, is based on every person that came before [you],” Laura Jensen, vice provost for planning and effectiveness at Colorado State University, says. “It’s not based on [you] in any way, shape, or form.”
With enrollment algorithms, colleges risk “widespread missteps” and discrimination when they focus too intently on data like ZIP codes, GPA, and test scores, and on engagement metrics such as attendance at recruitment events, adds Amelia Parnell, vice president for research and policy at the National Association of Student Personnel Administrators.
These algorithms “don’t know my parents, they don’t know my friend circle, they don’t know my study patterns,” says Parnell. “It’s a tool to be used in a suite of other pieces for a decision. I know it’s tempting when you get a model that continues to come back with certain levels of predictability, to say, ‘Oh, let’s just go there first.’ … [But] that’s where you miss the richness” of a more diverse population.
Behavioral analytics, too, can form an incomplete picture. The University of Iowa, for example, collected data on a semester’s worth of ID-card swipes at the dining hall, generating a “connectedness” score for students based how many meals the students had, and with how many different people. Not addressed with that score, though, is the reality that a student with one or two close friends but no broader network can have a low score but still be happy.
This all underlines the importance of making sure analysis isn’t happening in a vacuum with a machine, experts say. Human beings should still always be reviewing analyses and considering context.
Nothing with data is “cookie cutter,” says Vanessa Roof, assistant director of enterprise data solutions at the University of Nebraska system. Colleges should still ask “and then what?” and have conversations.
The Role of Faculty Data Literacy
Improper analyses are also less likely the more data-literate faculty and staff members are.
Faculty members are constantly surrounded by data. Professors can track their own students’ performance and activity in learning-management systems. Depending on the institution, they may have ready access to aggregated institutionwide data filterable by student demographics and fields of study.
But the ability to make responsible use of that data isn’t an ingrained skill: In the 2018 AIR National Survey of IR Offices, 59 percent of respondents were either neutral or disagreed with the statement that faculty data-literacy levels were “high” at their institution.
“Data literate,” as a simplified definition, means the ability to access, interpret, and communicate data in context. It means knowing to ask questions that foster critical thinking rather than cherry-picking data to justify a decision that has already been made. It means knowing the difference between correlation and causality.
Having faculty members who aren’t data literate can come with repercussions. At Dartmouth College, misinterpretation of Canvas data spurred false cheating allegations against 17 medical students. Or it can be less overt; a faculty member may simply not attempt to draw on data at all, either not knowing it exists or how to access it, or finding it overwhelming rather than beneficial.
That’s a problem because “faculty are key” in this work, says Nic Richmond, chief strategist and vice chancellor for strategy, analytics, and research at Pima Community College. “In terms of the people our students have the most contact with, it’s their instructors in the classroom. … There’s nothing like being in a class, building a relationship directly with a student. … You get to know them.”
What’s Effective in Faculty Training
Ellen Mandinach, director of the Data for Decisions Initiative at WestEd, a nonprofit research and development company, has compiled dozens of data-literacy skills that should be covered when training faculty members. Some of the most pressing for her:
- Knowing when you have bad data. Is the data complete? Is it timely/recent? Is it from a reliable source?
- Responsibly using data. Are you asking the right question about the data? Is it granular enough? Do you have everything you need to make a decision?
- Communication. Can you communicate to a student, to other faculty members, what the data is showing?
Just as with data collection, experts say a campus culture that encourages data use and has set institutionwide expectations — and time — for faculty professional development is critical. It also helps to deliver a clear message that the chief reason for using data is to better serve students, and not to weed out and punish a faculty member whose students are underperforming, Richmond says.
At Pima, faculty members have a wealth of data “at their fingertips.” The college, which served about 33,000 students in 2020-21, used capital funding to adopt a business-intelligence platform in 2015 — called “Pima Reports” — that includes folders of interactive data reports on topics such as enrollment trends, persistence, and retention.
Faculty members can “slice and dice” that data by factors like student demographics, subject code, course number, and instructional delivery method, Richmond says. More personally identifiable and sensitive data, such as grade distributions and financial aid, are kept in more restricted folders.
But simply having this platform doesn’t mean faculty members will know how to access or analyze it, she notes. And that’s where data-literacy training has come in.
Pima launched an analytics academy in fall 2019: a cohort of 20 volunteer administrators and faculty and staff members who met for four days over the course of the semester. Sessions included how to navigate the Pima Reports system, where to find existing student- and faculty-engagement surveys, an overview of mandated data the college collects for Ipeds, and a crash course on the ways faculty members should and shouldn’t use data.
Then Covid-19 hit. In-person training was no longer an option, and Pima was juggling myriad competing priorities that Richmond admits sometimes placed data training on the back burner.
For now, Richmond’s division is offering a monthly, two-hour session virtually — one hour of training, one hour of hands-on practice — on how to use the Pima Reports platform. Leaders run through how to log in, where to locate the reports, and how to filter and search them. They’re also offering this service on demand and customized for individuals, teams, or departments that request training.
Richmond said about 80 faculty members (including adjuncts) had taken a custom training as of December. The college’s goal is to reach 50 percent of its nearly 250 full-time faculty members by the end of this academic year.
The analytics academy is likely to return in some form, Richmond adds, but discussions are still underway about who should take it, the timing and intensity of the sessions, and the amount of hands-on practice needed to be effective.
Brooke Anderson, who teaches writing and humanities courses, took Pima Reports training in October. She says accessing and disaggregating data has provided evidence of what she’s already observed in the classroom: that much teaching on campus is still benefiting the “traditional student,” notably, those who are full-time, “economically comfortable,” white, and male. Those students tend to do better in composition classes, which traditionally dock points for failure to use standard, written English.
The data will inform how she teaches from now on, Anderson says, reinforcing her “student-centered and student-directed” approach. For example, she and her students develop the rubric for writing assignments together. She doesn’t penalize them for grammar or typos — though she’s happy to provide guidance if they ask.
“When they do not see themselves as the problem, they are more likely to not interpret faculty feedback and grades as a reason to drop out of school or not pursue a field that interests them,” she says.
Knowing more about their students allows faculty members to take action. They can make changes as simple as pulling a student aside to ask what support they need to succeed, or swapping in new materials, and as complex as reconfiguring a lesson plan or redesigning a course.
Taking action could also mean piloting new tools and scaling up gradually. The University of Central Florida, for example, expanded the use of adaptive-learning software — which crafts personalized learning paths for students based on their performance — after a pilot program in a few online foreign-language courses saw a 23-percent increase in the number of students earning A, B, or C grades.
Institutional Action
Action can even expand to the institutional level, as with Georgia State, which tracks 800 factors the university has identified as putting students at academic risk.
An early-alert system lets an adviser know when a student displays any one of those risk factors, and the adviser reaches out to the student within 48 hours. An adviser might call, text, or email, for example, if a student has enrolled in both an organic-chemistry course and a calculus class in the same semester — a combination that’s considered “toxic.”
“When we’ve had innovations at Georgia State, they haven’t been because we had the luxury to say, ‘We have enough time [and resources] to innovate,’ it’s because we haven’t had the luxury not to innovate,” says Tim Renick, executive director of the National Institute for Student Success at Georgia State.
Other colleges have found themselves in a similar position of adopting campuswide initiatives to improve student success and retention.
At Hillsborough Community College, a chatbot named Gwen will text students who’ve stopped attending classes with 15 credits or fewer left in their degree, offering assistance for course scheduling, setting up advising appointments, or filling out a financial-aid application.
Many colleges are distributing emergency aid to students who have unpaid balances or financial holds on their accounts, recognizing that more than one-third of college dropouts leave because of financial pressures. Dillard University, in New Orleans, La., for example, built up a philanthropic fund that provides microgrants averaging $2,500 per academic year to students struggling with the cost of attendance.
Another historically Black college, Morgan State University, in Baltimore, homed in on sophomore retention as a way to boost its graduation rates. The university runs a Second Year Experience program that encourages sophomores to participate in either an internship, a research experience, study abroad (pre-Covid-19), a service-learning course, or a campus-to-career trip to, for example, a local radio station or museum.
These opportunities are about “getting students to see the importance of what they’re doing in the classroom,” says Kara Turner, Morgan State’s vice president for enrollment management and student success.
Between May 2016 and May 2020, Morgan State saw its four- and six-year graduation rates reach historic highs, increasing nine percentage points and 13 percentage points, respectively.
Keeping It Sustainable
Resources aren’t limitless, though,and colleges with tight budgets may worry more broadly about how financially sustainable investments in data and analytics are.
Georgia State’s chatbot, through the engagement platform Mainstay, costs about $4 per student a year, a rate the company says is “fairly typical.” The University of Central Florida paid $150,000 in 2020-21 for student licenses for its main adaptive-learning system, Realizeit — roughly $30 per student.
Both Renick and Allison Calhoun-Brown, Georgia State’s senior vice president for student success and chief enrollment officer, emphasize that colleges don’t have to break the bank to do this work. Georgia State, they note, isn’t collecting 800 “discreet personal details” about every single student. The 800 risk factors are based on data the college has long collected in order to be accredited: registration records, majors, minors, degree programs, attendance, and midterm grades, to name a few.
Outside partnerships can also fill resource gaps. Amarillo College, which dispatches a 10-question basic-needs survey to students each semester, leans on more than 60 partners, including lawyers doing pro bono work and staff at a nonprofit health-care facility, to get students the assistance they need. (While it’s drawn from $15.6 million in federal stimulus aid to help meet students’ needs for things like stable housing and child care, that money will run out.)
What’s most important, Renick and Calhoun-Brown say, is the approach — the decision to be proactive rather than reactive.
Taking initiative for identifying and providing help rather than “waiting for students to diagnose their own problems and figure out where to get help … that methodology works at all levels,” says Renick.
Communicating With Students
No matter a college’s resources, experts emphasized that no action will be successful if it’s communicated poorly to students.
“Communicating with students can seem easy, intuitive, and low stakes,” the former New America policy analyst Alejandra Acosta wrote in a multipart 2020 report. “But having good intentions does not guarantee success, and ineffective communication can cause serious harm.”
It helps if the information a college gathers translates clearly and quickly into action to support the student. At Amarillo, for example, Claudia Rubi Zaragoza Hernández, a sophomore, told The Chronicle in the fall that she and friends she’s spoken with aren’t bothered by the college’s questions on topics like food and housing security. “If they collected data and we don’t know why and the college doesn’t do anything with that information, I think we’d be more concerned,” she said. “But at least here … we get benefits from it.”
Especially when communicating sensitive information — perhaps a student is at risk of failing a class or is behind on payment — a message meant to inform and guide can come across as invasive and creepy if framed incorrectly. Action based on predictive analytics can also, if not communicated properly, give students a feeling of being pigeonholed or doomed to a self-fulfilling prophecy.
“Don’t tell me what I can or can’t do,” one student said in a case study published in 2019 that discussed colleges using predictive data to recommend course pathways. “Who can decide my future besides myself?” another asked.
It can even feel discriminatory. Amelia Parnell, of Naspa, recalls an anecdote she heard at a conference a few years ago about a Black man who was feeling “targeted” by “constant emails” about support services. “The campus had made the decision that they were going to focus [outreach] on Black men who were enrolled because they wanted to retain them, because the numbers were showing they were not performing as well,” Parnell says. “And he was like, ‘They don’t know me. They don’t know that both my parents went to college. My dad’s a doctor. I know how to do college.’”
The equity implications expand beyond threatening students’ feelings of belonging. Messaging without clear instructions — perhaps on how to make an advising appointment or fill out a financial-aid form — could confuse a first-generation student who needs these services urgently “but does not know how to schedule one or have relatives and friends who can help,” Acosta wrote in her report.
So what to do? Acosta offered a few strategies for effective communication:
- Social proof. Students are more likely to take action if their college highlights that other students are doing it too.
- Loss aversion. Making students aware of what they may lose as a result of inaction (in a positive way) is likely to encourage them to change their behavior.
- Minimizing hassle. Offer clear and straightforward instructions so that students know exactly what they have to do, how to do it, and can act immediately.
- Timing. Periods such as course registration, midterms, or financial-aid deadlines may be the optimal time to reach students. Holiday break or summer vacation? Less so.
- Preserving choice. Avoid messaging that suggests students have no other option if they don’t take a certain action. Some may not be able to.
And even after all of that action, evaluate.
Colleges should be circling back and evaluating any action they take, says Denise Nadasen, the Association of Public and Land-Grant Universities’ assistant vice president for data and policy analysis — even if they’ve followed all the right steps for responsible data collection and use on the front end. It will ensure the most efficient use of college’s limited resources.
“If we have evidence to show that they are [working], then we should continue supporting them and putting resources there,” she says. “If they are not … then we need to step back, and find out why.”
