Higher Ed’s Data Experts Face a Crossroads

Latino students had been enrolling at Indiana University-Purdue University Indianapolis in steadily growing numbers in recent years, but something was stopping them from graduating at the same rate as their peers. So senior leaders at the institution formed a task force last year to come up with solutions.

At one point during the group’s meetings, some of the members proposed ways to attack the problem based on the assumption that the one-year retention rate of Latino students at IUPUI was lower than that of the student body as a whole — a trend that plays out at many colleges nationwide.

But IUPUI was different. And one task-force member, Michele J. Hansen, was especially qualified to say so.

Ms. Hansen, the assistant vice chancellor for institutional research and decision support at IUPUI, offered up hard data that dispelled the task force’s misconception. She explained that Latino students at IUPUI persist from the first year to the second at slightly higher rates than do students in general. The problem was elsewhere.

Ms. Hansen drew the group’s attention to a campus-climate issue that was likely to work against the university’s efforts to recruit and retain Latino students for the long haul: Just 2 percent of IUPUI’s tenured and tenure-track faculty were Latino in 2016, compared with a student body that was 7-percent Latino.

Another group was already tackling the issue of recruitment and retention of Latino faculty at IUPUI at the time. But with these climate factors in mind, the task force’s recommendations on how to recruit and retain more Latino students included creating support groups among entering Latino students since faculty to serve as mentors were few, and providing more scholarships and need-based aid to attract Latino and other underrepresented students.

Ms. Hansen considers her contributions to the task force to be a key example of how institutional researchers can help guide discussions with their data-informed understanding of an issue, rather than simply supply information — often after decisions are made.

“It’s critical for us to be at the table and part of the conversation so we can hear the issues firsthand,” says Ms. Hansen, whose unit is in the vanguard of institutional-research offices transforming how they operate. “We’re able to better understand the context of the problem, and we can be proactive in providing information before decisions are made.”

The prominence that institutional research has gained on IUPUI’s campus is an exception. It also reflects, in part, how higher education’s relationship with data has changed. On many campuses, data are often housed across departments, and their lessons left untapped. If colleges are to accomplish their goals and prove their worth to an increasingly skeptical public, it makes sense for data-informed decision making to be a linchpin in those efforts — and for institutional researchers who collect and interpret those data to play a central role.

Yet institutional-research offices — long known as home to a university’s data crunchers — are battling an identity crisis that has played a key role in keeping the field from reaching its full potential. For institutional-research offices to be most effective, observers say, they’ll need to embrace a different way of doing business, including providing data support to a wider pool of people on campus, skillfully putting data in the right context, and pushing for a seat at the table early in the decision-making process when their input can better shape the outcome.

For some institutional-research professionals, this will mean acquiring different skills — most of them not technical, but interpersonal, educational, and political. And at many colleges, senior administrators will need to create a supportive environment for institutional researchers to work under a new model.

“It’s a new day, and people will need new kinds of skills,” says Randy L. Swing, former executive director of the Association for Institutional Research. “This is a field that’s highly in transition.”

Institutional research’s earliest roots, according to a book about the institutional-research association’s first 50 years, are said to date back to 1701, the year Yale University was established. Yale’s founders studied the organizational structure at Harvard University to decide which governance model it wanted to follow. During the rest of the 18th and 19th centuries, some form of institutional research took place as colleges relied on data to make decisions about such things as curriculum or instruction.

In the mid-1960s, the formation of the Association for Institutional Research gave a stamp of legitimacy to the field. Colleges sought out researchers whose technical and analytical skills were their biggest strengths. The job didn’t require being able to talk through what numbers do and don’t mean.

For a long time, doing institutional research meant working in the equivalent of a service unit, crunching numbers or providing analysis largely at the request of two clients: the president and the provost.

The time-consuming task of collecting and reporting data to state, federal, and accreditation officials was also a key part of the work. Today, especially, that task leaves thinly staffed offices little time to do the kind of in-depth data analysis that could help senior leaders tackle problems with more precision.

The current demand for data reporting, combined with one- or two-person staffs, means “a lot of people are just keeping their necks above water,” says William E. Knight, assistant provost for institutional effectiveness at Ball State University. “People want to add value, but there’s no time.”

Adding to institutional researchers’ workload over the last decade or so has been the movement to hold higher education accountable. Institutions are under pressure to demonstrate that they’re worth the money students pay to attend. Senior officials know the way to do that is with data — supplied by institutional researchers, who ideally can also put the information in context. Meanwhile, the list of people who are thought to need access to data is no longer so exclusive. Faculty, staff, and even students are considered decision makers now.

In a move to formally capture the changes in the field that were already underway, the institutional-research association, in 2016, released a “Statement of Aspirational Practice for Institutional Research.” It lays the groundwork for a new model in the field that focuses on how institutional-research offices can support student success by providing data to all the parties invested in the core mission of the college. IUPUI is one of 10 institutions that participated in a formal pilot program that sought to test the vision outlined in the statement.

The statement’s authors — Mr. Swing is one of them — acknowledge that its goals remain aspirational at many institutions. Some institutional researchers are unaccustomed to using the data to tell a story or to suggesting solutions to problems that the numbers reveal.

“By and large, people in IR are really ill-equipped to do that kind of work at their institutions,” says Mark Salisbury, assistant dean and director of institutional research and assessment at Augustana College, in Illinois. “They were hired to do a different job than what exists now. Institutions are saying, Help us make sense of this data and tell us what we need to do to change, and that’s a different skill set than being short-order cooks for data.”

Guided by the association’s aspirational statement of practice, IUPUI in 2015 consolidated all the offices that generate data to support decisions and called the new unit institutional research and decision support, now housed in academic affairs. Ms. Hansen says that although the new office is considered large and decentralized, “we basically operate like a bunch of mini data teams.” Her coworkers provide data support to individual offices, like the division of enrollment management or the division of diversity, equity, and inclusion.

This summer Ms. Hansen was promoted to her current position and given oversight of the university’s testing center, which collects data through the course-evaluation instruments it administers, which are then used to support decisions about instruction. She now leads about 20 people who, in line with the statement of practice, serve a wider pool of decision makers. Some level of data crunching and order-fulfillment is still a part of their work.

On paper, it’s easy to tell that the evolution of the field of institutional researcher has been fitful. Today’s job ads for directors of institutional research call for “seasoned computer and analytical skills” and “expert skills with data management and query tools” — all par for the course in a field that values technical and analytical abilities. Rarer are mentions of an outgoing personality, superb communication skills, or the ability to build relationships across campus — traits that matter now more than ever before.

They’re especially helpful when it comes to using data to tell a story that goes beyond the numbers, provides context, and makes it clear what the data actually mean for decision makers, a skill Mr. Knight says is key.

“Our job is not done when we produce the numbers,” says Mr. Knight, the author of a book on leadership and management in institutional research. “We have to give them their voice, and an important part of our job is to translate them so people know what they should do.”

Yet an important part of that translation is knowing the context surrounding the data. Understanding how an institution works and how faculty members do their jobs is key, says Richard Howard, an institutional-effectiveness consultant who formerly directed offices at four public universities. “You need to be willing to be out talking to people to learn as much as you can,” Mr. Howard says.

Mr. Knight says he has also benefited from using a different style of intelligence gathering. He suggests reading a book on the history of the campus or “sitting in the back of faculty-senate meetings and not saying anything, just to find out what things people are struggling with.” There’s even helpful information about the campus to glean from the student newspaper, he says.

Institutional researchers will also need to hone their teaching skills to focus on data literacy. Just because faculty and staff have unprecedented access to data doesn’t mean they understand what the numbers signify, know how to interpret them, or a have a good sense of how they could be used.

“Decision support has now become something that touches every level of an institution,” Mr. Swing says. “We see academic advisers saying, ‘I need data support’; department chairs want data to help them decide how many courses to offer or what time of day to offer them. They’re all making decisions.”

At IUPUI, Ms. Hansen runs a data-inquiry group where select faculty, staff, and academic administrators gather to examine various types of data, ask questions about how to interpret it, and discuss how it could be used to inform decisions. Kristina Horn Sheeler, a member of the group and associate dean for academic programs in the School of Liberal Arts at IUPUI, uses enrollment data as a guide when helping 12 programs and 20 departments figure out course scheduling.

For instance, when a course hasn’t quite met its enrollment threshold at a certain point during registration, Ms. Sheeler looks at data to see when enrollment tends to level off to help her make a decision about whether leaving the course open will make a difference.

“Sometimes you can see, OK, this course has already passed its peak, but other times you can see that the time when it’s going to gain in enrollment is coming up,” says Ms. Sheeler, who is also a professor of communication studies.

In years past, Ms. Sheeler had to piece together reports from multiple places. The data she can access now, she says, gives her additional standing in the face of potentially unpopular pronouncements.

“When I go to my programs and departments and say, You can’t offer this many sections going forward, it’s not just me being a pain in the neck. I have the data to back it up,” Ms. Sheeler says.

In the math department at IUPUI, Jeffrey X. Watt used data to help place about 8,000 students in the right math courses this fall. Two years ago the department began using a new, unproctored math placement test. For most students, the placement test served its purpose. But there was an exception: those who scored well on the placement test but fared poorly on standardized ones like the SAT and ACT. For these students, the placement test made it more difficult to steer them into a course that matched their true ability. So the department decided in January to compare placement-test scores with SAT or ACT scores to see how closely the two matched. With the help of Ms. Hansen’s staff, the math professors could see that they needed more data: Nearly 8 percent of students needed to take an additional math placement test this summer, which was proctored.

“We wouldn’t want everybody to end up in an engineering calculus course,” Mr. Watt says. “We needed a lot of data to decide how we were going to determine who would take the proctored test.”

After the fall semester, the department will analyze the data to see how those students performed to help determine the next math course they should take.

A 2015 survey of institutional-research offices provides some insight into the barriers keeping a transformational shift in the field from happening. Most offices of institutional research are three-person shops, a factor that can contribute to a heavy workload. For 83 percent of offices, filing data to federal agencies is a core responsibility.

Christine Keller, the new executive director of the institutional-research association, says the focus on data-informed decision making provides a tremendous opportunity for the field. “In the era of big data, we still need people who can interpret the data,” Ms. Keller says. “It’s easy to get a data point, but what does it mean? What’s the context for it?”

Yet there are challenges to achieving the status that the association aspires to. “At some campuses, institutional research is still off in the corner somewhere,” Ms. Keller says.

Organizational support for institutional research can make the difference, particularly when it comes to democratizing data, which refers to the process of making data widely accessible. But being included among an institution’s key decision makers brings its own set of challenges.

“We’re members of so many councils, our leadership has made sure of that,” says Ms. Hansen. “It involves a lot more meeting time than I’m used to.”

Among the meetings Ms. Hansen attends are the deans’ council — which meets monthly and focuses on things like enrollment management, faculty hiring, or planning new degree programs — and a gathering of senior leaders that meets biweekly. These biweekly meetings serve a larger goal, says Kathy Johnson, the executive vice chancellor and chief academic officer to whom Ms. Hansen reports, and that is to “relentlessly use data to help find and solve problems that could ultimately affect student success at IUPUI.”

The upside of being in the mix, Ms. Hansen says, is that institutional research can serve as a link between the many corners of campus. “Because we work with multiple sources of data when we’re trying to solve really complex problems, we benefit from having a more comprehensive view of the institution.”

Ms. Johnson is also a key supporter of democratizing data, which is well underway at IUPUI. But that means that Ms. Hansen and her staff must bolster their colleagues’ data literacy. “Now that everyone has access to the data,” Ms. Johnson says, “that’s definitely not the end of the story.”

Indeed, data without context can be misused, with an impact that extends far beyond campus. Several public-university systems have grappled with the fallout in recent years from efforts that have been perceived as using data as a weapon. For example, efforts to measure faculty productivity often begin with calls from lawmakers or trustees who want to know how many courses professors teach each year. The goal is typically to provide evidence that professors aren’t doing enough work to earn their salaries. But course-load information is the kind of data institutional researchers know has nuance — a single class could serve several hundred students, or a faculty member with no courses on the books could be on sabbatical. Without that insight, the data could lead to an erroneous conclusion.

Collecting an increasing volume of data, and putting data in more hands, means institutional-research offices have to take on new responsibilities, like making sure people really understand what the numbers are and aren’t saying, says Mr. Swing.

“As everybody has access to the data, it’s going to get messy,” he says. “That’s better to me than nobody has any access to the data. I don’t worry about that messiness. The community will take care of that over time.”

Audrey Williams June is a senior reporter who writes about the academic workplace, faculty pay, and work-life balance in academe. Contact her at audrey.june@chronicle.com, or follow her on Twitter @chronaudrey.

Correction (9/1/2017, 12:25 p.m.): The new unit created at IUPUI in 2015 was called institutional research and decision support, not institutional research and data support, as this article originally said. The text has been corrected.