Has there ever been an enterprise that produced so much data to so little effect as higher education? We are drowning in data, awash in analytics. Yet, critics demand even more data, contending that higher education remains persistently opaque and lacking true accountability.
Here’s a heretical thought: Perhaps the problem is not a lack of data, but rather, that metrics alone are a poor measure of accountability. Our critics prefer lists over paragraphs, but sometimes words are important to interpret statistics.
The data industry is huge, including magazine rankings and credit-rating agencies; accreditors; and the mother of all data collections, housed at the U.S. Department of Education: Ipeds, the Integrated Postsecondary Education Data System. Easy access to voluminous data allows just about anyone to extract random factoids as evidence to assail or affirm collegiate value. Politicians assail high-tuition rates as bad for consumers, but Moody’s rewards them for generating ever-higher net-tuition revenues. Critics pummel elite universities for failing to enroll enough low-income students, while berating colleges that enroll majorities of Pell grantees for low graduation rates. More nuanced analyses of the relationships among high-net tuition, volume of Pell grantees, and graduation rates rarely make it into a public discussion that fixates on the numbers, not the narrative.
Big data is helpful to understand megatrends like the impact of student-debt burdens by race and ethnicity, the alarming growth in discount rates, or changes in demand for majors. But statistics are no substitute for professional judgment about the meaning of data for a specific institution. Unfortunately, magazine rankings and the federal College Scorecard choose to present isolated data points as institutional quality measures without interpretation.
Qualitative measures are also important for accountability analysis. Rankings are silent on the ways in which the first-year faculty members help students discover that they really can learn statistics, write laboratory reports, analyze complex texts, conduct research, or engage in professional work through internships. The College Scorecard does not provide data on the campus climate for women or students of color, or the scope of services for students with disabilities, or food pantries and support for students who are also parents.
Accreditation has always been the place where both quantitative and qualitative evidence is presented within the larger institutional context; interpretation of performance data through the lens of mission and student-body characteristics is essential to level-set the basis for continuous quality improvement. Even more important are the collegial conversations among visiting teams, institutional leaders, and faculty to focus on challenges needing serious repair and opportunities to move forward constructively. Those conversations, summarized in team reports, often remain private, a fact that frustrates critics craving public shaming of institutions that fall outside of traditional benchmarks.
In recent years, pushed by the critics who push Congress and the U.S. Department of Education, accreditation has inexorably moved toward even more data-driven assessment processes in both regional and specialized accreditation. Whether this migration has produced more accountability is unclear. While the idea of self-study and collegial peer review continues, the hegemony of data analytics threatens to diminish the most useful parts of the accreditation process in the collegial discussions that honor mission and institutional context while also challenging institutions to improve.
Some elite universities lobbied for this change on the theory that if they surpass some normative benchmarks, they should not have to bear the burden of the more onerous hands-on accreditation processes beyond, perhaps, cursory reviews. Aside from the arrogance of insisting that some universities are above collegial scrutiny (the climate that fostered the Varsity Blues scandal notwithstanding), the use of data to exonerate wealthy elite schools also perpetuates higher education’s caste system. Institutions serving large numbers of at-risk students will probably not qualify for lesser scrutiny since their students move through college at variance from traditional norms; the more variance, the deeper the scrutiny.
This dichotomy illustrates one of the fundamental problems of the current fixation on the superiority of statistics alone to measure institutional worth: Many of the data sets are premised on an increasingly outmoded model of collegiate attendance formed in the days when the majority of college students were full-time, residential 18- to-22-year-olds with parents paying their bills.
Today, that traditional stereotype accounts for fewer than 20 percent of all college students; the vast majority of students now have what used to be called “nontraditional” characteristics — variably attending part-time or full-time as their schedules or finances allow, working while going to school, parenting and caring for families, paying their own way, attending multiple institutions. The data methods have not caught up with the profound changes in the characteristics and behaviors of student populations.
Consider the graduation rate, one of the most frequently cited but badly constructed data points. The rate measures seat time in one school using first-time full-time enrollment. If anything, the rate is a measure of brand loyalty as well as an index of admissions risk — the higher the rate, the lower the risk the institution took at the point of admission.
The rate reinforces an outmoded view of intercollegiate transfer as somehow shameful evidence of institutional deficiency rather than a legitimate consumer choice for many reasons. By treating transfer students as dropouts, even if they complete degrees, Ipeds distorts completion rates and harms institutions that do important work providing foundational pathways for post-traditional students.
Factoids about earnings after graduation are another example of data gone awry. The U.S. Department of Education vacuums data from IRS tax returns to come up with its infamous “salary after attending” on the College Scorecard; the number is actually the median of salaries extracted on a cohort 10 years after they first received federal financial aid, which is not the same as earnings 10 years after graduation. The factoid is also agnostic on critical issues that affect salaries such as gender and race discrimination in employment, geography, and career pathways that are a matter of consumer choice, not institutional determination.
Accountability is essential, yes, but the methods and metrics must be right for the colleges and students examined. Those who shine spotlights on institutional data must do a better job of presenting the full context. Knowing the flaws of graduation rates or earnings data, for example, it is simply unethical to cite the data as indicia of institutional quality with no larger interpretation. We might also ask why certain data collection continues if the data sets are poorly constructed or outmoded. Most of the data sets we are using today are premised on notions of college attendance in the late 20th century. My institution, Trinity Washington University, looks nothing like the college I attended in 1974, and it is not alone in that transformation.
Change in student demographics, programs, and delivery methods have been rapid and radical. Rather than simply creating more data sets to mask the deficiencies of the existing collections, we should insist on a more refined and carefully targeted data framework, including necessary interpretative components, aligned with the realities of students and institutions approaching the mid-21st century.
