Here’s a seemingly simple question: How have the educational-attainment rates of various groups of Americans changed over the years?
It’s a question with considerable impact. For example, the answer could help determine how well the country’s colleges and universities are meeting its labor needs, and how equitable education is across various demographic groups.
And the answer is? Well … that’s the hard part.
In a new report from George Washington University’s Graduate School of Education and Human Development, Sandy Baum and two co-authors asked that and a handful of related questions, but they found the answers far more elusive than one might suspect. The result is a pretty good guide to the challenges of education data, including these five “gotchas” that are good to remember:
Combining Data
In order to answer the questions set out in the report, the George Washington researchers needed to assemble data from eight sources:
- American Community Survey (ACS), for demographic data
- Beginning Postsecondary Students Longitudinal Study (BPS), for degree attainment
- Current Population Survey (CPS), for labor statistics
- Integrated Postsecondary Education Data System (Ipeds), for a wide variety of college data, including graduation rates
- Digest of Education Statistics, which is largely drawn from Ipeds
- National Postsecondary Student Aid Study (NPSAS), for student aid
- National Student Clearinghouse (NSC), which tracks students as they transfer between colleges
- Survey of Income and Program Participation (SIPP), which is a household survey that includes educational-attainment data
From differing methodologies to varying accuracy of the underlying data, it’s not hard to imagine the difficulty in reconciling those disparate databases. Or, as the report puts it, “some of the differences in these data sets lead to systematic differences in the results they generate.”
Determining Definitions
Sometimes the challenges in unifying data start with developing a standard set of definitions. While some people might use “attainment” and “completion” interchangeably, they are not synonymous. Attainment measures “the highest level of education that individuals have completed,” while “completion” describes “how many people finish the programs they begin.”
One could hold the completion rate steady, but expand enrollment in order to increase attainment. Likewise, one could hold enrollment steady, but expand completions to increase attainment. The two are not interchangeable.
Even terms like “college graduate” and “undergraduate degree” can be open to interpretation. In 2011-12, 3.8 million students completed their undergraduate studies. Yet fewer than half — 43 percent — earned bachelor’s degrees. The other credentials were short-term certificates and associate degrees. That matters because enrollment rates include students at two-year institutions, many of which don’t offer bachelor’s degrees. Their students, therefore, could never “attain” a bachelor’s degree, even though they could certainly complete their degrees.
Demographic Shifts
For those seeking a look at demographic trends, beware. How people identify has changed over the years, and different sources use different methodologies. For example, before 2008, Ipeds used a different methodology than did the Census Bureau, though now the two are mostly aligned. Furthermore, some reporting combines ethnic groups, which can mask variances within one of them. For example, the categories of whites or blacks sometimes include people of Hispanic origin; other times they do not. Likewise, sometimes Asians and Pacific Islanders are grouped together, obscuring the differences between the groups.
Survey Says
Some of the data are culled from institutions; other statistics come from individuals. Some data are complete censuses of everyone involved; other data come from selected and varying samples. The Current Population Survey is based on a sample that excludes people residing in military barracks, prisons, and old-age homes. The American Community Survey is based on samples that include those populations.
The decennial census attempts to gather information about every individual, just as Ipeds attempts to gather data from every provider of postsecondary education.
Additionally, different populations have characteristics the data might not reflect. For example, many older age groups have higher attainment levels than do younger groups. That pattern might suggest a drop in education levels among younger populations. In reality, as the report explains, it shows that many people earn their degrees well after the traditional college age.
Keeping Track
Tracking the data over space and time raises all sorts of problems. For example, how many degrees a state awards doesn’t necessarily correspond to the distribution of degrees within that state, because graduates can and do move after college. As the report points out, “California ranks 21st in the percentage of adults between 25 and 44 years old with at least a bachelor’s degree (32 percent), but 46th in the number of bachelor’s degrees awarded in 2009-13 relative to the number of 18- to 24-year-olds (4 percent).”
Furthermore, tracking students over time can be tricky if they transfer. For privacy reasons, Ipeds data don’t track students who transfer away from the institutions where they started. By contrast, the National Student Clearinghouse does track most of the students who transfer, but their data cannot be broken out by institution.
There have been calls to resolve some of those problems with better data collection, such as rescinding the ban on a federal unit-record system, but until that happens scholars will just have to remember the data researcher’s motto: caveat emptor!
