Assessment Denied: the National Research Council's Sins of Omission

Jon Kraus

August 31, 2009

For almost three years, research universities and doctoral programs across the country have pulled out all the stops to participate in the Assessment of Research-Doctorate Programs in the United States, conducted by the National Research Council of the National Academies. With its much-delayed release due this fall, the report affects the reputations and relative standings of universities across the nation, and stands as the authoritative source on the quality of American doctoral programs.

The NRC has apparently gone to great lengths in its effort to assess doctoral programs. It surveyed institutions, programs, faculty members, and doctoral students on a wide range of issues, including faculty diversity, productivity, research focus, educational background, and advising load; doctoral-student admission criteria, financial support, health benefits, completion rates by race/ethnicity and gender, time to degree, and training and mentoring received; and faculty perceptions of the quality of doctoral programs at other universities in their disciplinary fields. The council even included questions regarding postdoctoral positions and the way they are defined in local settings.

In the spring of 2008, the NRC reported an impressive array of response rates to their multipronged assessment of doctoral programs: a 100-percent response rate by universities to the 29-page institutional survey, a 97-percent response rate by doctoral programs to the 34-page program survey, an 86-percent response rate among faculty members to the 19-page faculty survey, and a 72-percent response rate among doctoral students to the 18-page doctoral-student questionnaire. At our institution alone, the University of California at Berkeley, we estimate that participation in the study resulted collectively in the filling out of roughly 30,000 pages of questionnaires by faculty and administrative-staff members and doctoral students. Such lofty response rates are all the more remarkable in light of the fact that participating universities bore the significant cost of responding to the exhaustive data-collection phase, and actually paid the NRC to help support the effort.

For such thoroughness and high rates of participation among subject institutions and programs, the National Research Council is to be lauded, particularly as it offers an alternative to the oft-criticized U.S. News & World Report "Best Graduate Schools" rankings.

But something is missing from this influential report: a diverse range of fields that accurately reflects the breadth of academe and its research-doctorate recipients. Our analysis of Ph.D. fields that will be included and excluded from the NRC's assessment, viewed in light of demographic patterns of recent degrees granted, suggests that the selection deemed worthy of assessment may suffer from a form of implicit bias.

Excluded fields appear to have granted more than a quarter of all research doctorates from 2001 to 2005 to U.S. citizens and permanent residents—27 percent, according to our calculations, based on data from the Survey of Earned Doctorates, conducted for the National Science Foundation and other government agencies. But African-American, American Indian, and Mexican-American people, as well as women, are disproportionately likelier to have received Ph.D.'s in fields that are not included in the NRC's assessment, relative to white men and Asian-American people. For example, 54 percent of African-American women, 49 percent of American Indian women, 43 percent of Mexican-American women, 42 percent of African-American men, and 34 percent of women in general were awarded doctorates in fields not currently assessed by the NRC. In contrast, only 10 percent of Asian/Pacific Islander men, 18 percent of Asian/Pacific Islander women, 19 percent of white men, and 19 percent of men in general received Ph.D.'s in fields excluded from the assessment. The disparity is not as great among all groups, with Mexican-American men, American Indian men, and Puerto Rican men showing rates of degree receipt in non-assessed fields (31, 28, and 22 percent, respectively) that are closer to those of the larger population.

What accounts, then, for this seemingly odd pattern of doctorates awarded to people in various demographic groups by fields included and excluded in the current NRC assessment? The exempted fields—including education, business, social work, psychological counseling, library science, home economics, and subfields of the health and agricultural sciences—share a professional and applied orientation as opposed to a basic research one. Many also emphasize public scholarship, a tradition that favors the interweaving of intellectual pursuit with social improvement.

The exclusion of fields favoring public scholarship helps to explain the large demographic disparities among recipients of Ph.D.'s from fields included in and excluded from the assessment. A growing body of literature supports the premise that underrepresented minority and female scholars are disproportionately likely to channel their efforts into academic endeavors that are configured to directly benefit the larger society and local communities. For example, Anthony Lising Antonio, Helen S. Astin, and Christine M. Cress (2000) found "high levels of commitment to and engagement in community service" to be most common among "women faculty, faculty at the lower ranks, and faculty of color." So, too, they observed that "faculty trained in social work, ethnic studies, women's studies, education, and health sciences—fields that focus on improving people and communities—exhibit the highest levels of personal commitment to service." They worried, however, that this predisposition toward altruism among these groups might work to their disadvantage because of the prevailing academic structure: "Why are the faculty who have the least to gain professionally from contributing to the civic education of students participating the most often in these important but largely undervalued activities?"

Unfortunately, however, it appears that the call to broaden the definition of academic scholarship has gone largely unheard in the latest round of the NRC's doctoral-program assessment. Although the inclination to engage in public scholarship should be rewarded—or, at least, not marginalized—the NRC has provided no compelling rationale for disproportionately excluding fields of this type from one of academe's most widely accepted assessments of quality and opportunities to achieve professional recognition. For example, when asked by a university coordinator on an NRC-sponsored e-mail list why, "other than reasons of continuity with past assessments," education and business were excluded from the current assessment, Charlotte V. Kuh, director of the study, wrote: "Education and business are both large and applied and closely associated with professional programs. On practical grounds, including them would involve a major increase in size (and expense) of the study. On conceptual grounds, we seriously considered including education research, but it turned out that for many schools of education it was difficult to separate education research from specializations in applied areas of school administration, curriculum, etc. Schools of business did not want to be included in the study. ... [C]ontinuity with the past is not a trivial argument, although we are including a number of new fields this time around."

Thus cost, the applied nature of excluded disciplines, and continuity of approach seem to account for the exclusion of certain fields from the current study. Those seem poor grounds for exclusion, particularly given the fact that universities end up shouldering a substantial portion of the expense and have a clear and compelling interest in diversifying the academic landscape and demonstrating larger social relevance.

To be sure, the National Research Council is to be commended for guarding against commercialization of its doctoral-program assessments and for its commitment to detailed quality assessment, methodology, and disciplinary taxonomy. And it is moving in the right direction by including nursing, public health, communications, and emerging, cutting-edge fields such as biotechnology, nanoscience, and race, ethnicity, and postcolonial studies. But the exclusion of other fields that produce large numbers of research doctorates seems insular and retrograde. We urge the NRC and others to consider those issues and to design future assessments that are truly comprehensive and reflective of the diversity of academe.

Marc Goulden is director of data initiatives for academic affairs; Angelica Stacy is a professor of chemistry and associate vice provost for faculty equity; and Mary Ann Mason is a professor of law and social welfare and co-director of the Center on Health, Economic and Family Security at the University of California at Berkeley.