Policy organizations, education researchers, and the news media too often abandon their responsibility for “honesty, accuracy, and due diligence” in reporting on education data, according to a blistering lecture delivered here this month by Clifford Adelman, a senior research analyst at the U.S. Department of Education.
Mr. Adelman, speaking at the annual meeting of the American Educational Research Association, argued that researchers, reporters, and legislators tend to lazily embrace obviously shoddy statistical claims about graduation rates, grade inflation, and student demographics.
“Descriptions of reality matter,” said the famously combative Mr. Adelman, who is one of the major architects of the National Education Longitudinal Study and other widely cited research projects produced by the Education Department. “We have an obligation to prevent statistical representations from turning into propaganda.”
The longest portion of Mr. Adelman’s jeremiad was devoted to recent discussions of the “leaky pipeline” — a term for the education system’s failure to move many students successfully from high school to college graduation.
The leaky pipeline is indeed a serious problem, Mr. Adelman said. But researchers and reporters have lately embraced faulty figures that make the problem seem much more dismal than it actually is, he argued.
In particular, Mr. Adelman complained, many people have echoed numbers that were presented in a “policy alert” published in April 2004 by the National Center for Public Policy and Higher Education.
That document — which was based on data assembled by the National Center for Higher Education Management Systems — suggested that of every 100 American ninth-grade students, only 68 would earn a high-school diploma within four years, only 40 would enter college, only 27 would return for a second year of college, and only 18 would earn either an associate degree within three years of matriculating or a bachelor’s degree within six years.
All of those figures were shoddily calculated, Mr. Adelman said. “By the time you get done with this study,” he said, “it does not pass the laugh test.”
‘Best Available Recent Data’
Patrick M. Callan, president of the National Center for Public Policy and Higher Education, said in an interview that the policy alert was based on the best available recent data.
“Most people agree, and the international comparisons agree, that this country has a very serious problem with the educational attainment of its young people,” he said. “And we don’t really have any data source for measuring that as adequately as we would like to.”
Mr. Adelman argued in particular that the 18-percent figure for degree completion was far too low. The National Education Longitudinal Study, which tracked students who were in the eighth grade in 1988, found that 35 percent of them had earned an associate or bachelor’s degree within 11 years.
Mr. Callan praised Mr. Adelman’s longitudinal study, known as NELS, but argued that it was too old to be a definitive guide for policy makers today. Immigration and other demographic changes, he said, mean that the ninth graders of today are significantly different from the ninth graders of 1989.
But Mr. Adelman also drew on more-recent data in his lecture: The Census Bureau’s Current Population Survey found that in 2003, 28.4 percent of Americans between the ages of 25 and 29 had earned at least a bachelor’s degree. (People are typically 15 years old when they complete the ninth grade, so 25-year-olds are a fair proxy for ninth graders 10 years later.) If you add people who earned associate degrees, Mr. Adelman said, the total is close to the longitudinal study’s figure of 35 percent.
The census data are more recent, Mr. Callan said in the interview, but not that much more recent than the data collected on Mr. Adelman’s longitudinal cohort. The 25-year-olds of 2003 would have graduated from high school in 1996, only four years after the contingent Mr. Adelman studied and referred to in his talk.
More broadly, Mr. Adelman said that Mr. Callan’s policy alert was riddled with apples-and-oranges comparisons, and he said that the 18-percent figure should have been immediately recognized as flawed. He asked the audience: “Does that number resonate with anything that you could possibly imagine in your common sense and your everyday experience? Does it sound right?”
Figure Reported Widely
And yet the figure has been repeated, he pointed out, in a White House press release, in Congressional testimony, on the Web sites of Achieve Inc. and the Education Trust (“both of which are organizations that I respect greatly,” Mr. Adelman said), and in the pages of The Chronicle.
The 18-percent figure was cited in the School & College supplement that The Chronicle published in March. The article in question was written by Peter Schmidt, a deputy editor at The Chronicle. In an interview, Mr. Schmidt said that he vetted the figure with a variety of prominent education researchers. “The pipeline figure that Mr. Adelman has challenged has been published and cited in a thick stack of reports by educational research groups,” he said.
Mr. Schmidt’s article did note two potential flaws in the policy group’s analysis. “Estimates of high-school-dropout rates are notoriously unreliable,” he wrote, “and the center’s analysis was unable to account for students who transfer from one high school or college to another.” Mr. Schmidt also wrote that the 18-percent figure referred to college students who graduated within three or six years “from the institutions where they started.”
In an e-mail message, Peter Ewell, a vice president at the National Center for Higher Education Management Systems who helped to develop the original data, said that the purpose of the study was to highlight state-level variation in graduation rates and college persistence. “This was the intent of the analysis,” he wrote, “not to provide an accurate national estimate of overall flow.”
In his lecture, Mr. Adelman said that Mr. Ewell’s original, state-level work was well-intentioned and valid, but that the National Center for Public Policy and Higher Education had vulgarized Mr. Ewell’s work when it released its policy alert. More grievously, Mr. Adelman said, the public-policy center failed to object when the White House and others started to cite the faulty national-level figures.
“Why would anybody want to make this worst-case story for U.S. education?” Mr. Adelman asked. He concluded that education researchers and policy organizations had sat silently while the figures were circulated because a perception of crisis might mean that they would receive more resources. The tacit belief is that “we can work harder to fix it, and we get more attention, and we get more business,” he said.
Questioning Grade Inflation
Mr. Adelman also criticized policy and media organizations’ treatment of grade inflation. Almost all existing studies of the question, he said, are so methodologically shoddy that they cannot shed light on whether grade inflation is a problem or not.
And he challenged the widely held belief that community-college students are much older than students at four-year institutions. The often-cited figure that the average age of community-college students is 29 is no longer correct, if it ever was, Mr. Adelman said. (“Maybe it’s true,” he said, “if you include Elderhostels and noncredit students.”)
If anything, Mr. Adelman said, students at community colleges are younger now than a decade ago. In 1991, 32 percent of students at two-year public institutions were younger than 22. By 2001, that figure had risen to 42 percent, he said, citing figures from the Education Department’s National Center for Education Statistics. During that same period, the proportion of students 30 and older fell from 34 percent to 31 percent.
“What difference does this make?” Mr. Adelman asked. “Well, if you’re a community-college administrator, and you assume with every breath that your mean student age is 29, the way you program your services is pushed toward that student.” He cited unpublished data that suggest that between 1993 and 2003, community colleges added certificate programs aimed at older students and reduced the number of majors and programs aimed at associate-degree-oriented students, who tend to be younger.
Mr. Adelman repeatedly emphasized that he was speaking only for himself, and that his lecture had not been reviewed or endorsed by the Education Department. He also mentioned in the lecture and a subsequent question-and-answer session that he will be leaving the department before the end of 2006.
http://chronicle.com Section: Government & Politics Volume 52, Issue 33, Page A31
