Two years after many colleges resisted attempts by the Bush administration to impose more rigorous methods of student assessment, a leading coalition of private institutions is pressing its members to adopt a popular standardized achievement test—just as the four-year-old exam faces new questions over its reliability.
This week the 600-member Council of Independent Colleges will accept a $666,000 grant from the Teagle Foundation to expand a consortium through which the council’s members are promoting the use of the Collegiate Learning Assessment, which is designed to test skills such as critical thinking, problem solving, and writing. The test, which is also known as the CLA, is given to college students in their freshman and senior years.
“The CLA provides one of the first ‘value added’ measures that can reliably compare institutional contributions to student learning,” says a report by the council that was released with the announcement of the grant.
But two studies being presented this week at the annual conference of the Association for Institutional Research, in Seattle, challenge the CLA’s method of calculating the added value an institution gives its students. A fix to the way the test is administered would be simple but costly, the researchers say.
“After you do all the math” using the current methodology, said the author of one of the reports, Gary N. Larson, dean of information and technology at Wheaton College, in Illinois, “you can’t in fact make a statistically reliable inference as to whether Institution X is doing better than Institution Y.”
The Commission on the Future of Higher Education, established by Education Secretary Margaret Spellings, repeatedly praised the CLA in its final report released in September 2006. The commission said the CLA is one of “the most comprehensive national” tests that colleges can use to demonstrate the value they provide to students.
A Tool for Private Colleges
Richard H. Ekman, president of the Council of Independent Colleges, said that smaller private colleges—often with tight budgets and low public recognition—increasingly realize the benefit of a standard student examination that allows objective nationwide comparisons of their quality. “More and more of them are citing this in their self-studies” presented to accreditors, he said.
The Spellings commission may have recognized that value, he said, but many colleges resisted at the time, fearing that their acceptance of standardized tests could encourage a government-led educational process. “If anything,” he said, “Spellings made people want to dig in their heels and resist.”
The CLA was first offered in 2004 and now has about 235 participating colleges. The Council of Independent Colleges has its own consortium of 33 member institutions that work together to improve the way the test is administered and evaluated. The grant from the Teagle Foundation, established in 1944 by Standard Oil Company’s longtime president, Walter C. Teagle, would help expand the consortium to 47 members.
The two studies being presented this week—the one by Mr. Larson and another by Philip Garcia, director of analytic studies for the California State University system—both challenge the methods by which the CLA calculates the improvement in scores between a college’s freshmen and its seniors.
The 90-minute CLA is usually given only to a sample of members of any class—a sample chosen by the college to be representative of the class as a whole. Typically, a college tests sample groups from its first- and fourth-year classes at the same time, and compares the two classes’ scores to determine how much the fourth-year students have learned. Each class’s average score on another test for which the college has data, such as the SAT, is factored in to help the college judge whether a particular class actually made big advances in four years or performed unusually well to begin with.
It’s that last calculation—weighting CLA results according to performance on another test—that critics say is especially problematic. It involves so many variables that the CLA gives “insufficient reliability” to make meaningful comparisons between institutions, Mr. Larson said. Colleges could overcome that difficulty by giving the test to the same group of students at both their freshmen and senior years, he said, rather than attempting to make useful comparisons between different groups. Used that way, the CLA could be a “gold standard” for comparing colleges, he said.
Comparing 2 Classes
But for reasons of cost and complexity, about 85 percent of CLA users try to make comparisons between their freshmen and senior classes in the same year, Mr. Larson said. Cost factors include the need to recruit students to take the test, which typically requires cash payments of at least $25 per student because the test has no bearing on the student’s academic record. The cost and complexity would multiply if the college tried to ensure that a large sample of the same students took the test three years apart, Mr. Larson said.
The inability of colleges to compare the first- and fourth-year scores of the same set of students doesn’t deter institutions from citing the CLA when making claims about the educational value their institutions provide, he said.
“The reason that the CLA has gotten so much buzz is that it in theory is something that you could use at Harvard and you could use at the College of Du Page,” Mr. Larson said, citing the community college just outside Chicago, “and you could in fact control for enough factors to say, All things being as equal as they could ever be, Harvard is or isn’t doing a better job of educating students than the College of Du Page.”
Practical Applications
The independent-colleges group, in the report accompanying its announcement of the Teagle Foundation grant, cites examples of member institutions using data from the CLA to improve teaching and learning.
One institution—Cabrini College, in Pennsylvania—uses data from both the CLA and another standardized measure, the National Survey of Student Engagement, in which students describe their college experiences and estimate the number of papers they wrote and the number of times they made class presentations. Cabrini faculty members use the results of the two measures to revise their curriculum, the report said.
CLA test scores have no effect on students’ grades, and the results generally are not shared with students or the public. Yet some institutions, such as Barton College, in North Carolina, share the test’s results with prospective students “to foster a campus culture that insists on hard evidence,” Mr. Ekman said in his council’s report.
The council’s enthusiasm for the CLA is not universally shared. The National Association of Independent Colleges and Universities, which represents nearly 1,000 private institutions, has argued that its colleges have missions so varied and complex that their students’ progress cannot be accurately captured by any single standardized test. Naicu says, however, that it has no objection to colleges using the CLA voluntarily.
But Mr. Ekman and his group’s member colleges say that the CLA, along with other tests, helps verify the Council of Independent Colleges’ belief “that cognitive growth in small private colleges would be more than in other types of institutions,” he said.
Despite disagreeing with the council over the reliability of the testing methodology, researchers such as Mr. Larson find value in the CLA, especially insofar as it can help an institution improve its instruction.
The CLA’s creator, the nonprofit Council for Aid to Education, and its critics both have expressed a willingness to learn from each other. Mr. Larson said he expected feedback from Council for Aid to Education representatives at this week’s conference in Seattle that could affect his findings. And Richard J. Shavelson, a Stanford University education professor who helps the Council for Aid to Education develop the CLA, acknowledged that critiques such as the one offered by Mr. Garcia may have validity in some instances.
The Council of Independent Colleges “has been aware of various methodological challenges to the CLA, some from well-known scholars,” Harold V. Hartley III, the group’s senior vice president, wrote in an e-mail message. He said he believed the Council for Aid to Education “has adequately answered these challenges.”
