> Skip to content
FEATURED:
  • The Evolution of Race in Admissions
Sign In
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Find a Job
    • Post a Job
Sign In
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Find a Job
    • Post a Job
  • News
  • Advice
  • The Review
  • Data
  • Current Issue
  • Virtual Events
  • Store
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
    • Featured Products
    • Reports
    • Data
    • Collections
    • Back Issues
  • Jobs
    • Find a Job
    • Post a Job
    • Find a Job
    • Post a Job
Sign In
ADVERTISEMENT
News
  • Twitter
  • LinkedIn
  • Show more sharing options
Share
  • Twitter
  • LinkedIn
  • Facebook
  • Email
  • Copy Link URLCopied!
  • Print

International Group Announces Audit of University Rankings

By  Aisha Labi
October 10, 2010
At a conference in Berlin, higher-education officials are discussing the growth of university rankings, including a new effort to evaluate the ratings systems themselves.
Stefan Zeitz
At a conference in Berlin, higher-education officials are discussing the growth of university rankings, including a new effort to evaluate the ratings systems themselves.
Berlin

University rankings organizations could soon find themselves on the receiving end of the kinds of evaluations that have made them so newsworthy and influential. At a conference here last week for academics and institutions focused on rankings, the organizer unveiled a project that would effectively rank the rankers.

The IREG Observatory on Academic Ranking and Excellence, which was created a few years ago to develop quality-control mechanisms for rankings, announced that a volunteer trial audit of two or three rankings will soon be under way.

We’re sorry. Something went wrong.

We are unable to fully display the content of this page.

The most likely cause of this is a content blocker on your computer or network. Please make sure your computer, VPN, or network allows javascript and allows content to be delivered from c950.chronicle.com and chronicle.blueconic.net.

Once javascript and access to those URLs are allowed, please refresh this page. You may then be asked to log in, create an account if you don't already have one, or subscribe.

If you continue to experience issues, contact us at 202-466-1032 or help@chronicle.com

University rankings organizations could soon find themselves on the receiving end of the kinds of evaluations that have made them so newsworthy and influential. At a conference here last week for academics and institutions focused on rankings, the organizer unveiled a project that would effectively rank the rankers.

The IREG Observatory on Academic Ranking and Excellence, which was created a few years ago to develop quality-control mechanisms for rankings, announced that a volunteer trial audit of two or three rankings will soon be under way.

The International Ranking Expert Group, or IREG, first met in Washington in 2004 and two years later came up with a set of principles on the ranking of higher-education institutions.

There has “always been the idea that IREG could evolve into a quality assurance” body, said Gero Federkeil, who oversees the rankings of German institutions by the CHE Centre for Higher Education Development, which co-hosted last week’s conference.

As rankings proliferate around the world, they are increasingly having a direct impact on the decisions of students, academic staff, institutions, and policy makers, but each of those groups differs in its use of rankings and the sophistication it brings to evaluating them.

ADVERTISEMENT

Less informed groups, such as students, “don’t have a deep understanding of the limitations of rankings,” Mr. Federkeil said, and an audit would provide an assessment tool for users. The rankers themselves also need to be held accountable for possible deficits in their tabulations or methodological flaws, he said.

The audit project, which he is helping to manage, will be based closely on IREG’s principles, which emphasize clarity and openness in the purposes and goals of rankings, the design and weighting of indicators, the collection and processing of data, and the presentation of results.

“We all say that rankings should aim at delivering transparency about higher-education institutions, but we think there should be transparency about rankings too,” Mr. Federkeil said. The audit process could eventually give rise to an IREG quality label, which would amount to an identification of trustworthy rankings, thereby enhancing the credibility of rankings and improving their quality, Mr. Federkeil said.

At the Berlin meeting last week, Mr. Federkeil and Ying Cheng, of the Center for World-Class Universities at Shanghai Jiao Tong University, which produces the best-known and most influential global ranking of universities, outlined the proposed methodology and procedure for the audit. The IREG executive committee will nominate audit teams consisting of three to five people. The chair of each team must not have any formal affiliation with a ranking organization, and at least one member of the audit team must be a member of the IREG executive committee. Audits will be based on self-reported data as well as possible on-site visits, and each full audit is expected to take about five months to complete.

ADVERTISEMENT

Skepticism and Unease

Whether the audit will actually work remains to be seen. Many of the people who attended the meeting expressed deep skepticism and unease about how effectively a rigorous and independent audit procedure could be applied.

“In principle, I think it’s a good thing,” said Ben Sowter, head of the intelligence unit at QS, which produces the QS World University Rankings. But “there is a long way to go before this audit looks like the kind of measure it needs to be.”

Still, if it eventually evolves into a widely accepted and respected quality-assurance mechanism, the audit could become a useful tool “and enable us to counter some of the criticism that we receive,” he added.

Robert J. Morse, director of data research at U.S. News & World Report, said the magazine would most likely participate in the audit, but only “after we fully understand the processes and how it’s going to be scored.”

He agreed that it is important for rankers to be held to standards and to be transparent in their work.

ADVERTISEMENT

“We communicate very frequently with academics, but maybe we would need to also post in more detail about the mathematical processes and quality controls and other steps we take from the academic level, and that’s something that we would consider doing,” he said.

Mr. Morse and others also asked whether there would really be critical distance between the audit committee and IREG’s executive committee.

Ellen Hazelkorn, executive director of the Higher Education Policy Research Unit at the Dublin Institute of Technology and a well-known critic of the growing influence of rankings in shaping institutional and governmental policy, noted that rankings have become an intensely competitive business, and that any audit procedures would need to be clear and open enough to ensure that competitors were not pronouncing on one another’s work.

She also said that auditors should ensure that all constituencies are involved in the process, including academics, policy makers, and students.

“I think it could potentially go somewhere,” she said of the audit project. “I’m just not sure as to how it would work and who might subject themselves to it.”

ADVERTISEMENT

Some people invoked a comparison between the proposed audit and the accreditation process in the United States, in which universities participate voluntarily, but Ms. Hazelkorn emphasized that “universities have a compulsion to participate in accreditation” in order to secure eligibility for such financial benefits as Pell Grants, and that such an incentive is absent in when it comes to rankings.

But Nian Cai Liu, dean of the Graduate School of Education at Shanghai Jiao Tong University, which began producing the Academic Ranking of World Universities in 2003, applauded the effort.

“We need something, we need to start,” he said. “I think there will be more and more rankings, but there will be in a sense be more concentration of rankings,” he predicted. Those with an IREG approval label will grow in influence, but the rest will lose significance.

We welcome your thoughts and questions about this article. Please email the editors or submit a letter for publication.
International
ADVERTISEMENT
ADVERTISEMENT

Related Content

  • International Conference Takes a Fresh Look at University Rankings
  • Explore
    • Get Newsletters
    • Letters
    • Free Reports and Guides
    • Blogs
    • Virtual Events
    • Chronicle Store
    • Find a Job
    Explore
    • Get Newsletters
    • Letters
    • Free Reports and Guides
    • Blogs
    • Virtual Events
    • Chronicle Store
    • Find a Job
  • The Chronicle
    • About Us
    • DEI Commitment Statement
    • Write for Us
    • Talk to Us
    • Work at The Chronicle
    • User Agreement
    • Privacy Policy
    • California Privacy Policy
    • Site Map
    • Accessibility Statement
    The Chronicle
    • About Us
    • DEI Commitment Statement
    • Write for Us
    • Talk to Us
    • Work at The Chronicle
    • User Agreement
    • Privacy Policy
    • California Privacy Policy
    • Site Map
    • Accessibility Statement
  • Customer Assistance
    • Contact Us
    • Advertise With Us
    • Post a Job
    • Advertising Terms and Conditions
    • Reprints & Permissions
    • Do Not Sell My Personal Information
    Customer Assistance
    • Contact Us
    • Advertise With Us
    • Post a Job
    • Advertising Terms and Conditions
    • Reprints & Permissions
    • Do Not Sell My Personal Information
  • Subscribe
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    • Manage Newsletters
    • Manage Your Account
    Subscribe
    • Individual Subscriptions
    • Institutional Subscriptions
    • Subscription & Account FAQ
    • Manage Newsletters
    • Manage Your Account
1255 23rd Street, N.W. Washington, D.C. 20037
© 2023 The Chronicle of Higher Education
  • twitter
  • instagram
  • youtube
  • facebook
  • linkedin