From celebration to criticism: How college rankings really work and can schools game the system?
Graduation. Credit: Getty Images
From clapping and cheering to publishing praises from politicians, Florida got a lot of publicity this week about some big boosts in the popular college rankings published by U.S. News & World Report.
But beyond the good news, particularly about Florida State’s big hike in the rankings, little was said about how the rankings are calculated and how the methodology changes in most years – for better or worse.
U.S. News uses various factors to rank schools, including college admissions exam scores, freshman retention rates and graduation rates.
But not everyone is happy with the measures.
Forbes contributor Michael Nietzel – a president emeritus of Missouri State University – wrote a column this week titled: “U.S. News Releases Its Annual College Rankings. Here’s What’s Wrong With Them.”
Nietzel expounded on the factors used to compile the college rankings:
“Though U.S. News has revised its methodology over time and continues to tinker with the weights assigned to various measures, fundamental problems remain with the approach, which is based on 15 measures of ‘academic quality,’” Nietzel writes.
“It is too easy for schools to game the system and falsify data. Several of the U.S. News’s measures have become proxies for institutional wealth, and their relevance to academic quality is questionable. Nearly 50% of the weighted indicators are problematic on at least one of these grounds.”
(Just to let readers know: Forbes also publishes college rankings, as do other publications.)
In the U.S. News analysis, academic reputation counts for 20 percent in the rankings analysis. That’s “an entirely subjective measure at best and a bogus one at worst,” Nietzel says.
The academic reputation category is based on peer surveys by top college officials who rate the quality of academic programs at their school and similar schools.
“How does any president or provost even pretend to know the academic quality of so many institutions, let alone make quantitative distinctions among them?” Nietzel asks. “No wonder some college presidents have admitted they delegate a staffer in their office to fill out the survey. Or worse, they confess that they downgrade the ratings of peer institutions to make their own college look better by comparison.”
U.S. News officials described the surveys as subjective, but also important.
Robert Morse, the chief data strategist for U.S. News, told the Florida Phoenix that the weight of the academic reputation category has gone down – from 22.5 percent in 2018, to 20 percent now. Morse answered questions by email for the Phoenix.
The weight of ACT and SAT scores – crucial exams used in part to admit students to selective and other colleges – also has gone down, from 8.125 percent in 2018 to 7.75 percent in 2020.
Still the percentages raise questions about why college admissions scores carry far less weight than peer surveys in the rankings analysis.
As to the reductions in the academic reputation and ACT/SAT categories, Morse said U.S. News wanted to reduce the weight of “input measures” in order to give more weight to “outcome measures.”
Input measures include characteristics of students and faculty. Output measures relate to results, such as graduation rates, freshman retention rates and what’s called “social mobility” – which includes graduation rates for low-income students or students who are first in their families to attend college.
But Forbes contributor Nietzel wasn’t a fan about the weights for college admissions scores.
“One would think U.S. News might want to forget about standardized tests as a factor, given numerous revelations about the degree to which they are subject to cheating, a problem made so painfully apparent in the recent college admissions scandal,” Nietzel writes.
“But that objection misses the bigger point: colleges use the SAT and ACT to exclude applicants so they can score higher on ‘student excellence,’ turning the rejection of students into an institutional asset. And they are not above using various tricks – such as delaying the admission of low-scoring students to the spring semester so their test scores will not be included in the fall semester’s submitted data – to make entering students’ scores look better than they actually are.”
In a significant change for the 2020 rankings, some schools were switched to different categories, such as national universities, regional colleges and universities and national liberal arts colleges. The categories allow for comparisons with schools that have similar missions.
But the changes – 13 percent of schools – mean those schools can’t be compared to rankings from the prior year, according to U.S. News.
“It would not be fair to compare year-over-year rankings because these schools are effectively in different playing fields,” Morse said in an email to the Phoenix.
And “even schools that remained in the same category may have been impacted by the reshuffling secondhand,” according to a “frequently asked questions” U.S. News document provided to the media. In some cases, regional universities were newly ranked as national universities.
“The swelling of this (national university) category caused some schools to drop in the rankings simply because of competing against a larger field.” And regional schools “may have risen in rank because some of their peers left their categories.”
Overall, “some year-to-year volatility is to be expected,” according to the document.
In the case of Florida State, the change was dramatic.
In the category of national universities – which have a large range of undergraduate, master’s and doctoral degrees and a strong focus on research – Florida State moved up to a rank of 57, compared to 70 the year before.
For national universities that are public, not private, Florida State moved up to a rank of 18, compared to 26 the year before.
Those numbers got accolades from Gov. Ron DeSantis, other politicians and top academics.
The Phoenix asked FSU officials if methodology changes could be a reason for the huge boosts in rankings.
“There may be subtle changes to this year’s ranking calculations as a result of the changed methodology, spokesman Dennis Schnittker told the Phoenix in an email. “But any effect on the rankings would have been felt by all public schools equally.”
He added, “It is undeniable that FSU’s gain in the rankings was the product of the university’s improvement in areas such as student graduation and retention rates, peer assessment, financial resources and alumni giving.”
You can look at the U.S. News 2020 college rankings at: https://www.usnews.com/best-colleges
Our stories may be republished online or in print under Creative Commons license CC BY-NC-ND 4.0. We ask that you edit only for style or to shorten, provide proper attribution and link to our web site.