After pushing back against popular hospital ratings systems for years, Rush University Medical Center last month released its own composite ranking of 70 U.S. hospitals.

Just updated: Get 1-page cheat sheets on how hospital quality ratings programs work

Rush's brushes with ratings in the past

Rush has taken a critical eye to hospital ratings since 2015 when U.S. News & World Report erroneously gave it the lowest rating for patient safety on U.S. News' Best Hospitals list. Up until that point, Chicago-based Rush had earned high patient safety scores.

Surprised by the U.S. News rating, Rush investigated the methodology and found flaws in the data had led U.S. News to over report patient safety events. Rush's findings ultimately led U.S. News to boost Rush's overall score and change how it conducts its ratings.

But Rush's skepticism of hospital ratings didn't end there. Rush in May 2018 pushed back against upcoming changes to CMS' Star Ratings. Others, including U.S. senators, the American Hospital Association, and Modern Healthcare have raised similar concerns about CMS' Star Ratings in the past.

After learning its rating would fall from five stars to three stars under an updated methodology, Rush undertook an internal analysis and found that the statistical model CMS used was dynamically changing the weighting of certain measures in every release—meaning that one specific performance measure could play an outsized role in determining a hospital's final rating.

An updated analysis from Rush dug deep into the most recent February 2019 ratings and again found what they consider to be major flaws in the calculation methodology, which Rush stated "inadvertently penalizes large hospitals, academic medical centers" and those providing "heroic care." 

Since then, Rush has worked with the federal government to improve the methodology.

Rush's hospital rankings

Rush took the next step in its quest for better hospital ratings last month—by publishing its own ranking of 70 hospitals across the United States based on the composite score of existing rating systems. The ratings were published in the peer-reviewed American Journal of Medical Quality.

For the ratings, Rush compared and combined U.S. News' Best Hospitals, Vizient Quality and Accountability Study, CMS Star Rating, Leapfrog Hospital Safety Grade, and the Truven Top 100 Hospitals ratings to generate scores for each hospital. According to Rush, "combining the results of these ranking systems into a composite, the measurement of hospital quality can be simplified."

Rush then assigned each hospital a number from one to 10, with one being the highest.

Under Rush's composite ranking system, only two hospitals received a grade of one: The Mayo Clinic in Rochester, Minnesota, and the Mayo Clinic in Arizona.

Rush was named the top Chicago hospital on the list, with a grade of four. The only other Illinois hospitals on the list were Northwestern Memorial Hospital and Loyola University Medical Center, which received scores of five and seven, respectively. Rush CEO Omar Lateef said even if Rush wasn't highest ranked, "we would have published the paper."

Discussion

The paper is Rush's "attempt to make [hospital rankings] more understandable for people," according to Bala Hota, Rush's chief analytics officer and lead author of the paper. "We want there to be measures of quality, and we just don’t feel like we’ve gotten there on measures that are providing the most meaning and use for patients."

Ben Harder, chief of health analysis for U.S. News, said Rush's study was "interesting," adding that "trying to sort out which ratings are good for what is an important thing for scientists to look at."

But, Harder warned that rating systems will always deliver differing results, since many are judging hospitals on different strengths. For instance, Leapfrog analyzes patient safety, while U.S. News & World Report grades on specific clinical services.

"They may all be right," Harder said. "Depending on what lens you're looking at a hospital through, they may be good at one thing and not another" (Schencker, Chicago Tribune, 7/9; Hota et al., American Journal of Medical Quality, 6/28).