Hospital ratings get lots of buzz in the news, but do you know what factors actually determine how your hospital ranks?

We dove into four of the most popular lists—from CMS, Leapfrog, U.S. News, and Watson Health—to find out how they work and why they're so controversial.

Access the quick guides to understand hospital rating metrics and methodologies

CMS Hospital Compare Star Ratings

CMS unveiled its inaugural Hospital Compare Star Ratings in July 2016 to help Medicare patients select a hospital based on quality performance.

The ratings cover seven categories of metrics: mortality, readmission, safety of care, patient experience, effectiveness of care, timeliness of care, and efficient use of imaging. Mortality, readmission, safety of care, and patient experience are each worth 22% of a hospital's total score, while the other three categories are worth 4% each.

CMS this year updated its methodology, causing a greater share of hospitals to fall at the ends of the bell curve and receive a 5- or 1-star ratings, Modern Healthcare reports.

In the latest list, released in December 2017, CMS reviewed 3,692 hospitals and awarded:

• 337 (9%) five stars;
• 1,155 (31%) four stars;
• 1,187 (32%) three stars;
• 753 (20%) two stars; and
• 260 (7%) one star.

Leapfrog Group Hospital Safety Grade

The Leapfrog Group's Hospital Safety Grade examines how hospitals perform on 12 process/structural measures, such as ICU physician staffing, and 15 outcome measures, such as hospital-acquired infections. Each domain makes up 50% of a hospital grade.

Leapfrog, a private nonprofit, developed the ratings to encourage patients to assess hospital safety and encourage patients to use that information in hospital selection.

In the latest list, released in October 2017, Leapfrog reviewed 2,632 hospitals and awarded:

• 832 (32%) As;
• 662 (25%) Bs;
• 964 (37%) Cs;
• 159 (6%) Ds; and
• 15 (less than 1%) Fs.

US News & World Report Best Hospitals

The annual U.S. News & World Report Best Hospitals list focuses on specialty care performance, ranking eligible hospitals' performance on up to 16 specialties.

U.S. News assesses 12 specialties based on four categories of metrics: outcome, worth 37.5% of a total score; structure, worth 30%; process, which includes the results of a reputation survey, worth 27.5%; and patient safety, worth 5%. U.S. News ranks the four other specialties based solely on reputation.

In the latest list, 152 hospitals scored highly enough to be included in the rankings. In addition, U.S. News recognizes the top 20 performers across the specialties on its Honor Roll.

Watson Health 100 Top Hospitals

Watson Health's Top 100 Hospitals Rankings, previously known as the Truven Health Top 100 Hospitals, examine hospital quality and operational efficiency.

For the rankings, Watson Health examines hospitals within several groups: major teaching, teaching, large community, medium community, and small community hospitals. The rankings include about 15 to 25 hospitals in each category.

The Top Hospitals ranking assess measures in five weighted domains: inpatient/clinical outcomes, extended outcomes, operational efficiency, financial health, and patient experience.

For all categories except small hospitals, inpatient/clinical outcomes are worth 30% of the score, extended outcomes are worth 20%, operational efficiency is worth 30%, financial health is worth 10%, and patient experience is worth 10%.

For small hospitals, inpatient/clinical outcomes are worth 37.5% of the score, while extended outcomes are worth 12.5%. The other three domains are the same for small hospitals as they are for the other four hospital categories. Mean 30-day risk-adjusted mortality rate isn't factored into the extended outcomes calculation for small hospitals, but it is for larger hospitals.

In addition, Watson Health aggregates information on individual hospitals into system-wide assessments to produce the Watson Health Top 15 Top Health Systems.

Why do hospitals rate so differently on different reports?

While the different ratings systems might seem similar on the surface, a 2015 paper that looked at four ratings systems—Leapfrog, U.S. News, Consumer Reports, and HealthGrades—found none of the 83 hospitals that all four raters reviewed were considered a high performer by multiple raters.

Further, the authors also drew attention to differences in risk-adjustment methods, finding, for instance, that 43% of hospitals that had below-average mortality by one risk-adjustment method had above-average mortality by another.

"These contradictions have created challenges for stakeholders concerned with hospital quality," the authors wrote. "For patients, differences across hospital ratings add complexity to ascertaining a hospital's actual quality. For payers, conflicting ratings make it difficult for them to recognize and reward hospitals for high quality. For hospital leadership, differences across rating systems complicate decisions regarding the focus of their improvement efforts."

Further, teaching hospitals and safety-net hospitals take a higher share of sick and low-income patients and tend to perform relatively poorly in CMS Star ratings, which penalize for higher rates of readmission, Ross reports.

7 imperatives to transform your quality strategy

J.B. Silvers, a professor of health care finance at Case Western Reserve University, told STAT News in 2016, "Big teaching hospitals are taking care of the most complex cases. Hardly any of them are in the five-star category."

But even with the confusion, some say consumers can still use ratings to make informed decisions.

In an interview with WBUR, Ashish Jha, a professor at the Harvard T.H. Chan School of Public Health, said, "For most people, the number one [concern] is having a good outcome. If you're really sick, you don't wanna die. You wanna be treated with dignity and respect. You can go track that kind of data down for hospitals—that's out there, it's available."

Ratings matter—a lot

For hospitals, a lot can ride on a particular rating.

Saint Anthony Hospital in Chicago filed a lawsuit against the Leapfrog Group last fall, alleging the organization lowered the hospital's grade based on data Leapfrog knew to be inaccurate. The hospital in the suit said the lower score could hurt its reputation in the community, threaten its bottom line, and compromise potential partnerships.

Leapfrog responded to the lawsuit by saying the hospital previously had affirmed the data it submitted was correct and that the hospital failed to raise the issue during a post-submission review period, Healthcare Informatics reports. Leapfrog also had said it doubts that the data in question is solely responsible for the lower grade.

However, Leapfrog agreed not to publish a Safety Grade for Saint Anthony last fall and removed the disputed data, according to Becker's Hospital Review.

Jha, who was on the committee that helped set standards when Leapfrog was established, told Kaiser Health News, "In highly competitive markets, hospitals are likely to see poor grades as a challenge, and I think many will be tempted to sue the rating agencies."

Jha added that he views hospitals reacting to the data as a positive. "If they're going to use that as motivation to get better, that's perfect. As a patient, you don't care why a hospital is investing in safety, you just care that they are."

Get the cheat sheets: How hospital quality ratings programs work

Download our one page sheets for summaries on the methodology and metric categories used in five hospital quality rating programs:

• US News and World Report Best Hospital Rankings
• CMS Star Ratings
• Leapfrog Group Hospital Safety Grade
• Watson Health 100 Top Hospitals

• Download the Cheat Sheets