This site is only available on a desktop computer.

New Hospital Compare Star Rating Proposals Fail to Fix Flaws, Add Complication

Allowing customized star ratings based on personal preferences would make the system more confusing for consumers, not less.

CMS is considering several updates to Hospital Compare’s overall hospital star ratings system, including allowing consumers to customize hospitals’ star ratings based on their personal preferences. This approach would in many cases shift hospital scores higher or lower than the CMS-issued ratings, which are already flawed due to the existing ratings methodology.

HANYS supports making meaningful, accurate hospital quality and safety information publicly available. However, CMS’ star ratings, in their current form and with the proposed changes, do not accomplish this goal. Until they do, HANYS believes these ratings should be removed from public view.

More information on Hospital Compare’s star ratings system methodology and the consumer-customization proposal are below for your reference.

CMS’ star rating program

Since 2016, CMS’ Hospital Compare website has publicly reported a star rating between 1 and 5 for nearly every hospital in the nation. As you will discover below, the star ratings have a “normal” distribution on the 1-5 scale, but there is substantial variation within and among states.

New York’s ratings have been persistently skewed low, due in part to CMS’ measure selection and methodology approaches.

How are stars assigned?

Unlike ratings for other products and services that are based on direct consumer feedback (e.g., Yelp, Google, etc.), CMS’ hospital star ratings are based on healthcare quality measures, complex statistical formulas, and policy decisions made by CMS. The current approach combines up to 57 measures across seven quality domains (measure groups) into the rating. These measures and methods are flawed.

The data below shows that not all types of hospitals report the same set of quality measures, due to variation in services offered, geographic location and other factors. While the existing statistical formulas for calculating the star ratings are intended to account for these variations, they are imperfect, leading to potentially inappropriate variation across hospital types.

Allowing consumers to customize ratings based on their preferences can compound this underlying variation.

Hospital Measure Reporting Examples:

All Star Measures Critical Access Hospitals Large Academic Hospitals Specialty Hospitals Outpatient/ED-focused Hospitals

The current hospital star ratings combine up to 57 measures across seven quality domains that encompass a wide variety of healthcare services. Some measures, such as mortality and readmissions, assess hospital inpatient care, while other measures, such as timeliness of care, assess hospital outpatient services. Additionally, measures vary in the patient populations represented. Some assess care provided to Medicare patients (age 65+); others assess care provided to all patients.

Critical Access Hospitals are small hospitals located in rural/more remote areas. Due to their low patient volumes, CAHs often restrict public reporting of their quality data as meaningful conclusions are not possible. If CAH ratings are made public, they are typically assessed on a smaller subset of measures that excludes some patient satisfaction, safety and readmissions measures, further lessening their value in the star ratings.

Large academic medical centers not only train the nation’s physicians and perform medical research, they also treat a high proportion of patients with socio-demographic risk factors and are providers of high-acuity specialty services (e.g., trauma and burn services). Because of the range of services offered, these hospitals tend to be evaluated across the majority of star measures. Factoring in the number of measures assessed and their challenging patient population, these hospitals tend to be disadvantaged under the star rating system.

Hospitals that specialize in providing certain types of services (e.g., orthopedic services) can be evaluated on fewer measures since the minimum patient thresholds for certain measures may not be met. While specialty hospitals are generally recognized for providing high-quality care, being scored on a smaller subset of measures tends to be an advantage under CMS’ hospital star rating system.

With patients shifting from inpatient care to outpatient care, some hospitals have strategically focused on providing outpatient care and in turn have low inpatient volumes. Often, these hospitals will not meet the minimum patient thresholds and therefore are assessed on a smaller subset of measures that exclude some patient satisfaction, safety and readmissions measures.

Domain: foo
Measure: foo
Description: foo
Measure Loading: foo

What if consumers could customize the ratings?

CMS has proposed to allow consumers to create custom star ratings using a three-point scale (low importance, neutral, high importance). This will cause major ratings swings, making the rating system more confusing for consumers, not less.

This proposed customization assumes that patients and families possess the clinical and statistical knowledge and the time needed to decode the star ratings and understand their relevance.

Below, we explore how this approach could swing the star ratings by showing current results and a modeling of how the ratings would look with the proposed consumer customization.

United States
New York State
5 hospitals change from 1 star to 2 star

Consumer customization: Aligning with the current ratings

To maintain alignment with the current ratings, under this scenario consumers would select “very important” for patient experience, safety, readmissions and mortality domains. The other three domains would be scored at “low importance.”

Even with this alignment, you can begin to see the star ratings shift between hospitals nationally. While most hospitals that shift as a result of this consumer preference see a one-star improvement to their rating, some hospitals would drop in the ratings. New York hospitals would see similar shifts, but overall, they would still skew toward the lower side of the ratings.

Low Importance Neutral High Importance
  • Timeliness of Care
  • Medical Imaging Efficiency
  • Effectiveness of Care
  • Readmissions
  • Mortality
  • Safety of Care
  • Patient Experience

Consumer customization: Focusing on mortality, imaging efficiency and care effectiveness measures

If a consumer placed a high importance on the mortality, efficiency and effectiveness domains by rating them as “very important,” while scoring the other four domains at “low importance,” the ratings would shift substantially.

Under this scenario, you can see bands of hospitals moving both up and down in rating nationally, with some hospitals jumping from 5 stars to 3, and vice versa. A 2-star hospital even jumps to a 5-star rating. New York would see similar shifts, while continuing to skew toward the lower side of the ratings. This movement begins to show the risks and potential confusion associated with the consumer-customization proposal.

Low Importance Neutral High Importance
  • Readmissions
  • Safety of Care
  • Patient Experience
  • Timeliness of Care
  • Medical Imaging Efficiency
  • Effectiveness of Care
  • Mortality

Consumer customization: Eliminating safety measures

Allowing consumers to eliminate a quality domain that feeds into the star ratings would dramatically shake up the ratings when compared to the current system. If consumers could turn off the safety of care domain, we observe a substantial shift in ratings nationally and even show a 1-star hospital jump to a 5-star rating. Under this scenario, New York would begin to see a more “normal” distribution in its ratings.

This massive shift in ratings clearly shows the risks and potential confusion associated with the consumer-customization proposal.

CMS did not specify that consumers would be allowed to eliminate quality domains altogether. However, HANYS has long advocated for the elimination of the patient safety domain due to its bias toward Patient Safety Indicator-90, a single composite patient safety measure that is widely viewed as flawed. As shown by the line chart below, the difference in median performance by measure in the patient safety domain is meaningful for PSI-90 but is negligible for important measures such as CAUTI, CLABSI and other hospital-acquired infection measures.

Median Z-Score by Measure (Safety Domain)

  • 1 Star Hospitals
  • 2 Star Hospitals
  • 3 Star Hospitals
  • 4 Star Hospitals
  • 5 Star Hospitals
Low Importance Neutral High Importance
  • Timeliness of Care
  • Medical Imaging Efficiency
  • Effectiveness of Care
  • Readmissions
  • Safety of Care
  • Patient Experience
  • Mortality

Click here to access HANYS' full star rating comment letter addressing the most significant methodology changes proposed by CMS.

© 2019 Healthcare Association of New York State. One Empire Drive, Rensselaer, NY 12144

All Rights Reserved. By using this website, you agree to HANYS Terms of Use.