US News Hospital Rankings Reliability for Urology Patients Questioned - Kathy Huen
October 18, 2024
Ruchika Talwar hosts Kathy Huen to discuss her research on the accuracy of US News and World Report hospital rankings for urology. Dr. Huen discusses her study comparing the U.S. News methodology to electronic medical record data, revealing significant discrepancies in patient attribution and outcome measures. The research finds that U.S. News rankings often misidentify patients as being under urological care and inaccurately report mortality and discharge-to-home rates. Dr. Huen suggests several strategies to improve the ranking system, including expanding data sources beyond Medicare, using index procedures, and removing numerical rankings. They discuss the implications of these findings for patient care decisions and the challenges of providing accurate quality metrics. The conversation highlights the need for more reliable methods to identify centers of excellence in urology and the potential role of urologists in developing alternative ranking systems that better reflect the quality of care provided.
Biographies:
Kathy Huen, MD, MPH, Assistant Clinical Professor of Urology, UCLA Health, Los Angeles, CA
Ruchika Talwar, MD, Assistant Professor of Urology, Urologic Oncologist, and Associate Medical Director in Population Health, Vanderbilt University Medical Center, Nashville, TN
Biographies:
Kathy Huen, MD, MPH, Assistant Clinical Professor of Urology, UCLA Health, Los Angeles, CA
Ruchika Talwar, MD, Assistant Professor of Urology, Urologic Oncologist, and Associate Medical Director in Population Health, Vanderbilt University Medical Center, Nashville, TN
Read the Full Video Transcript
Ruchika Talwar: Hi, everyone. Welcome back to UroToday's Health Policy Center of Excellence. As always, my name is Ruchika Talwar, and today I'm excited to be joined by Dr. Kathy Huen from UCLA. She's going to be sharing some recent work that she did exploring the accuracy of US News and World Report rankings. Thank you, Dr. Huen, for being here with us.
Kathy Huen: Thank you so much for the invitation, and we're very excited to share our results. So thank you again, Dr. Talwar, for the invitation. I'm a pediatric urologist by training, but I serve as a quality officer for the Department of Urology and also a senior quality officer at UCLA Health. So the perceived importance of US News and World Report hospital rankings is all around us. I snapped this photo as I was walking to the main hospital, and this is a very widespread use of US News and World Report as a marketing tool. So that, as well as the withdrawal from US News and World Report of really dozens of law and medical schools, has prompted scrutiny of its methodology in recent years.
So as the department quality officer, hospital administrators had come to us regarding fluctuations regarding the department's rankings over the years, which really personally prompted a deep dive into the US News and World Report and specialty rankings. And what I found was kind of unsettling, that the patient population within our internal records was very discordant with who US News and World Report was identifying. And that motivated my team to really assess the accuracy of US News and World Report mortality and discharge-to-home measures for patients being treated for urologic conditions.
So US News and World Report, as a background, uses Medicare fee-for-service inpatient encounters that fall within certain Medical Severity Diagnosis-Related Groups, or MSDRGs, as we're more commonly understanding them. And because there can be significant errors in patient attribution when using these claim-based methods, we hypothesize that an approach based on the electronic medical record would be more useful as a reference standard to basically assess the accuracy of US News and World Report measures. So through our electronic medical record, which we use EPIC, we replicated US News and World Report's method of assigning patients to a hospital's urology service. And then from there we queried that same database for patients who were admitted or discharged under the urology service, who here we're calling the primary urology group.
So why mortality and why discharge to home? And this is because the outcomes component typically constitutes the most heavily weighted part of the US News and World Report rankings. It counts for about 40% of the rankings and other components are shown here as well. And this is the latest iteration because every year US News and World Report puts out a new methodology report, and the outcomes component continues to contribute the most. From a mortality standpoint, we found that there were 13 patient deaths that were identified in the US News and World Report group over the span of four years. At chart review, about four of these 13 patient deaths were primarily associated with urology procedure or hospitalization, which meant that the remaining nine, which is about 70%, were not. And in comparison, all three deaths that were identified in the primary urology group were associated with urology procedure hospitalization.
Some patients in the US News and World Report group had really no urology touch or consultation. Some had procedures that were unrelated to the cause of death, and a concrete example would be a ureteral stent placement for an ovarian tumor compression of the ureters where the cause of death was progression of the malignancy.
The discharge-to-home rates were also significantly lower in the US News and World Report group, and at chart review of a random sample, these were very much, very similar; the issue was really a misattribution of patients to the urology service. So similarly, many had no urology contact or minimal consultative care. So again, for example, would be an inpatient exchange of an indwelling suprapubic tube that was unrelated to the hospitalization. We're not the only group who have looked at this. Other groups have reported also on this discrepancy between data that mimics US News and World Report methodology and what their actual care encompasses. And similar to what we have found, there's a lot of noise in the data to sift through.
So there are really two main take-home points that I draw from this work. The first is that the current methodology of US News and World Report doesn't accurately identify or capture urology service patients in its assessment of mortality or discharge to home. And it meant to me that looking at this data for reporting did not necessarily translate into concrete data for improvement to a granular level. So to the degree that US News and World Report provides this inaccurate assessment of quality of care, we believe it misleads patients regarding venues of care.
We point out some mitigating strategies, and the first is to include data from other payers. The newest iteration of US News and World Report methodology now does include, in addition to the Medicare fee-for-service, Medicare Advantage patients, but this likely will not be sufficient. The second is to use index procedures and their associated outcomes. The third, and US News and World Report has actually started to do this for the best hospitals but hasn't done this with specialty rankings, is to actually remove a numerical rank. Because is a patient who is number five really significantly better than number eight? And even from my standpoint, what does that really actually mean? And the fourth, and I put a star here, it's because I recognize this can be a bit of a hot take, is to have urologists remove ourselves from the US News and World Report rankings.
But I think unlike law schools and medical schools, the data that US News and World Report uses for healthcare is publicly available. So being able to recuse ourselves can be much more difficult and challenging than in other fields. So again, thank you for your time and I'd be more than happy to talk in depth on different aspects of our work.
Ruchika Talwar: Really, really interesting data that you present. And I think you're right, this has been a hot topic. We've hosted several conversations with folks who have done similar analyses of rankings. And a big challenge is that patients are looking to be more empowered in making their healthcare decisions, and so they do want to make sure that they are going to centers of excellence. I love the fact that the US News and World Report rankings are moving away from ordinal rankings. I think that's helpful because we can still designate high-volume centers of excellence that patients can use as a guide. But I think your point is well taken, what's the difference between number two and number 10? And frankly, I think there's very little variation between number one and number 20 in a lot of ways. And then patients are using resources and obviously financial resources being one of them to get to these real centers of excellence that may be ranked number one, but across the country, and it's really not a good use of healthcare dollars or resources in general. So really great points.
I'm curious as to what you think we can do as urologists when perhaps we are seeing patients in the community for rare disorders that we want to refer to different centers, or we're encouraging our own patients to seek second opinions to make sure that they're aligned with their plan of care. What's your advice to us as urologists?
Kathy Huen: Yeah, I would say, I think you pointed out a lot of great points. The first is these rankings are being used by patients, especially US News and World Report, over many years. And in some ways the hospital is a little complicit in this. We implicitly license their badges and marketing tools at a fee, which is undisclosed, in order to use that to also attract patients. And again, a bit of a hot take is we probably as hospital systems need to step away from doing that.
There are other ranking systems, and again, they each have their own pitfalls, such as Leapfrog, that also attempt to rank quality of care. I think what they all suffer from, again, is what is being used as a public metric and whether it's replicable to truly actually represent or reflect quality of care. So I think as a urologist, kind of a very personal opinion is I really try to step away from what I perceive as a US News and World Report ranking when recommending patients for quality of care, because again, myself as well as other groups have really shown that these rankings or these ordinal rankings definitely don't necessarily reflect the encounters and the procedures that patients are undergoing.
And I think a big point is to also point out there's value. These are aggregate findings, these are in Medicare patients over 65 years old, so these rankings might be completely not applicable to someone else who is undergoing—a 40-something-year-old person who might be undergoing a different procedure.
Ruchika Talwar: That's right. That's right. And certain, I think rare conditions also would suffer from the same inability to extrapolate US News and World Report rankings because they're not captured in a Medicare data set. I think diversifying payer mix is one way forward, of course, as you alluded to. But a lot of times it's really the rare things that patients need those centers of excellence for.
Kathy Huen: Yeah, no, I absolutely agree. I think that most people—and I'm in the peds world, and again, the peds best hospitals is kind of a separate topic too—I think identification of centers of excellence, a possibility is, and some of our specialties have already done that, is within our own field, rather than relying on external sources, identifying those centers of excellence that we feel would be able to take care of our patients in the best way.
Ruchika Talwar: Yeah, absolutely. Well, thank you so much for your time sharing this really important work. We're excited to get the conversation going and hopefully come up with solutions to a challenging problem as a field.
Kathy Huen: Thank you. Thank you, Dr. Talwar.
Ruchika Talwar: And to our audience, I hope you enjoyed this conversation. Thanks again for joining and we'll see you next time.
Ruchika Talwar: Hi, everyone. Welcome back to UroToday's Health Policy Center of Excellence. As always, my name is Ruchika Talwar, and today I'm excited to be joined by Dr. Kathy Huen from UCLA. She's going to be sharing some recent work that she did exploring the accuracy of US News and World Report rankings. Thank you, Dr. Huen, for being here with us.
Kathy Huen: Thank you so much for the invitation, and we're very excited to share our results. So thank you again, Dr. Talwar, for the invitation. I'm a pediatric urologist by training, but I serve as a quality officer for the Department of Urology and also a senior quality officer at UCLA Health. So the perceived importance of US News and World Report hospital rankings is all around us. I snapped this photo as I was walking to the main hospital, and this is a very widespread use of US News and World Report as a marketing tool. So that, as well as the withdrawal from US News and World Report of really dozens of law and medical schools, has prompted scrutiny of its methodology in recent years.
So as the department quality officer, hospital administrators had come to us regarding fluctuations regarding the department's rankings over the years, which really personally prompted a deep dive into the US News and World Report and specialty rankings. And what I found was kind of unsettling, that the patient population within our internal records was very discordant with who US News and World Report was identifying. And that motivated my team to really assess the accuracy of US News and World Report mortality and discharge-to-home measures for patients being treated for urologic conditions.
So US News and World Report, as a background, uses Medicare fee-for-service inpatient encounters that fall within certain Medical Severity Diagnosis-Related Groups, or MSDRGs, as we're more commonly understanding them. And because there can be significant errors in patient attribution when using these claim-based methods, we hypothesize that an approach based on the electronic medical record would be more useful as a reference standard to basically assess the accuracy of US News and World Report measures. So through our electronic medical record, which we use EPIC, we replicated US News and World Report's method of assigning patients to a hospital's urology service. And then from there we queried that same database for patients who were admitted or discharged under the urology service, who here we're calling the primary urology group.
So why mortality and why discharge to home? And this is because the outcomes component typically constitutes the most heavily weighted part of the US News and World Report rankings. It counts for about 40% of the rankings and other components are shown here as well. And this is the latest iteration because every year US News and World Report puts out a new methodology report, and the outcomes component continues to contribute the most. From a mortality standpoint, we found that there were 13 patient deaths that were identified in the US News and World Report group over the span of four years. At chart review, about four of these 13 patient deaths were primarily associated with urology procedure or hospitalization, which meant that the remaining nine, which is about 70%, were not. And in comparison, all three deaths that were identified in the primary urology group were associated with urology procedure hospitalization.
Some patients in the US News and World Report group had really no urology touch or consultation. Some had procedures that were unrelated to the cause of death, and a concrete example would be a ureteral stent placement for an ovarian tumor compression of the ureters where the cause of death was progression of the malignancy.
The discharge-to-home rates were also significantly lower in the US News and World Report group, and at chart review of a random sample, these were very much, very similar; the issue was really a misattribution of patients to the urology service. So similarly, many had no urology contact or minimal consultative care. So again, for example, would be an inpatient exchange of an indwelling suprapubic tube that was unrelated to the hospitalization. We're not the only group who have looked at this. Other groups have reported also on this discrepancy between data that mimics US News and World Report methodology and what their actual care encompasses. And similar to what we have found, there's a lot of noise in the data to sift through.
So there are really two main take-home points that I draw from this work. The first is that the current methodology of US News and World Report doesn't accurately identify or capture urology service patients in its assessment of mortality or discharge to home. And it meant to me that looking at this data for reporting did not necessarily translate into concrete data for improvement to a granular level. So to the degree that US News and World Report provides this inaccurate assessment of quality of care, we believe it misleads patients regarding venues of care.
We point out some mitigating strategies, and the first is to include data from other payers. The newest iteration of US News and World Report methodology now does include, in addition to the Medicare fee-for-service, Medicare Advantage patients, but this likely will not be sufficient. The second is to use index procedures and their associated outcomes. The third, and US News and World Report has actually started to do this for the best hospitals but hasn't done this with specialty rankings, is to actually remove a numerical rank. Because is a patient who is number five really significantly better than number eight? And even from my standpoint, what does that really actually mean? And the fourth, and I put a star here, it's because I recognize this can be a bit of a hot take, is to have urologists remove ourselves from the US News and World Report rankings.
But I think unlike law schools and medical schools, the data that US News and World Report uses for healthcare is publicly available. So being able to recuse ourselves can be much more difficult and challenging than in other fields. So again, thank you for your time and I'd be more than happy to talk in depth on different aspects of our work.
Ruchika Talwar: Really, really interesting data that you present. And I think you're right, this has been a hot topic. We've hosted several conversations with folks who have done similar analyses of rankings. And a big challenge is that patients are looking to be more empowered in making their healthcare decisions, and so they do want to make sure that they are going to centers of excellence. I love the fact that the US News and World Report rankings are moving away from ordinal rankings. I think that's helpful because we can still designate high-volume centers of excellence that patients can use as a guide. But I think your point is well taken, what's the difference between number two and number 10? And frankly, I think there's very little variation between number one and number 20 in a lot of ways. And then patients are using resources and obviously financial resources being one of them to get to these real centers of excellence that may be ranked number one, but across the country, and it's really not a good use of healthcare dollars or resources in general. So really great points.
I'm curious as to what you think we can do as urologists when perhaps we are seeing patients in the community for rare disorders that we want to refer to different centers, or we're encouraging our own patients to seek second opinions to make sure that they're aligned with their plan of care. What's your advice to us as urologists?
Kathy Huen: Yeah, I would say, I think you pointed out a lot of great points. The first is these rankings are being used by patients, especially US News and World Report, over many years. And in some ways the hospital is a little complicit in this. We implicitly license their badges and marketing tools at a fee, which is undisclosed, in order to use that to also attract patients. And again, a bit of a hot take is we probably as hospital systems need to step away from doing that.
There are other ranking systems, and again, they each have their own pitfalls, such as Leapfrog, that also attempt to rank quality of care. I think what they all suffer from, again, is what is being used as a public metric and whether it's replicable to truly actually represent or reflect quality of care. So I think as a urologist, kind of a very personal opinion is I really try to step away from what I perceive as a US News and World Report ranking when recommending patients for quality of care, because again, myself as well as other groups have really shown that these rankings or these ordinal rankings definitely don't necessarily reflect the encounters and the procedures that patients are undergoing.
And I think a big point is to also point out there's value. These are aggregate findings, these are in Medicare patients over 65 years old, so these rankings might be completely not applicable to someone else who is undergoing—a 40-something-year-old person who might be undergoing a different procedure.
Ruchika Talwar: That's right. That's right. And certain, I think rare conditions also would suffer from the same inability to extrapolate US News and World Report rankings because they're not captured in a Medicare data set. I think diversifying payer mix is one way forward, of course, as you alluded to. But a lot of times it's really the rare things that patients need those centers of excellence for.
Kathy Huen: Yeah, no, I absolutely agree. I think that most people—and I'm in the peds world, and again, the peds best hospitals is kind of a separate topic too—I think identification of centers of excellence, a possibility is, and some of our specialties have already done that, is within our own field, rather than relying on external sources, identifying those centers of excellence that we feel would be able to take care of our patients in the best way.
Ruchika Talwar: Yeah, absolutely. Well, thank you so much for your time sharing this really important work. We're excited to get the conversation going and hopefully come up with solutions to a challenging problem as a field.
Kathy Huen: Thank you. Thank you, Dr. Talwar.
Ruchika Talwar: And to our audience, I hope you enjoyed this conversation. Thanks again for joining and we'll see you next time.