Read the Full Video Transcript

Ruchika Talwar: Hi everyone. Welcome back to UroToday's Health Policy Center of Excellence. As always, my name is Dr. Ruchika Talwar. And today, I'm joined by Dr. Kaldany, who's a PGY-4 urology resident at Rutgers Robert Wood Johnson's Department of Urology. He'll be discussing his recent work exploring the effect of USMLE cutoffs on the urology match. Thanks, Dr. Kaldany, for being here with us.

Alain Kaldany: Thank you for having me. Happy to be here. So, to provide some background, essentially, the USMLE Step 1 and 2 scores have historically been used as screening tools for applicants to urology residency in the US, and a 2015 survey suggested that more than 80% of urology programs actually do use USMLE scores as a screening tool. However, this process has been criticized for disproportionately negatively impacting those who are underrepresented in urology in their efforts to seek interviews. Furthermore, it's been shown that step scores actually don't correlate with your clinical performance as a resident. Notably, in 2022, Step 1 was transitioned from a numeric scoring system to a pass/fail scoring system. And that move was criticized by many because there were concerns that we would lose an objective way of evaluating applicants. And so, we sought to determine how the use of USMLE score cutoffs as a screening tool during the match might affect the recruitment of underrepresented applicants.

We did this by looking at data from the ERAS system, which is the online system that applicants use to apply to residency. We looked at a five-year time period from 2018 to 2022 and looked at all applicants to our institution's residency program. And then we extracted demographic and academic variables. Demographic variables included age, sex, gender, and self-reported identity, which is essentially an estimation of self-reported ethnicity or race. And academic variables included whether they graduated from a US medical school, how much research activity they engaged in, their membership in any honor societies, and of course, their USMLE scores. Applicants were then categorized into either underrepresented or non-underrepresented (or well-represented) based on their self-reported identity, where underrepresented applicants included Black/African American, Hispanic/Latino, and Native American applicants. And well-represented applicants included Asian and White applicants. And then, essentially, we just performed some descriptive statistics where we looked at the percentage of applicants based on different USMLE score cutoffs in 10-point intervals from 200 up to 260.

And you'll see what I mean in the next slide. So overall, we had more than 1200 applicants over the five-year time period, about 70% were male and 30% were female. And similar numbers regarding 70% were well-represented and about 20% were underrepresented applicants. And what we saw was that as the cutoff score increased, the underrepresented applicants were disproportionately affected. So, in other words, for example, if you used a cutoff score of 240 for Step 1, you'd be excluding 60% of the underrepresented applicants compared to only 40% of the well-represented applicants, and this held true for Step 1 as well as Step 2. So, basically, to summarize, we found that underrepresented applicants were disproportionately affected by USMLE score cutoffs compared to those who are well-represented. And we hypothesized that perhaps transitioning from numeric scoring to a pass/fail grading system and eliminating the use of USMLEs as screening tools may actually benefit underrepresented applicants and potentially increase diversity among urology residents.

Ruchika Talwar: Thanks so much, Dr. Kaldany. I think this is a really important topic lately, especially since there's been a focus on diversifying our workforce for several reasons. I think most importantly, it is important that the urology workforce resembles the patient population that they treat. We know that this will allow for higher quality care overall, but also the demographic of physicians is really changing. We're seeing a larger trend towards female physicians, for example, whereas historically there were fewer female graduates from medical school and also obviously fewer female urologists entering the workforce. So, to kick off our discussion here, I'm curious, how did you get interested in this topic? What was sort of the nidus for this research question?

Alain Kaldany: Yeah, it's actually kind of a funny story. We were just... Dr. Jang, who is my attending and our program director here at Rutgers, and we were just kind of talking about... he was giving us a presentation about the applicants who we had received, and he looked at just one year, it was last year's data, and he was kind of presenting everything. He gave a couple of different figures in this presentation, and one of them showed the step score ranges of the applicants that we had, and one of them was the demographic makeup of the applicants that we had. And I noticed there was a spread across both of them. And I kind of thought, "I wonder if there's any correlation between demographics and step score." The other thing that really got me thinking about it is because we've recently, last year, transitioned to a more holistic application reviewing process at our institution, which is something that I think is happening across the country in programs.

And we actually transitioned as part of this holistic screening process; it's score-blind. And so we don't take into account USMLE scores when we're screening applicants for our residency program. And then the third thing that got me thinking about it is sort of what I alluded to before, which is people were kind of up in arms when... there were mixed reviews, I should say, when the USMLE announced that they would be transitioning to pass/fail scoring. Right. And I think there are pros and cons to that decision, but I've been hearing a lot of people pointing out the cons. And I think that it's a little bit harder to suss out what are some of the pros, and this represents one of the pros, essentially.

Ruchika Talwar: Yeah, I couldn't agree more. And to address your latter point, what are your thoughts on a better criterion? Because the criticism we always hear is, well, if USMLE step scores are going to pass/fail, we need some sort of criteria to screen out applicants. And the point you made in your introduction slides was that USMLE step scores don't necessarily correlate with clinical performance. We know that, yet still, people latch onto this idea of needing a screening tool. So, in your opinion, based on this investigation and obviously the research you've done on this topic, what are your thoughts on what tool could we use as a proxy?

Alain Kaldany: Yeah, so there are a lot of different things we can look at. An applicant is an extremely unique person, and they have all these different criteria in their application that we should be looking at. I think some of the more salient points from their application are research activity, what their letters of recommendation look like, what kind of leadership experience they have, reading their personal statements, and even clerkship grades potentially. Each of these things has issues. And you could argue that they're not really objective either, but there's a way to make that objective, right? If you have a rubric, for instance, that all of your application reviewing committee members use, and then they can sort of rank applicants based on that or screen applicants based on a minimum score in that rubric perhaps. But I think those are the sorts of things that you should be looking for from a more holistic point of view, life experiences, research experience, that sort of thing.

Ruchika Talwar: Yeah, exactly. And it'll be interesting because there's been emerging literature in this space demonstrating the value of holistic review. Lots of programs have been reporting their experience in using these sorts of alternative residency screening tools. So, I'm really looking keenly to see what the next few years bring as we try to improve the way we conduct the urology interview and match process. I'm glad we're having these kinds of conversations because I think it's important to highlight the fact we do need to review our practices and work towards diversifying our workforce from all aspects. So kudos to you for doing this work, and as we wrap up our discussion here, what are your final thoughts for the urologic community regarding takeaways from this study?

Alain Kaldany: Yeah, well, thank you so much. It was well said. I think the main takeaways from this study are that we do need to increase the diversity of our workforce. There's good data out there that suggests that patients have better outcomes, patients have better experiences, and providers have better experiences when the physician workforce is diversified. And I think that moving away from using scores, standardized test scores as a screening tool, we've demonstrated could potentially be one way to do so.

Ruchika Talwar: Absolutely. Well, we look forward to your future work in this space, and thanks again for taking the time to chat with us today.

Alain Kaldany: Happy to chat. Thanks for having me.

Ruchika Talwar: And to our audience, thanks for joining. We'll see you next time.