Urology Program Directors Rank Letters of Recommendation as Top Factor in Residency Applications Post-USMLE Step 1 Change - Angeline Johny & Wesley Mayer
May 6, 2024
Ruchika Talwar hosts Angeline Johny and Wesley to discuss the implications of the USMLE Step 1 exam's shift to a pass/fail format on residency applications in urology. They examine the growing need for objective measures in the application process amidst increasing application volumes and the fading of traditional metrics like class rank and detailed clerkship grades. Their study, utilizing a survey conducted by the Society of Academic Urology, sought the perspectives of urology program directors on the importance of various application parameters. Their findings highlight the significance of standardized letters of recommendation, especially in the context of an evolving medical education landscape where traditional evaluation metrics are becoming less reliable. The discussion underscores the challenges and potential biases in residency selection, emphasizing the need for innovative approaches to fairly assess candidates.
Biographies:
Angeline Johny, MD, Urologist, Baylor College of Medicine, Houston, TX
Wesley Mayer, MD, Associate Professor of Urology, Vice Chair of Education for Urology, Baylor College of Medicine, Houston, TX
Ruchika Talwar, MD, Urologic Oncology Fellow, Department of Urology, Vanderbilt University Medical Center, Nashville, TN
Biographies:
Angeline Johny, MD, Urologist, Baylor College of Medicine, Houston, TX
Wesley Mayer, MD, Associate Professor of Urology, Vice Chair of Education for Urology, Baylor College of Medicine, Houston, TX
Ruchika Talwar, MD, Urologic Oncology Fellow, Department of Urology, Vanderbilt University Medical Center, Nashville, TN
Read the Full Video Transcript
Ruchika Talwar: Hi, everyone. Welcome back to UroToday's Health Policy Center of Excellence. Today, I'm joined by Dr. Johny, who's a chief resident, and Dr. Mayer, who's an associate professor of urology, both from Baylor College of Medicine. They'll be discussing their work on program directors' perspectives regarding residency applications in the post-USMLE Step 1 era. Is this a case for standardized letters of recommendation? Thanks to you both for being here with us today.
Wesley Mayer: Thank you.
Angeline Johny: Thanks so much, Dr. Talwar, and to UroToday for offering us this opportunity. The catalyst for this project was the transition of the USMLE Step 1 licensing exam going from a numeric score to a binary pass/fail. In this context, there have been several additional medical education changes, including the shift from a clerkship grade to a binary pass/fail, subsequently leading to a shift away from class rank, and as a result of both of these changes, a shift in how AOA is being defined. Along with this context, we also see a steady rise in the volume of applications per candidate every year, even from 2015 onwards. With the addition of preference signaling, which is a new addition to the urology application in 2024 at a 30 preference signal limit, we still see an average number of applications at 67, so there is a clear need to assess objective application parameters along with the backdrop of a rise in the number of urology applications.
The primary objective of this study was to see what perspectives urology program directors had regarding residency application parameters and how they view the relative importance of these. As I mentioned, the catalyst was the Step 1 transition to pass/fail. The SAU, Society of Academic Urology, commanded a task force to understand and get a better appreciation of this impact. They disseminated a survey to all 147 program directors starting in January 2022, and the survey closed in March 2022. The survey itself had different components to it. The first was a Likert-style questionnaire in which they wanted to assess their level of agreement with how certain application parameters predicted clinical performance. The second aspect of the questionnaire was to ask program directors to physically rank 11 different parameters on a scale of one to 11, one being the most important, 11 being the least important.
We worked with our statistician here at Baylor and asked them to perform a variety of different tests, including Friedman's test as well as a Pairwise Wilcoxon test to assess if this ranking was statistically significant. One strength I wanted to highlight in this study was the response rate. Of the 147 program directors, 89 responded, and that yielded a 60.5% response rate, which is great because it showcases a large majority of the program directors and represents a majority of perspectives across the nation. The most exciting part, in my opinion, of our study is the results. This plot here has 11 different hourglasses. Each hourglass represents a parameter that was ranked, and the center of the hourglass represents the median. The length of the hourglass represents the interquartile range or the confidence interval over the median. The center of the hourglass, if there's a lack of overlap, that denotes statistical significance.
You can see here that the first parameter that's ranked highest is the letter of recommendation, and there is no overlap of this hourglass with any other parameter. You do see some overlap with the second and third, fourth and fifth, but by and large, the letter of recommendation was ranked higher than any other parameter. If you look at the length of the hourglass, this has the least amount of heterogeneity of response across the board compared to any other parameter that program directors ranked. Looking at their Likert scale questions, we found that program directors also agreed that letters of recommendation and clerkship grades did predict clinical performance, and so it is clear, looking at this, that these are parameters that are prioritized and valued by program directors. But what does this mean? What do we do with this information? I think before we answer that question, we have to take a step back and look at the backdrop of medical education and the urology application process as a whole.
As I mentioned before, application volume is a huge challenge. Even with the introduction of preference signaling, it's an ongoing battle for program directors to holistically and fairly review a large number of applications. I mentioned objective parameters dwindling and becoming increasingly nebulous, namely clerkship grades. This was ranked as the second most important parameter; however, there's been a gradual transition away from objective grades to a binary pass/fail system. In institutions where there are still grades, there's rampant grade inflation. Studies have shown that there are racial and cultural biases that are imbued in the clerkship grading process and there are limitations to the current traditional narrative letter of recommendation. While it does offer greater flexibility for the letter writer to endorse a candidate, there is greater subjectivity imbued in the writing process. There's less inter-rater reliability, and many studies, both within and outside of urology, have shown that letters of recommendation have biases against female applicants.
Where does that lead us? What trajectory are we going towards now? I think that the case for standardizing letters of recommendation comes from a place of trying to balance all of those factors and nuances that I mentioned while also trying to maintain a holistic review and mitigate bias when it comes to reviewing applications. This has been done previously in other specialties including emergency medicine, as well as a slow adoption within urology, and we've seen studies in both fields that have shown that this process allows for a mitigation of bias in general. It's not a perfect system, though. I think that it's important to note that there are challenges with this system, namely the ceiling effect.
Everyone can't be an exceptional candidate, and it's hard to distinguish applicants when you have a writer saying that everyone is exceptional or everyone is outstanding. I think that given the findings that we have from this paper, it makes an argument for putting our priorities on innovating this opportunity to not only streamline the application process for urology PDs but for applicants as well, and continue bettering the field of urology. That's all I have. These are my citations here. Once again, thank you guys so much for this opportunity.
Ruchika Talwar: Thanks, Dr. Johny. This is really important work. Here at the Health Policy Center of Excellence for UroToday, we've been focusing a lot on our workforce shortage, and ensuring a diverse and steady pipeline of trained urologists is going to be one key aspect of addressing that issue. All of the work that you've presented, I think, is really important to understand the context in which we now evaluate residency applicants. You mentioned Step 1 has become pass/fail. I'll also add that there has been some criticism of the standardized letter of recommendation, although letters are universally agreed to be the most important aspect of evaluating an applicant. Dr. Mayer, I'll pose this question to you. What advice do you have for letter writers when using the standard template to address some concerns out there regarding the inability to effectively get an idea of an applicant, as well as perhaps different evaluators having different biases in their own rating techniques?
Wesley Mayer: Yeah. I think that's a great question. I think Dr. Johny mentioned some of the biggest shortcomings here being that not everybody can be exceptional, so then we have to, as a specialty, not brand people that just because they didn't get the top rank on these six categories, "Sorry, that's not Baylor material," for example, and the flexibility of being able to talk about a future trainee, an applicant, in terms of what they could bring to your program, culture. What's the cultural fit? I think that's important, so I think, for me, the letter writers who do a little bit of both, who give you some idea of how that person interacts with the service, interacts with the trainees, what they are like as a person, and then also gives some benchmarks and some very essential characteristics that we're all looking for in applicants, I think that's probably where...
It's a little bit of everything, but finding that's tough. I know Dr. Richstone has worked on this and done some important work looking at this. When we did the virtual elective experience in the COVID era, we all accelerated this process and looked at it quite a bit in some task forces through the SAU, but I think it's a great question, and I think we're going to have to find the answer together.
Ruchika Talwar: Yeah, absolutely. Dr. Johny, I'm curious. You mentioned work that has shown that women are often affected negatively by letters of recommendation due to the fact that certain adjectives are used to describe women more frequently than men, and there have been several studies, specifically outside of UNC, looking at this in urology letters. What advice do you have for letter writers as they create these standardized letters with that background in mind? What can we do better to help alleviate some of the gender disparities we see, but probably also racial and ethnic disparities that are embedded in this process?
Angeline Johny: Yeah, that's a great question. I think the first thing that came to my mind is when we describe female applicants, there's a lot of emotional pathos involved in it, and it's less so about their skillset, what their knowledge base is, or their intellect. I think being aware of that bias is the most important thing to tackling it. As a letter writer, when you're writing your letter, avoiding adjectives that convey any emotional pathos is crucial. I think it's important to have future trainees and physicians who are emotionally in tune, sensitive, and have a high EQ, but that shouldn't be the only thing they're known for. We're training surgeons, and we want to be sure that we have future trainees who are teachable and willing to learn. I think number one is awareness, and number two is just being consciously aware of it and consciously trying to address the fact that these biases can infect their way into a letter.
It's a really hard aspect. I think you mentioned cultural biases as well. Coming from an Indian background, humility is highly valued, and sometimes that can be negatively viewed if you're not perceived as confident. I think just having that awareness and having diversity amongst letter writers is really valuable and important too. I mean, I've seen gradual changes even within my five years of training, and I think urology as a field and community is doing the right thing by asking these questions and not just resting on our laurels. I think these are all really important catalysts for change.
Ruchika Talwar: Yeah, totally. I think these conversations are really important, just even addressing the fact that sometimes these issues exist and having people hear these discussions makes you more aware when you sit down to write your letter. I think this is all one step in the right direction, and your work certainly helps us start those conversations. Dr. Mayer, what about, in general, applying to the residency process now, we have virtual interviews, some schools and some fields have limitations on how many rotations can be done, what's your advice to people who may be coming from smaller schools, perhaps schools that don't have a urology program or perhaps don't have well-known urologists as it relates to getting a strong letter of recommendation from a well-known person in the field?
Wesley Mayer: Yeah. It's a great question because it's a big challenge. I think there's been a lot of flexibility towards those applicants in those situations, from schools that don't have programs. They're allowed to do more urology-away rotations and things like that. I think some of the challenges that you brought up, the virtual world also creates some potential opportunities because I think a lot of applicants have been able to collaborate remotely with well-known people that actually can write letters and do have that name recognition, even if their local spheres are mostly private urologists that got them into urology, so I think it's about leveraging the technology that can allow you to connect.
Certainly, social media and other forms would be ways to reach out and learn about programs and meet people, but I think that it's going to take work. I mean, it always has taken work when you come from a smaller program. In many ways, perhaps the connected world makes it maybe a little easier, although I guess it still remains to be seen if we continue to do virtual interviews. The world is changing. AAMC hasn't spoken yet this year, to my knowledge, so I guess we'll see how that all goes.
Ruchika Talwar: Yeah, it really is a rapidly evolving landscape post-COVID. I think a lot of the changes came with positives, a lot of the changes came with some negatives, but we'll see how the way forward looks. Thank you both for joining us today. Again, I think these discussions are really important because ensuring that we have a workforce that can care for our urologic patients in the future means that we need to make sure we're attracting the very best talent and perfecting our residency selection process. Congratulations on this work being published in Urology Practice, and we appreciate your time.
Wesley Mayer: Thanks. If I can, I just wanted to mention that I want to thank the SAU who put this task force together and all the task force members. By no means, I just didn't want to confuse the idea that the SAU is supporting or advocating for a standardized letter of recommendation just because it might be as we lose the... As the second most important parameter trends towards pass/fail and the third most important parameter is gone, we pretty much... I think there's opportunity there, and also to Dr. Chuck Green at the University of Texas for helping us with the statistics. Thanks, everyone.
Angeline Johny: It was definitely a team effort and a lot of hard work was put into this study, so it was an honor to get to present it.
Ruchika Talwar: Great. Well, we appreciate that. To our audience, thanks so much for joining us. We'll see you next time.
Ruchika Talwar: Hi, everyone. Welcome back to UroToday's Health Policy Center of Excellence. Today, I'm joined by Dr. Johny, who's a chief resident, and Dr. Mayer, who's an associate professor of urology, both from Baylor College of Medicine. They'll be discussing their work on program directors' perspectives regarding residency applications in the post-USMLE Step 1 era. Is this a case for standardized letters of recommendation? Thanks to you both for being here with us today.
Wesley Mayer: Thank you.
Angeline Johny: Thanks so much, Dr. Talwar, and to UroToday for offering us this opportunity. The catalyst for this project was the transition of the USMLE Step 1 licensing exam going from a numeric score to a binary pass/fail. In this context, there have been several additional medical education changes, including the shift from a clerkship grade to a binary pass/fail, subsequently leading to a shift away from class rank, and as a result of both of these changes, a shift in how AOA is being defined. Along with this context, we also see a steady rise in the volume of applications per candidate every year, even from 2015 onwards. With the addition of preference signaling, which is a new addition to the urology application in 2024 at a 30 preference signal limit, we still see an average number of applications at 67, so there is a clear need to assess objective application parameters along with the backdrop of a rise in the number of urology applications.
The primary objective of this study was to see what perspectives urology program directors had regarding residency application parameters and how they view the relative importance of these. As I mentioned, the catalyst was the Step 1 transition to pass/fail. The SAU, Society of Academic Urology, commanded a task force to understand and get a better appreciation of this impact. They disseminated a survey to all 147 program directors starting in January 2022, and the survey closed in March 2022. The survey itself had different components to it. The first was a Likert-style questionnaire in which they wanted to assess their level of agreement with how certain application parameters predicted clinical performance. The second aspect of the questionnaire was to ask program directors to physically rank 11 different parameters on a scale of one to 11, one being the most important, 11 being the least important.
We worked with our statistician here at Baylor and asked them to perform a variety of different tests, including Friedman's test as well as a Pairwise Wilcoxon test to assess if this ranking was statistically significant. One strength I wanted to highlight in this study was the response rate. Of the 147 program directors, 89 responded, and that yielded a 60.5% response rate, which is great because it showcases a large majority of the program directors and represents a majority of perspectives across the nation. The most exciting part, in my opinion, of our study is the results. This plot here has 11 different hourglasses. Each hourglass represents a parameter that was ranked, and the center of the hourglass represents the median. The length of the hourglass represents the interquartile range or the confidence interval over the median. The center of the hourglass, if there's a lack of overlap, that denotes statistical significance.
You can see here that the first parameter that's ranked highest is the letter of recommendation, and there is no overlap of this hourglass with any other parameter. You do see some overlap with the second and third, fourth and fifth, but by and large, the letter of recommendation was ranked higher than any other parameter. If you look at the length of the hourglass, this has the least amount of heterogeneity of response across the board compared to any other parameter that program directors ranked. Looking at their Likert scale questions, we found that program directors also agreed that letters of recommendation and clerkship grades did predict clinical performance, and so it is clear, looking at this, that these are parameters that are prioritized and valued by program directors. But what does this mean? What do we do with this information? I think before we answer that question, we have to take a step back and look at the backdrop of medical education and the urology application process as a whole.
As I mentioned before, application volume is a huge challenge. Even with the introduction of preference signaling, it's an ongoing battle for program directors to holistically and fairly review a large number of applications. I mentioned objective parameters dwindling and becoming increasingly nebulous, namely clerkship grades. This was ranked as the second most important parameter; however, there's been a gradual transition away from objective grades to a binary pass/fail system. In institutions where there are still grades, there's rampant grade inflation. Studies have shown that there are racial and cultural biases that are imbued in the clerkship grading process and there are limitations to the current traditional narrative letter of recommendation. While it does offer greater flexibility for the letter writer to endorse a candidate, there is greater subjectivity imbued in the writing process. There's less inter-rater reliability, and many studies, both within and outside of urology, have shown that letters of recommendation have biases against female applicants.
Where does that lead us? What trajectory are we going towards now? I think that the case for standardizing letters of recommendation comes from a place of trying to balance all of those factors and nuances that I mentioned while also trying to maintain a holistic review and mitigate bias when it comes to reviewing applications. This has been done previously in other specialties including emergency medicine, as well as a slow adoption within urology, and we've seen studies in both fields that have shown that this process allows for a mitigation of bias in general. It's not a perfect system, though. I think that it's important to note that there are challenges with this system, namely the ceiling effect.
Everyone can't be an exceptional candidate, and it's hard to distinguish applicants when you have a writer saying that everyone is exceptional or everyone is outstanding. I think that given the findings that we have from this paper, it makes an argument for putting our priorities on innovating this opportunity to not only streamline the application process for urology PDs but for applicants as well, and continue bettering the field of urology. That's all I have. These are my citations here. Once again, thank you guys so much for this opportunity.
Ruchika Talwar: Thanks, Dr. Johny. This is really important work. Here at the Health Policy Center of Excellence for UroToday, we've been focusing a lot on our workforce shortage, and ensuring a diverse and steady pipeline of trained urologists is going to be one key aspect of addressing that issue. All of the work that you've presented, I think, is really important to understand the context in which we now evaluate residency applicants. You mentioned Step 1 has become pass/fail. I'll also add that there has been some criticism of the standardized letter of recommendation, although letters are universally agreed to be the most important aspect of evaluating an applicant. Dr. Mayer, I'll pose this question to you. What advice do you have for letter writers when using the standard template to address some concerns out there regarding the inability to effectively get an idea of an applicant, as well as perhaps different evaluators having different biases in their own rating techniques?
Wesley Mayer: Yeah. I think that's a great question. I think Dr. Johny mentioned some of the biggest shortcomings here being that not everybody can be exceptional, so then we have to, as a specialty, not brand people that just because they didn't get the top rank on these six categories, "Sorry, that's not Baylor material," for example, and the flexibility of being able to talk about a future trainee, an applicant, in terms of what they could bring to your program, culture. What's the cultural fit? I think that's important, so I think, for me, the letter writers who do a little bit of both, who give you some idea of how that person interacts with the service, interacts with the trainees, what they are like as a person, and then also gives some benchmarks and some very essential characteristics that we're all looking for in applicants, I think that's probably where...
It's a little bit of everything, but finding that's tough. I know Dr. Richstone has worked on this and done some important work looking at this. When we did the virtual elective experience in the COVID era, we all accelerated this process and looked at it quite a bit in some task forces through the SAU, but I think it's a great question, and I think we're going to have to find the answer together.
Ruchika Talwar: Yeah, absolutely. Dr. Johny, I'm curious. You mentioned work that has shown that women are often affected negatively by letters of recommendation due to the fact that certain adjectives are used to describe women more frequently than men, and there have been several studies, specifically outside of UNC, looking at this in urology letters. What advice do you have for letter writers as they create these standardized letters with that background in mind? What can we do better to help alleviate some of the gender disparities we see, but probably also racial and ethnic disparities that are embedded in this process?
Angeline Johny: Yeah, that's a great question. I think the first thing that came to my mind is when we describe female applicants, there's a lot of emotional pathos involved in it, and it's less so about their skillset, what their knowledge base is, or their intellect. I think being aware of that bias is the most important thing to tackling it. As a letter writer, when you're writing your letter, avoiding adjectives that convey any emotional pathos is crucial. I think it's important to have future trainees and physicians who are emotionally in tune, sensitive, and have a high EQ, but that shouldn't be the only thing they're known for. We're training surgeons, and we want to be sure that we have future trainees who are teachable and willing to learn. I think number one is awareness, and number two is just being consciously aware of it and consciously trying to address the fact that these biases can infect their way into a letter.
It's a really hard aspect. I think you mentioned cultural biases as well. Coming from an Indian background, humility is highly valued, and sometimes that can be negatively viewed if you're not perceived as confident. I think just having that awareness and having diversity amongst letter writers is really valuable and important too. I mean, I've seen gradual changes even within my five years of training, and I think urology as a field and community is doing the right thing by asking these questions and not just resting on our laurels. I think these are all really important catalysts for change.
Ruchika Talwar: Yeah, totally. I think these conversations are really important, just even addressing the fact that sometimes these issues exist and having people hear these discussions makes you more aware when you sit down to write your letter. I think this is all one step in the right direction, and your work certainly helps us start those conversations. Dr. Mayer, what about, in general, applying to the residency process now, we have virtual interviews, some schools and some fields have limitations on how many rotations can be done, what's your advice to people who may be coming from smaller schools, perhaps schools that don't have a urology program or perhaps don't have well-known urologists as it relates to getting a strong letter of recommendation from a well-known person in the field?
Wesley Mayer: Yeah. It's a great question because it's a big challenge. I think there's been a lot of flexibility towards those applicants in those situations, from schools that don't have programs. They're allowed to do more urology-away rotations and things like that. I think some of the challenges that you brought up, the virtual world also creates some potential opportunities because I think a lot of applicants have been able to collaborate remotely with well-known people that actually can write letters and do have that name recognition, even if their local spheres are mostly private urologists that got them into urology, so I think it's about leveraging the technology that can allow you to connect.
Certainly, social media and other forms would be ways to reach out and learn about programs and meet people, but I think that it's going to take work. I mean, it always has taken work when you come from a smaller program. In many ways, perhaps the connected world makes it maybe a little easier, although I guess it still remains to be seen if we continue to do virtual interviews. The world is changing. AAMC hasn't spoken yet this year, to my knowledge, so I guess we'll see how that all goes.
Ruchika Talwar: Yeah, it really is a rapidly evolving landscape post-COVID. I think a lot of the changes came with positives, a lot of the changes came with some negatives, but we'll see how the way forward looks. Thank you both for joining us today. Again, I think these discussions are really important because ensuring that we have a workforce that can care for our urologic patients in the future means that we need to make sure we're attracting the very best talent and perfecting our residency selection process. Congratulations on this work being published in Urology Practice, and we appreciate your time.
Wesley Mayer: Thanks. If I can, I just wanted to mention that I want to thank the SAU who put this task force together and all the task force members. By no means, I just didn't want to confuse the idea that the SAU is supporting or advocating for a standardized letter of recommendation just because it might be as we lose the... As the second most important parameter trends towards pass/fail and the third most important parameter is gone, we pretty much... I think there's opportunity there, and also to Dr. Chuck Green at the University of Texas for helping us with the statistics. Thanks, everyone.
Angeline Johny: It was definitely a team effort and a lot of hard work was put into this study, so it was an honor to get to present it.
Ruchika Talwar: Great. Well, we appreciate that. To our audience, thanks so much for joining us. We'll see you next time.