ChatGPT is an artificial intelligence platform available to patients seeking medical advice. Traditionally, urology patients consulted official provider-created materials, particularly the Urology Care Foundation™ (UCF). Today, men increasingly go online due to the rising costs of health care and the stigma surrounding sexual health. Online health information is largely inaccessible to laypersons as it exceeds the recommended American sixth to eighth grade reading level. We conducted a comparative assessment of patient education materials generated by ChatGPT vs UCF regarding men's health conditions.
Methods
All 6 UCF men's health resources were identified. ChatGPT responses were generated using patient questions obtained from UCF. Adjusted ChatGPT responses were generated by prompting, "Explain it to me like I am in sixth grade." Textual analysis was performed using sentence, word, syllable, and complex word count. Six validated formulae were used for readability analysis. Two physicians independently scored responses for accuracy, comprehensiveness, and understandability. Statistical analysis involved Wilcoxon matched-pairs test.
Results
ChatGPT responses were longer and more complex. Both UCF and ChatGPT failed official readability standards, although ChatGPT performed significantly worse across all 6 topics (all P < .001). Conversely, adjusted ChatGPT readability typically surpassed UCF, even meeting the recommended level for 2 topics. Qualitatively, UCF and ChatGPT had comparable accuracy, although ChatGPT had better comprehensiveness and worse understandability.
Conclusions
When comparing readability, ChatGPT-generated education is less accessible than provider-written content, although neither meets the recommended level. Our analysis indicates that specific artificial intelligence prompts can simplify educational materials to meet national standards and accommodate individual literacy.
Yash B Shah,1 Anushka Ghosh,1 Aaron R Hochberg,1 Eli Rapoport,2 Costas D Lallas,1 Mihir S Shah,1 Seth D Cohen2
- Department of Urology, Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania.
- Department of Urology, NYU Langone, New York, New York.