To evaluate the accuracy, comprehensiveness, empathetic tone, and patient preference for AI and urologist responses to patient messages concerning common BPH questions across phases of care.
Cross-sectional study evaluating responses to 20 BPH-related questions generated by 2 AI chatbots and 4 urologists in a simulated clinical messaging environment without direct patient interaction. Accuracy, completeness, and empathetic tone of responses assessed by experts using Likert scales, and preferences and perceptions of authorship (chatbot vs. human) rated by non-medical evaluators.
Five non-medical volunteers independently evaluated, ranked, and inferred the source for 120 responses (n = 600 total). For volunteer evaluations, the mean (SD) score of chatbots, 3.0 (1.4) (moderately empathetic) was significantly higher than urologists, 2.1 (1.1) (slightly empathetic) (p < 0.001); mean (SD) and preference ranking for chatbots, 2.6 (1.6), was significantly higher than urologist ranking, 3.9 (1.6) (p < 0.001). Two subject matter experts (SMEs) independently evaluated 120 responses each (answers to 20 questions from 4 urologist and 2 chatbots, n = 240 total). For SME evaluations, mean (SD) accuracy score for chatbots was 4.5 (1.1) (nearly all correct) and not significantly different than urologists, 4.6 (1.2). The mean (SD) completeness score for chatbots was 2.4 (0.8) (comprehensive), significantly higher than urologists, 1.6 (0.6) (adequate) (p < 0.001).
Answers to patient BPH messages generated by chatbots were evaluated by experts as equally accurate and more complete than urologist answers. Non-medical volunteers preferred chatbot-generated messages and considered them more empathetic compared to answers generated by urologists.
World journal of urology. 2024 Dec 27*** epublish ***
Eric J Robinson, Chunyuan Qiu, Stuart Sands, Mohammad Khan, Shivang Vora, Kenichiro Oshima, Khang Nguyen, L Andrew DiFronzo, David Rhew, Mark I Feng
Department of Urology, Los Angeles Medical Center, Kaiser Permanente, Los Angeles, CA, USA., Department of Anesthesiology, Baldwin Park Medical Center, Kaiser Permanente, Baldwin Park, CA, USA., Kaiser Permanente, Pleasanton, CA, USA., Microsoft Health & Life Sciences, Irvine, CA, USA., Microsoft Health & Life Sciences, Dallas, TX, USA., Kaiser Permanente, Oakland, CA, USA., Department of Family Medicine, Kaiser Permanente, Pasadena, CA, USA., Kaiser Permanente, Pasadena, CA, USA., Microsoft Health & Life Sciences, Redmond, WA, USA., Department of Urology, Baldwin Park Medical Center, Kaiser Permanente, 1011 Baldwin Park Blvd., Baldwin Park, CA, 91706, USA. .