We aimed to assess the appropriateness of ChatGPT in providing answers related to prostate cancer (PCa) screening, comparing GPT-3.5 and GPT-4.
A committee of five reviewers designed 30 questions related to PCa screening, categorized into three difficulty levels.
The questions were formulated identically for both GPTs three times, varying the prompts. Each reviewer assigned a score for accuracy, clarity, and conciseness. The readability was assessed by the Flesch Kincaid Grade (FKG) and Flesch Reading Ease (FRE). The mean scores were extracted and compared using the Wilcoxon test. We compared the readability across the three different prompts by ANOVA.
In GPT-3.5 the mean score (SD) for accuracy, clarity, and conciseness was 1.5 (0.59), 1.7 (0.45), 1.7 (0.49), respectively for easy questions; 1.3 (0.67), 1.6 (0.69), 1.3 (0.65) for medium; 1.3 (0.62), 1.6 (0.56), 1.4 (0.56) for hard. In GPT-4 was 2.0 (0), 2.0 (0), 2.0 (0.14), respectively for easy questions; 1.7 (0.66), 1.8 (0.61), 1.7 (0.64) for medium; 2.0 (0.24), 1.8 (0.37), 1.9 (0.27) for hard. GPT-4 performed better for all three qualities and difficulty levels than GPT-3.5. The FKG mean for GPT-3.5 and GPT-4 answers were 12.8 (1.75) and 10.8 (1.72), respectively; the FRE for GPT-3.5 and GPT-4 was 37.3 (9.65) and 47.6 (9.88), respectively. The 2nd prompt has achieved better results in terms of clarity (all p < 0.05).
GPT-4 displayed superior accuracy, clarity, conciseness, and readability than GPT-3.5. Though prompts influenced the quality response in both GPTs, their impact was significant only for clarity.
International urology and nephrology. 2024 Apr 02 [Epub ahead of print]
Giuseppe Chiarelli, Alex Stephens, Marco Finati, Giuseppe Ottone Cirulli, Edoardo Beatrici, Dejan K Filipas, Sohrab Arora, Shane Tinsley, Mahendra Bhandari, Giuseppe Carrieri, Quoc-Dien Trinh, Alberto Briganti, Francesco Montorsi, Giovanni Lughezzani, Nicolò Buffi, Craig Rogers, Firas Abdollah
VUI Center for Outcomes Research, Analysis, and Evaluation, Henry Ford Health System, 2799 W Grand Blvd, Detroit, MI, 48202, USA., Public Health Sciences, Henry Ford Health System, Detroit, MI, USA., Division of Urological Surgery and Center for Surgery and Public Health, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA., Department of Urology and Renal Transplantation, University of Foggia, Foggia, Italy., Division of Oncology, Unit of Urology, IRCCS Ospedale San Raffaele, Vita-Salute San Raffaele University, Milan, Italy., Department of Urology, IRCCS Humanitas Research Hospital, Humanitas University, Milan, Italy., VUI Center for Outcomes Research, Analysis, and Evaluation, Henry Ford Health System, 2799 W Grand Blvd, Detroit, MI, 48202, USA. .
PubMed http://www.ncbi.nlm.nih.gov/pubmed/38564079