(UroToday.com)The 2023 EAU annual meeting included a game changing session, featuring a presentation by Dr. Maarten de Rooij discussing preliminary results from the PI-CAI challenge assessing artificial intelligence and radiologists at prostate cancer detection on MRI.
Dr. de Rooij started by highlighting that prostate cancer is the second most common cancer in men, with more than 1 million diagnoses every year globally. Furthermore, prostate cancer is responsible for 6.8% of all cancer deaths in men, with over 350,000 deaths every year worldwide. Both the EAU and NICE guidelines recommend that an MRI should be performed prior to all prostate biopsies, however, this can be associated with overdiagnosis and inter-reader variability secondary to semi-quantitative assessments. Artificial intelligence can assist in MRI interpretation, but lacks adequate scientific evidence for wide-scale clinical translation. A typical clinic workflow starts with a clinical suspicion of prostate cancer, followed by a multiparametric MRI, followed by prostate biopsy, followed by treatment. However, there can be sampling errors and false positives along the way, requiring follow-up of >3 years to catch any missed malignant lesions:
The PI-CAI challenge was designed to have a dataset of 10,207 cases from multi-center experience. Part 1 was the PI-CAI Challenge with Artificial Intelligence Models for developing artificial intelligence algorithms, the equivalent of clinical trials or “grand challenges.” This was hosted on www.grand-challenge.org, where artificial intelligence algorithms can be uploaded and validated in a fully blinded setting. Part 2 was the Reader Study with Radiologists, where prostate radiologists from around the world were invited to participate. Combining the two parts of the study, the primary endpoint was artificial intelligence versus radiologists, and secondary endpoints included: (i) artificial intelligence versus artificial intelligence, and (ii) radiologists versus radiologists.
This was a true global study with more than 830 individuals, 60+ artificial intelligence teams, 290+ artificial intelligence algorithms submitted, from 50+ countries. Overall, 81 radiologists (54 centers in 22 countries) are enlisted in the PI-CAI Reader Study:
An overview of the PI-CAI Challenge Open Development Phase is as follows:
When trained on <2000 cases, diagnostic performance of the top artificial intelligence models is comparable to radiologists’ performance reported in the literature:
In the second phase of the PI-CAI Challenge, these artificial intelligence models will be retrained on a private dataset of 9,107 cases, and performance will be re-evaluated on the hidden testing cohort.
Dr. de Rooij then asked: What is the diagnostic value of dynamic contrast-enhanced (DCE) MRI for the detection of clinically significant prostate cancer? This is the aim of the Reader Study, where 400 cases were divided into 4 blocks of 100 cases, with data from 3 centers in The Netherlands and 1 external center in Norway. Among 81 readers from 54 centers (22 countries), the median years of prostate cancer MRI experience was 7, (range 1-23 years), broken down to 58 expert readers (>1,000 cases read in total; >200 cases/year) and 23 non-expert readers (<1000 cases read in total; <200 cases/year). The Reader Study interface and workflow broken down by biparametric MRI and multiparametric MRI is as follows:
The preliminary results for case level PI-RADS operating points are based on 14 readers with 2-15 years (median 9 years) of experience and 13/14 being expert readers (biparametric MRI vs multiparametric MRI):
Additionally, there was very little difference in AUCs:
Dr. de Rooij concluded his presentation discussing preliminary results from the PI-CAI challenge assessing artificial intelligence and radiologists at prostate cancer detection on MRI with the following take-home messages:
- For the artificial intelligence arm, the open development phase with the top 5 algorithms is finished and Dr. de Rooij and colleagues are currently running the closed development phase
- For the reader study arm, preliminary results show that biparametric MRI had similar clinically significant prostate cancer detection rates to multiparametric MRI
- Experience, workflow, image quality and protocol familiarity must be evaluated in future work
- In the near future, the primary endpoint will be reported for the comparison of artificial intelligence versus radiologists
Presented by: Maarten de Rooij, MD, PhD, Radboud University Medical Center, Nijmegen, Netherlands
Written by: Zachary Klaassen, MD, MSc – Urologic Oncologist, Assistant Professor of Urology, Georgia Cancer Center, Augusta University/Medical College of Georgia, @zklaassen_md on Twitter during the 2023 European Association of Urology (EAU) Annual Meeting, Milan, IT, Fri, Mar 10 – Mon, Mar 13, 2023.