ROC analysis for clinical decision support systems (CDSS) results in digital mammography images

Artificial intelligence and clinical decision support systems (CDSS) are being actively implemented in healthcare. Radiology is at the forefront of the use of such technologies. In this article, we describe a method for evaluating the performance of CDSS, including software based on artificial intelligence technologies (AI-based software), which is suitable for any medical organization that needs to assess the applicability of such software.

Purpose: The purpose of this study is to demonstrate the use of a web-based ROC analysis tool for evaluating the performance of clinical decision support systems (CDSS) using digital mammography images as an example.

Materials and methods: A retrospective dataset of mammography studies was used, based on the results of the calibration test report during the version change of one of the AI service participating in the Experiment on the use of innovative computer vision technologies for medical image analysis and subsequent applicability in the healthcare system of Moscow, with versions of the AI service dated 15.02.2023 and 30.05.2023. The sample size consisted of 100 trials. In this publication, ROC analysis implemented using a web-based tool will be used to evaluate the results of the AI service.

Results: The functionality of a web-based tool for ROC analysis was demonstrated using the example of evaluating the performance of AI-based software for processing digital mammography images.

Conclusion: By using the presented web-based ROC analysis tool, the verification of СDSS, including AI-based software, as well as the assessment of its performance, can be performed without the need for additional tools if necessary.

1. Chetverikov S.F., Arzamasov K.M., Andreichenko A.E. et al. Approaches to sampling for quality control of artificial intelligence in biomedical research. Sovremennye tehnologii v medicine. 2023; 2: 19–27. https://doi.org/10.17691/stm2023.15.2.02 (In Russian)

2. Govorukhina V.G., Semenov S.S., Gelezhe P.B. et al. The role of mammography in breast cancer radiomics. Digital Diagnostics. 2021; 2 (2): 185−199. https://doi.org/10.17816/DD70479 (In Russian)

3. Alekseeva M.G., Zubov A.I., Novikov M.Yu. Artificial intelligence in medicine. International Research Journal. 2022; 7: 10−13. https://doi.org/10.23670/IRJ.2022.121.7.038 (In Russian)

4. Roitberg PG, Blinov DS, Cheremisin VM. Integrating AI technologies in radiology workflows. Health Care Standardization Problems. 2020; 9 (10): 29−33. https://doi.org/10.26347/1607-2502202009-10029-033 (In Russian)

5. Rodriguez-Ruiz A., Lang K., Gubern-Merida A. et al. Stand-Alone Artificial Intelligence for Breast Cancer Detection in Mammography: Comparison With 101 Radiologists. J. Natl. Cancer Inst. 2019; 111 (9): 916−922. https://doi.org/10.1093/jnci/djy222

6. Sharova D.S., Zinchenko V.V, Akhmad E.S. et al. On the issue of ethical aspects of the artificial intelligence systems implementation in healthcare. Digital Diagnostics. 2021; 3: 356−368. https://doi.org/10.17816/DD77446 (In Russian)

7. Morozov S.P., Vladzimirskiy А.V., Gombolevskiy V.А. et al. Artificial intelligence in lung cancer screening: assessment of the diagnostic accuracy of the algorithm analyzing low-dose computed tomography. Tuberculosis and Lung Diseases. 2020; 8: 24−31. https://doi.org/10.21292/2075-1230-2020-98-8-24-31 (In Russian)

8. Vasilev Yu.A., Vladzimirskij A.V., Sharova D.E.. Klinicheskie ispytaniya sistem iskusstvennogo intellekta (luchevaya diagnostika). In: Luchshie praktiki luchevoj i instrumental'noj diagnostiki. 2-e izd., pererab. i dop. M.: “Research and Practical Clinical Center for Diagnostics and Telemedicine Technologies of the Moscow Health Care Department”, 2023. 40 p. (In Russian)

9. Park S.H., Goo J.M., Jo C.H. Receiver Operating Characteristic (ROC) Curve: Practical Review for Radiologists. Korean J. Radiol. 2004; 5 (1): 11−18. https://doi.org/10.3348/kjr.2004.5.1.11

10. Junge M.R., Dettori J.R. ROC Solid: Receiver Operator Characteristic (ROC) Curves as a Foundation for Better Diagnostic Tests. Global Spine J. 2024; 8 (4): 424–429. https://doi.org/10.1177/2192568218778294

11. Miroshnichenko I.I. Monitoring of the Antipsychotic Therapy Effectiveness. Psychiatry. 2022; 20: 128–138. https://doi.org/10.30629/2618-6667-2022-20-4-128-138 (In Russian)

12. Davis J., Goadrich M. The relationship between Precision-Recall and ROC curves. Proceedings of the 23rd International Conference on Machine Learning. 2006; 6: 233−240. https://doi.org/10.1145/1143844.1143874

13. Solodkiy V.A., Kaprin A.D., Nudnov N.V. et al. Сontemporary Medical Decision Support Systems Based on Artificial Intelligence for the Analysis of Digital Mammographic Images. Journal of radiology and nuclear medicine. 2023; 2: 151−162. https://doi.org/10.20862/0042-4676-2023-104-2-151-162 (In Russian)

14. Kumar R., Indrayan A. Receiver Operating Characteristic (ROC) Curve for Medical Researchers. Indian pediatr. 2011; 48: 277−287. https://doi.org/10.1007/s13312-011-0055-4

15. Fawcett T. ROC Graphs: Notes and Practical Considerations for Researchers. Machine Learning. 2004; 31: 1−38.

16. Kovalev A.A., Kuznetsov B.K., Yadchenko A.A., Ignatenko V.A. Assessment of the quality of a binary classifier in research. Health and ecology Issues. 2020; 4: 105–113. (In Russian)

17. Korneenkov A.A., Ryazantsev S.V., Vyazemskaya E.E. Calculation and interpretation of indicators of informativeness of diagnostic medical technologies. Medical Council. 2019; 20: 45−51. https://doi.org/10.21518/2079-701X-2019-20-45-51 (In Russian)

18. Electronic resource: Web tool for ROC analysis. https://roc-analysis.mosmed.ai.

19. Efron B., Tibshirani R. The Bootstrap Method for Assessing Statistical Accuracy. Behaviormetrika. 1985; 12: 1–35. https://doi.org/10.2333/bhmk.12.17_1

20. Vasilev Y.A., Tyrov I.A., Vladzymyrskyy A.V. et al. Double-reading mammograms using artificial intelligence technologies: A new model of mass preventive examination organization. Digital Diagnostics. 2023; 2: 93–104. https://doi.org/10.17816/DD321423 (In Russian)

21.