Review of meta-analyses on the use of artificial intelligence in radiology

Introduction. Artificial intelligence (AI) is an effective tool for automating routine tasks in radiology. The diagnostic accuracy of AI in detecting various pathologies on medical images has generated considerable interest in the scientific community: the number of studies and meta-analyses has been constantly growing. The abundance of published evidence and the diversity of outcomes necessitate the need to systematize the available publications. The aim of this paper is to conduct an umbrella systematic review of contemporary meta-analyses on the use of AI in radiology.

Materials and methods. PubMed was searched for studies published in the English language. Thirty-eight systematic reviews with meta-analyses published between 2021 and 2023 were selected for full-text analysis. The extracted data included the goal, study design, imaging modality, sample size, quality assessment of the included studies, AI diagnostic accuracy estimates, reference method parameters, and clinical efficacy metrics of AI implementation. The methodological quality of included systematic reviews was assessed using the AMSTAR-2 tool.

Results. Nearly half (47%) of the included meta-analyses focused on the diagnosis, staging and segmentation of malignancies. Four meta-analyses were related to detection of maxillofacial structures in dentistry, while another four meta-analyses addressed the diagnosis of brain lesions. The diagnosis of COVID-19 and the diagnosis of bone fractures were each covered in three meta-analyses. One meta-analysis was reviewed for each of the following fields: colorectal polyps, pneumothorax, pulmonary embolism, osteoporosis, aneurysms, multiple sclerosis, acute cerebrovascular accident, intracranial hemorrhage, burns, and the risk of intrauterine growth restriction. Thirty-five (92%) meta-analyses assessed the risk of bias. Twenty-eight (80%) meta-analyses utilized QUADAS-2 to assess the risk of bias. 14 out of 28 papers reported low risk of bias (50%); 4 (14%) – moderate; 10 (36%) – high. The major risks were associated with samples that were unbalanced in terms of size and composition, a lack of details about the methods, a low number of prospective studies, and a lack of external validation of the outcomes. The overall results indicate that the diagnostic accuracy of AI is comparable to or even greater than that of radiologists. The mean sensitivity, specificity and area under the ROC curve for AI and radiologists were 85.2%, 89.5%, 93.5% and 84.4%, 90.0%, 92.8%, respectively. However, many studies that compared the diagnostic accuracy of AI and radiologists lack the data on the number and experience of the latter. Only one paper presented results of implementing AI into routine clinical diagnosis.

Discussion. AI is capable of reducing the turnaround time for non-urgent examinations. When used to verify the primary interpretation, AI was effective in detecting false-negative results from radiologists. However, the efficacy of detecting false-positive results was inadequate. Our assessment of the quality of systematic reviews with AMSTAR-2 show that the methods of searching, selecting and analyzing literature must be improved and brought to a common standard. The development of a specialized tool for assessing the quality of systematic reviews in the AI implementation is also necessary. Due to high diagnostic accuracy, AI is currently considered a promising tool for optimizing the turnaround time. However, more evidence is needed to study the AI outcomes in routine clinical practice. Furthermore, it is necessary to standardize and improve the quality of research methodology.

1. Hosny A., Parmar C., Quackenbush J. et al. Artificial intelligence in radiology. Nat. Rev. Cancer. 2018; 18 (8): 500–510. http://doi.org/10.1038/s41568-018-0016-5

2. Sogani J., Allen B. Jr., Dreyer K., McGinty G. Artificial intelligence in radiology: the ecosystem essential to improving patient care. Clin. Imaging. 2019; 59 (1): A3–A6. https://doi.org/10.1016/j.clinimag.2019.08.001

3. Keane P.A., Topol E.J. With an eye to AI and autonomous diagnosis. NPJ Digital Medicine. 2018; 1: 40. https://doi.org/10.1038/s41746-018-0048-y

4. Vladzimirskyy A.V., Gusev A.V., Sharova D.E. et al. Health information system maturity assessment methodology. Vrach i informatsionnye tekhnologii. 2022; 3: 68–86. https://doi.org/10.25881/18110193_2022_3_68 (In Russian)

5. Sharova D.E., Garbuk S.V., Vasilev Y.A. Artificial intelligence systems in clinical medicine: the world’s first series of national standards. Standards and Quality. 2023; 1: 46–51. https://doi.org/10.35400/0038-9692-2023-1-304-22 (In Russian)

6. Kelly S., Kaye S.-A., Oviedo-Trespalacios O. What factors contribute to the acceptance of artificial intelligence? A systematic review. Telematics and Informatics. 2023; 77: 101925. https://doi.org/10.1016/j.tele.2022.101925

7. Page M.J., McKenzie J.E., Bossuyt P.M. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021; 372: n71. https://doi.org/10.1136/bmj.n71

8. Shea B.J., Reeves B.C., Wells G. et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017; 358: j4008. https://doi.org/10.1136/bmj.j4008

9. Yoon J.H., Strand F., Baltzer P.A.T. et al. Standalone AI for Breast Cancer Detection at Screening Digital Mammography and Digital Breast Tomosynthesis: A Systematic Review and Meta-Analysis. Radiology. 2023; 307 (5): e222639. https://doi.org/10.1148/radiol.222639

10. Hickman S.E., Woitek R., Vi Le E.P. et al. Machine Learning for Workflow Applications in Screening Mammography: Systematic Review and Meta-Analysis. Radiology. 2022; 302: 88–104. https://doi.org/10.1148/radiol.2021210391

11. Kuo R.Y.L., Harrison C., Curran T.-A. et al. Artificial Intelligence in Fracture Detection: A Systematic Review and Meta-Analysis. Radiology. 2022. 304 (1): 50–62. https://doi.org/10.1148/radiol.211785

12. Xue Y., Zhou Y., Wang T. et al. Accuracy of Ultrasound Diagnosis of Thyroid Nodules Based on Artificial Intelligence-Assisted Diagnostic Technology: A Systematic Review and Meta-Analysis. Int J Endocrinol. 2022: 9492056. https://doi.org/10.1155/2022/9492056

13. Lex J.R., Di Michele J., Koucheki R. et al. Artificial Intelligence for Hip Fracture Detection and Outcome Prediction A Systematic Review and Meta-analysis. JAMA Network Open. 2023; 6 (3): e233391. https://doi.org/10.1001/jamanetworkopen.2023.3391

14. Gao L., Jiao T., Feng Q., Wang W. Application of artificial intelligence in diagnosis of osteoporosis using medical images: a systematic review and meta-analysis. Osteoporosis International. 2021; 32: 1279–1286. https://doi.org/10.1007/s00198-021-05887-6

15. Poly T.N., Islam M., Li Y.-C. J. et al. Application of Artificial Intelligence for Screening COVID-19 Patients Using Digital Images: Meta-analysis. JMIR Med. Inform. 2021; 9 (4): e21394. https://medinform.jmir.org/2021/4/e21394

16. Zhang X., Yang Y., Shen Y.-W. et al. Diagnostic accuracy and potential covariates of artificial intelligence for diagnosing orthopedic fractures: a systematic literature review and meta-analysis. Eur. Radiol. 2022; 32 (10): 7196–7216. https://doi.org/10.1007/s00330-022-08956-4

17. Sugibayashi T., Walston S.L., Matsumoto T. et al. Deep learning for pneumothorax diagnosis: a systematic review and meta-analysis. Eur. Respirat. Rev. 2023; 32: 220259. https://doi.org/10.1183/16000617.0259-2022

18. Thong L.T., Chou H.S., Chew H.S.J., Lau Y. Diagnostic test accuracy of artificial intelligence-based imaging for lung cancer screening: A systematic review and meta-analysis. Lung Cancer. 2023; 176: 4–13. https://doi.org/10.1016/j.lungcan.2022.12.002

19. Menon N., Guidozzi N., Chidambaram S., Markar S.R. Performance of radiomics-based artificial intelligence systems in the diagnosis and prediction of treatment response and survival in esophageal cancer: asystematic review and meta-analysis of diagnostic accuracy. Dis. Esophagus. 2023; 36 (6): doad034. https://doi.org/10.1093/dote/doad034

20. Xing W., Gao W., Lv X. et al. Artificial intelligence predicts lung cancer radiotherapy response: A meta-analysis. Artif. Intell. Med. 2023; 142: 102585 https://doi.org/10.1016/j.artmed.2023.102585

21. Zheng X., He B., Hu Y. et al. Diagnostic Accuracy of Deep Learning and Radiomics in Lung Cancer Staging: A Systematic Review and Meta-Analysis. Front. Public Health. 2022; 10: 938113. https://doi.org/10.3389/fpubh.2022.938113

22. Agarwal S., Wood D., Grzeda M. et al. Systematic Review of Artificial Intelligence for Abnormality Detection in High-volume Neuroimaging and Subgroup Meta-analysis for Intracranial Hemorrhage Detection. Clin. Neuroradiol. 2023; 33: 943–956. https://doi.org/10.1007/s00062-023-01291-1

23. Bedrikovetski S., Dudi-Venkata N.N., Kroon H.M. et al. Artificial intelligence for pre-operative lymph node staging in colorectal cancer: a systematic review and meta-analysis. BMC Cancer. 2021a; 21. https://doi.org/10.1186/s12885-021-08773-w

24. Bedrikovetski S., Dudi-Venkata N.N., Maicas G. et al. Artificial intelligence for the diagnosis of lymph node metastases in patients with abdominopelvic malignancy: A systematic review and meta-analysis. Artif. Intell. Med. 2021b; 113: 102022. https://doi.org/10.1016/j.artmed.2021.102022

25. Din M., Agarwal S., Grzeda M. et al. Detection of cerebral aneurysms using artificial intelligence: a systematic review and meta-analysis. J. NeuroIntervent. Surg. 2023; 15: 262–271. https://doi.org/10.1136/jnis-2022-019456

26. Xu H.-L., Gong T.-T., Liu F.-H. et al. Artificial intelligence performance in image-based ovarian cancer identification: A systematic review and meta-analysis. eClinicalMedicine. 2022; 53: 101662. https://doi.org/10.1016/j.eclinm.2022.101662

27. Sadr S., Mohammad-Rahimi H., Motamedian S.R. et al. Deep Learning for Detection of Periapical Radiolucent Lesions: A Systematic Review and Meta-analysis of Diagnostic Test Accuracy. J. Endodont. 2023; 49 (3): 248–261.e3. https://doi.org/10.1016/j.joen.2022.12.007

28. Komolafe T.E., Cao Y., Nguchu B.A. et al. Diagnostic Test Accuracy of Deep Learning Detection of COVID-19: A Systematic Review and Meta-Analysis. Acad. Radiol. 2021; 28 (11): 1507–1523. https://doi.org/10.1016/j.acra.2021.08.008

29. Wang Q., Ma J., Zhang L., Xie L. Diagnostic performance of corona virus disease 2019 chest computer tomography image recognition based on deep learning: Systematic review and meta-analysis. Medicine. 2022; 101: 42 (e31346). http://dx.doi.org/10.1097/MD.0000000000031346

30. Campello C.A., Castanha E.B., Vilardo M. et al. Machine learning for malignant versus benign focal liver lesions on US and CEUS: a meta‑analysis. Abdom. Radiology. 2023; 48: 3114–3126. https://doi.org/10.1007/s00261-023-03984-0

31. Dumitrescu E.A., Ungureanu B.S., Cazacu I.M. et al. Diagnostic Value of Artificial Intelligence-Assisted Endoscopic Ultrasound for Pancreatic Cancer: A Systematic Review and Meta-Analysis. Diagnostics. 2022; 12: 309. https://doi.org/10.3390/diagnostics12020309

32. Potipimpanon P., Charakorn N., Hirunwiwatkul P. A comparison of artificial intelligence versus radiologists in the diagnosis of thyroid nodules using ultrasonography: a systematic review and meta‑analysis. Eur. Arch. of Oto-Rhino-Laryngol. 2022; 279: 5363–5373. https://doi.org/10.1007/s00405-022-07436-1

33. Rescinito R., Ratti M., Payedimarri A.B., Panella M. Prediction Models for Intrauterine Growth Restriction Using Artificial Intelligence and Machine Learning: A Systematic Review and Meta-Analysis. Healthcare. 2023; 11: 1617. https://doi.org/10.3390/healthcare11111617

34. Abesi F., Jamali A.S., Zamani M. Accuracy of artificial intelligence in the detection and segmentation of oral and maxillofacial structures using cone-beam computed tomography images: a systematic review and meta-analysisn. Polish J. Radiol. 2023; 88: e256–e263. https://doi.org/10.5114/pjr.2023.127624

35. Badr F.F., Jadu F.M. Performance of artificial intelligence using oral and maxillofacial CBCT images: A systematic review and meta-analysis. Nigerian J. Clin. Pract. 2022; 25: 1918–1927. https://doi.org/10.4103/njcp.njcp_394_22

36. Soffer S., Klang E., Shimon O. et al. Deep learning for pulmonary embolism detection on computed tomography pulmonary angiogram: a systematic review and meta-analysis. Scientific Reports. 2021; 11: 15814. https://doi.org/10.1038/s41598-021-95249-3

37. Islam M., Poly T.N., Walther B.A. et al. Deep Learning for the Diagnosis of Esophageal Cancer in Endoscopic Images: A Systematic Review and Meta-Analysis. Cancers. 2022; 14 (23): 5996. https://doi.org/10.3390/cancers14235996

38. Xu Y., Ding W., Wang Y. et al. Comparison of diagnostic performance between convolutional neural networks and human endoscopists for diagnosis of colorectal polyp: A systematic review and meta-analysis. PLoS ONE. 2021; 16 (2): e0246892. https://doi.org/10.1371/journal.pone.0246892

39. Taib B.G., Karwath A., Wensley K. et al. Artificial intelligence in the management and treatment of burns: A systematic review and meta-analyses. J. Plast. Reconstructive & Aesthetic Surg. 2023; 77: 133–161. https://doi.org/10.1016/j.bjps.2022.11.049

40. Yang C., Qin L., Xie Y., Liao J. Deep learning in CT image segmentation of cervical cancer: a systematic review and meta‑analysis. Radiation Oncology. 2022; 17: 175 https://doi.org/10.1186/s13014-022-02148-6

41. Electronic resource: QUADAS-2, University of Bristol: https://www.bristol.ac.uk/population-health-sciences/projects/quadas/quadas-2/

42. Electronic resource: Radiomics Quality Score - RQS 2.0.: https://www.radiomics.world/rqs2

43. Kim H.Y., Cho S.J., Sunwoo L. et al. Classification of true progression after radiotherapy of brain metastasis on MRI using artificial intelligence: a systematic review and meta-analysis. Neuro-Oncology Advances. 2021; 3 (1): 1–12. https://doi.org/10.1093/noajnl/vdab080

44. Electronic resource: International Prospective Register of Systematic Reviews. https://www.crd.york.ac.uk/PROSPERO/

45. Adamou A., Beltsios E.T., Bania A. et al. Artificial intelligence-driven ASPECTS for the detection of early stroke changes in non-contrast CT: a systematic review and meta-analysis. J. NeuroIntervent. Surg. 2023; 15: e298–e304. https://doi.org/10.1136/jnis-2022-019447

46. Nabizadeh F., Ramezannezhad E., Kargar A. et al. Diagnostic performance of artificial intelligence in multiple sclerosis: a systematic review and meta-analysis. Neurol. Sci. 2023; 44: 499–517. https://doi.org/10.1007/s10072-022-06460-7

47. Arbabshirani M.R., Fornwalt B.K., Mongelluzzo G.J. et al. Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. NPJ Dig. Med. 2018; 1: 9. https://doi.org/10.1038/s41746-017-0015-z

48. Ginat D. Implementation of machine learning software on the radiology worklist decreases scan view delay for the detection of intracranial hemorrhage on CT. Brain Sci. 2021. 11 (7): 832. https://doi.org/10.3390/brainsci11070832

49. Salehinejad H., Kitamura J., Ditkofsky N. et al. A real-world demonstration of machine learning generalizability in the detection of intracranial hemorrhage on head computerized tomography. Sci. Reports. 2021; 11: 17051. https://doi.org/10.1038/s41598-021-95533-2

50. Vladzimirskyy A.V., Omelyanskaya O.V., Shulkin I.M. et al. MosMedData: NDKT s priznakov raka legkogo. Svidetel'stvo o gosudarstvennoi registratsii bazy dannykh № 2023621253 Rossiiskaya Federatsiya. MosMedData: NDKT s priznakami raka legkogo: № 2023620623: zayavl. 10.03.2023: opubl. 18.04.2023 / zayavitel' Gosudarstvennoe byudzhetnoe uchrezhdenie zdravookhraneniya goroda Moskvy “Nauchno-prakticheskii klinicheskii tsentr diagnostiki i telemeditsinskikh tekhnologii Departamenta zdravookhraneniya goroda Moskvy”. (In Russian)

51. Morozov S.P., Andreychenko A.E., Blokhin I.A. et al. MosMedData: data set of 1110 chest CT scans performed during the COVID-19 epidemic. Dig. Diagn. 2020; 1 (1): 49–59. https://doi.org/10.17816/DD46826. (In Russian)

52. Gombolevskiy V.A., Kharlamov K.A., Pyatnitskii I.A. et al. Shablony protokolov opisanii issledovanii po spetsial'nosti “Rentgenologiya”. Komp'yuternaya tomografiya : Metodicheskie rekomendatsii № 23 / Moskva: Gosudarstvennoe byudzhetnoe uchrezhdenie zdravookhraneniya goroda Moskvy “Nauchno-prakticheskii tsentr meditsinskoi radiologii Departamenta zdravookhraneniya goroda Moskvy”, 2016a. 31 s. (In Russian)

53. Gombolevskiy V.A., Kharlamov K.A., Pyatnitskii I.A. et al. Shablony protokolov opisanii issledovanii po spetsial'nosti “Rentgenologiya”. Magnitno-rezonansnaya tomografiya: Metodicheskie rekomendatsii № 21 / Moskva: Gosudarstvennoe byudzhetnoe uchrezhdenie zdravookhraneniya goroda Moskvy “Nauchno-prakticheskii tsentr meditsinskoi radiologii Departamenta zdravookhraneniya goroda Moskvy”, 2016b. 41 s. (In Russian)

54. Aggarwal R., Sounderajah V., Martin G. et al. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ Dig. Med. 2021; 4: 65. https://doi.org/10.1038/s41746-021-00438-z

55. Liu M., Wu J., Wang N. et al. The value of artificial intelligence in the diagnosis of lung cancer: A systematic review and meta-analysis. PLoS ONE. 2023; 18 (3): e0273445. https://doi.org/10.1371/journal.pone.0273445

56. Jørgensen M.D., Antulov R., Hess S., Lysdahlgaard S. Convolutional neural network performance compared to radiologists in detecting intracranial hemorrhage from brain computed tomography: A systematic review and meta-analysis. Eur. J. Radiol. 2022; 146: 110073. https://doi.org/10.1016/j.ejrad.2021.110073

57. Evangelista K., de Freitas Silva B.S., Yamamoto-Silva F.P. et al. Accuracy of artificial intelligence for tooth extraction decision‑making in orthodontics: a systematic review and meta‑analysis. Clin. Oral Invest. 2022; 26: 6893–6905. https://doi.org/10.1007/s00784-022-04742-0

58.