Evaluating AI Performance in COVID-19 Diagnosis: A Systematic Review

Document Type : Systematic Review

Authors
Azadeh Abkar 1
Mahdi Mehrabi 2
Amin Golabpour 3
Mohammad Amin Shayegan 4
1 PhD student,, Department of computer engineering, Shiraz Branch,Islamic Azad University, Shiraz, Iran
2 Assistant Professor,, Department of computer engineering, Shiraz Branch, Islamic Azad University, Shiraz, Iran
3 Assistant Professor, Department of Health Information Technology, Shahrood University of Medical Sciences, Shahrood, Iran
4 Associate Professor, Department of Computer Science, Shiraz Branch, Islamic Azad University, Shiraz, Iran
10.48305/him.2025.43286.1247
Abstract
Introduction: Early detection of COVID-19 plays a crucial role in reducing mortality. With its ability to analyze large volumes of medical data, artificial intelligence (AI) can aid in the rapid and accurate diagnosis of this disease. This review study aims to assess the performance of AI models in detecting COVID-19.
Methods: A systematic search was conducted in the PubMed, Scopus, and WOS databases, yielding 143 relevant articles. The studies were analyzed based on the type of input data, algorithms used, performance evaluation metrics, the presentation of clinical rules, and the clinical validation of the models. Additionally, 451 review and meta-analysis articles were examined to determine the extent to which these algorithms have been integrated into clinical guidelines.
Results: The input data in the articles consisted of clinical records, medical images, and audio processing. More than 96% of the algorithms were found to be black-box models, lacking clinical validation by specialists. However, the average performance of these models was reported to be above 90%. The review of meta-analysis articles revealed that none of the algorithms had undergone formal clinical evaluation; only their performance on available data was assessed.
Conclusion: The review of studies shows that AI performs well in COVID-19 detection but faces limitations, including lack of explainability, reliance on training data, and the absence of extensive clinical evaluations. The development of white-box models and conducting broad clinical studies are crucial for ensuring the widespread acceptance of these algorithms in clinical settings.

Keywords

Subjects

  1. 1.Stokes K, Castaldo R, Federici C, Pagliara S, Maccaro A, Cappuccio F, et al. The use of artificial intelligence systems in diagnosis of pneumonia via signs and symptoms: A systematic review. Biomedical Signal Processing and Control. 2022;72:103325 .

    1. Huang S, Yang J, Fong S, Zhao Q. Artificial intelligence in the diagnosis of COVID-19: challenges and perspectives. Int J Biol Sci.2021;17(6):1581-7 .
    2. Vaishya R, Javaid M, Khan IH, Haleem A. Artificial Intelligence (AI) applications for COVID-19 pandemic. Diabetes Metab Syndr. 2020;14(4):337-9.
    3. Ahouz F, Golabpour A. Predicting the incidence of COVID-19 using data mining. BMC Public Health. 2021;21(1):1087 .
    4. Azodi CB, Tang J, Shiu SH. Opening the Black Box: Interpretable Machine Learning for Geneticists. Trends Genet. 2020;36(6):442-55 .
    5. Loyola-Gonzalez O. Black-box vs. white-box: Understanding their advantages and weaknesses from a practical point of view. IEEE access.2019;7:154096-113 .
    6. Khan PF, Meehan K, editors. Diabetes prognosis using white-box machine learning framework for interpretability of results. 2021 IEEE 11th Annual Computing and Communication Workshop and Conference (CCWC); 2021: IEEE.
    7. Wang S, Wang Y, Wang D, Yin Y, Wang Y, Jin Y. An improved random forest-based rule extraction method for breast cancer diagnosis.Applied Soft Computing. 2020;86 .
    8. Pipino LL, Lee YW, Wang RY. Data quality assessment. Communications of the ACM. 2002;45(4):211-8 .

    10 . Weinstein S, Obuchowski NA, Lieber ML. Clinical evaluation of diagnostic tests. American Journal of Roentgenology. 2005;184(1):14-9 .

    11 . Shakir MA, Abass HK, Jelwy OF, Al-Bayati HNA, Salman SM, Mikhav V, et al., editors. Developing Interpretable Models for Complex Decision-Making. 2024 36th Conference of Open Innovations Association (FRUCT); 2024 30 Oct.-1 Nov. 2024.

    12 . Qiao H, Chen Y, Qian C, Guo Y. Clinical data mining: challenges, opportunities, and recommendations for translational applications. Journal of Translational Medicine. 2024;22(1):185 .

    13 . Connor Gorber S, Singh H, Pottie K, Jaramillo A, Tonelli M. Process for guideline development by the reconstituted Canadian Task Force on Preventive Health Care. CMAJ. 2012;184(14):1575-81 .

    14 . Yu L, Shi X, Liu X, Jin W, Jia X, Xi S, et al. Artificial Intelligence Systems for Diagnosis and Clinical Classification of COVID-19. Front Microbiol. 2021;12:729455 .

    15 . Sekeroglu B, Ozsahin I. Detection of COVID-19 from Chest X-Ray Images Using Convolutional Neural Networks. SLAS Technol.2020;25(6):553-65 .

    16 . Giannotti F, Naretto F, Bodria F. Explainable for Trustworthy AI. In: Chetouani M, Dignum V, Lukowicz P, Sierra C, editors. Human-Centered Artificial Intelligence: Advanced Lectures. Cham: Springer International Publishing; 2023. p. 175-95 .

    17 . Ali S, Abuhmed T, El-Sappagh S, Muhammad K, Alonso-Moral JM, Confalonieri R, et al. Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence. Information Fusion. 2023;99:101805.

    18 . Vernon EM, Masuyama N, Nojima Y, editors. Integrating White and Black Box Techniques for Interpretable Machine Learning. Proceedings of Ninth International Congress on Information and Communication Technology; 2024 2024//; Singapore: Springer Nature Singapore .

    19 . Caniço A, Santos A. Witter: A Library for White-Box Testing of Introductory Programming Algorithms2023. 69-74 p