SMU Data Science Review

Analyzing Empirical Quality Metrics of Deep Learning Models for Antimicrobial Resistance

Huy H. Nguyen, Southern Methodist UniversityFollow
Sanjay Pillay, Southern Methodist UniversityFollow
Allison Roderick, Southern Methodist UniversityFollow
Hao Wang, Southern Methodist UniversityFollow
John Santerre, SMUFollow

Abstract

Antimicrobial Resistance (AMR) is a growing concern in the medical field. Over-prescription of antibiotics as well as bacterial mutations have caused some once lifesaving drugs to become ineffective against bacteria. However, the problem of AMR might be addressed using Machine Learning (ML) thanks to increased availability of genomic data and large computing resources. The Pathosystems Resource Integration Center (PATRIC) has genomic data of various bacterial genera with sample isolates that are either resistant or susceptible to certain antibiotics. Past research has used this database to use ML algorithms to model AMR with successful results, including accuracies over 80%. To better aid future biologists and healthcare workers who may need a predictive model without the benefit of thousands of bacteria samples, this paper explores quantifying the empirical quality of some machine learning models—that is, quantifying how well a model will perform without prior knowledge of how the model performed on a training dataset. WeightWatcher is a Python package that offers various algorithms to measure model quality. This research uses the empirical quality metrics that WeightWatcher introduces for Deep Neural Network (DNN) models to evaluate AMR models, even on datasets based on small sample sizes of bacterial strains. The use of ML in AMR and pharmacogenetic research can help lead to increased efficacy of antibiotic treatments by predicting whether a strain of bacteria will be resistant or susceptible to an antibiotic.

Recommended Citation

Nguyen, Huy H.; Pillay, Sanjay; Roderick, Allison; Wang, Hao; and Santerre, John (2021) "Analyzing Empirical Quality Metrics of Deep Learning Models for Antimicrobial Resistance," SMU Data Science Review: Vol. 5: No. 1, Article 10.
Available at: https://scholar.smu.edu/datasciencereview/vol5/iss1/10