There’s Room for Improvement in Machine Learning Publication Standards within Otolaryngology

by Linda Kossoff •

What is the status of existing reporting guidelines for machine learning (ML) in biomedical publications and present recommendations for their use in otolaryngology journal reports?

BOTTOM LINE

There is limited consensus and discussion regarding existing reporting guidelines for ML in biomedical publications and a relative lack of expertise in ML methodology in peer reviewers.

BACKGROUND: ML, a subfield of artificial intelligence (AI) pertaining to the training of predictive models based on historical data, is increasingly featured in publications in otolaryngology. Recently, there has been interest in generating frameworks to standardize ML publications to ensure their value to target readership.

STUDY DESIGN: Summary/report.

SETTING: Department of Otolaryngology–Head and Neck Surgery, Massachusetts Eye and Ear, Boston, Mass.

SYNOPSIS: Researchers observed that the reporting guideline scope for biomedical ML model publications is typically restricted to the first four of six phases of ML model development: data pre-processing, feature engineering, model development, and model validation, excluding model deployment and model maintenance phases. They note that considerable work has been done on reporting guidelines for ML application to studies pertaining to diagnostics, clinical trials, outcomes, and bias assessment risk. Reporting guideline tools for ML studies identified by researchers included a checklist for multivariable predictive models, a checklist for clinical trial protocols for inter-ventions involving AI, a checklist for clinical trial reports for interventions involving AI, a framework for minimum number of reportable domains for clinical decision support tools, and a multidimensional assessment tool for risk of bias and applicability of clinical prediction models. A notable omission from most of these tools was a lack of assessments for fairness, the possibility of bias in the training data or model, or some combination thereof. Researchers suggest that future appraisal tools include reporting guidelines commonly used in healthcare and the biomedical sciences and a bias mitigation assessment.

CITATION: Crowson MG, Rameau A. Standardizing machine learning manuscript reporting in otolaryngology–head & neck surgery. Laryngoscope. 2022;132:1698–1700.