Development of Machine Learning Copilot to Assist Novices in Learning Flexible Laryngoscopy

RESULTS

Anatomical Region Classifier

The best-performing model had six frozen layers and was trained with a batch size of 128 for four epochs with a learning rate of 0.0002. This model achieved an overall accuracy of 91.9% on the validation set and 80.1% on the test set. From the confusion matrix, it was apparent that the whiteout class was most frequently incorrectly classified as the nasal cavity, where most of the whiteouts happened in the training data. Anatomical regions were most likely to be misclassified as one of their adjacent regions because the transition point from one region to another is not discrete.

Anatomical Structure Detector

The best-performing model was trained for 80 epochs using a batch size of 128 and a learning rate of 0.001. This model achieved an overall mean average precision of 0.642. Large variability was seen in the mAP across classes. With strong delineations, structures like the vocal folds and epiglottis likely had more consistency and less noise in their bounding box labeling and were therefore easier for the model to learn.

AI Copilot Computer Performance

Initially, the AI Copilot was designed to run on a computer with a graphics processing unit (GPU), but we felt this limited the potential environments in which this might eventually be used. To make the software more broadly usable, we focused on making it work on a MacBook Pro M1. Through various optimizations, we were able to run the AI Copilot at approximately 28 frames per second (FPS), which is imperceptible from real-time and nearly matches the video frame rate of 30 FPS.

Ninety point nine percent of medical students naïve to FFL strongly agreed/agreed that the AI Copilot was easy to use. Medical students’ self-rating of FFL skills following use of AI Copilot, however, were equivocal overall compared to their self-rating without the Copilot.

CONCLUSIONS

We described the development and pilot testing of the first AI Copilot to help train novices to competently perform FFL on a manikin. The AI Copilot tracked successful capture of diagnosable views of key anatomical structures, effectively guiding users through FFL to ensure that all anatomical structures are sufficiently captured. This tool has the potential to assist novices in efficiently gaining competence in FFL.