Automated Technical Skill and Performance Assessment in Otology and Neurotology: A Scoping Review

CLINICAL QUESTION

What is the current state of semiautomated and fully automated methods for assessing technical skill and performance in otologic and neurotologic surgery?

BOTTOM LINE

Automated technical skill assessment in otology and neurotology is advancing, particularly for mastoidectomy in simulation settings, but real-time surgical applications and broader procedural validation remain limited and are key areas for future research.

BACKGROUND: Competency-based training has increased the demand for objective, efficient performance assessments in surgical education. Traditional methods are time-consuming and often subjective, particularly in otology, where procedures like temporal bone surgery are complex. Automated assessment offers a promising solution, yet a comprehensive review of its applications in otology is lacking.

STUDY DESIGN: Scoping review of studies published between 2007 and 2024 that reported semi-automated or fully automated methods of performance assessment in otology/ neurotology. Databases searched included PubMed, EMBASE, Web of Science, and IEEE Xplore, following PRISMA-ScR guidelines.

SETTING: Most studies were conducted in virtual reality (VR)-simulated mastoidectomy environments, with a few involving cadaveric or 3D-printed models. Only one study assessed live surgery performance, and AI use was limited.

SYNOPSIS: From 1,141 screened articles, 21 met the inclusion criteria. The majority (20/21) addressed mastoidectomy, with only one focused on myringotomy. Most (n=12) were VR-based simulations, while others used cadaveric (n=1), 3D-printed (n=1), or live surgery (n=4) settings. Participant numbers ranged from 4 to 74, most often residents. Assessment methods fell into motion analysis (e.g., instrument kinematics, drill force) and final product analysis (e.g., amount of bone removed, exposure of structures). Thirteen studies reported fully automated methods, mainly VR-derived; eight required semiautomated analysis such as post-hoc video review or sensor data processing. AI was underutilized, appearing in only a handful of studies, typically for machine learning classification or regression. Most studies demonstrated internal validity—showing feasibility or differentiation across experience levels—but few offered external validity by comparison with expert assessments. Limitations included heterogeneity in study design, small participant samples, and overrepresentation of iterative work by certain groups. While automated systems offer promise, their clinical translation is limited, particularly beyond mastoidectomy. Future work must focus on developing AI-driven, externally validated, real-time assessment systems that span a broader range of otologic and neurotologic procedures.

CITATION: Nwosu OI, et al. Automated technical skill and performance assessment in otology and neurotology: a scoping review. Otol Neurotol. 2025;46:248-255. doi:10.1097/ MAO.0000000000004427

COMMENT: This study was a scoping review of semi-automated and fully automated methods for technical skill assessment in otology and neurotology procedures. Due to current technical limitations of computer vision and its clinical applications, we are limited to mostly performance in VR-simulated mastoidectomy, and, as with many of these technology-based studies, there is limited external validation and risk of bias. This highlights a critical need for research in automated assessment of surgical skills, especially during non-video-based (endoscopic or microscopic) surgery. —Eric Gantwerker, MD