From Video Game Controllers to ORs: The Surprising Role of Gaming in Modern Medical Practices

World of Warcraft (WOW) helped Alfred Jay Iloreta, MD, become a better head and neck surgeon.

Like many current physicians (and med students), Dr. Iloreta grew up playing video games. First-person shooter games were among his favorites. He also loved complex strategy games, like WOW, a massive multiplayer online role-playing video game, and Civilization, a turn-based adventure where players lead a civilization from ancient times to the modern era through exploration, diplomacy, and conquest.

He knew those games challenged his brain, but at the time, he didn’t know that he was also building essential surgical skills like hand–eye coordination, visual–spatial perception, situational awareness, and decision-making under pressure. He also didn’t know that he’d one day use modified video game controllers to perform minimally invasive procedures—or don a virtual reality gaming headset to plan surgeries.

“The crossover skillset I got from video games is invaluable,” said Dr. Iloreta, an assistant professor of otolaryngology, neurosurgery, artificial intelligence, and human health at the Icahn School of Medicine at Mount Sinai in New York. “I’m an endoscopic skull-based surgeon; I operate nearly universally on a camera and screen, in a very complex three-dimensional space, using instruments with joystick-like controllers.”

A few decades ago (and in many homes even today), video games were considered amusements, at best. More often, they’re regarded as an unhealthy waste of time. But even as some people continue to debate the effects of video games on human minds, gaming and gaming technology have seeped into every aspect of our lives and society. We’ve come a long way from 8-bit games and the Atari 2600. Today, three-quarters of gamers are adults (USA Today. https://tinyurl.com/yc8zefk2). The average age of a gamer is 36; 29 percent of video game players are aged 50 or older (Entertainment Software Association. https://tinyurl.com/yjctstc5). Robotic, joystick-controlled surgeries are common. Artificial intelligence—a disruptive technology powered by computer chips initially created for video games—is quickly advancing both medical research and clinical practice (ENTtoday. https://www.enttoday.org/article/everything-ai-and-its-effect-on-otolaryngology/). Additional video game-fueled technologies are likely to shape the future of both otolaryngology and medical education, so physicians who ignore this evolution risk falling behind. Those who understand the attraction, power, and impact of video gaming will be able to harness it to advance the profession.

The Technology Pipeline: From Xbox to OR

The controller for the Auris Monarch interventional bronchoscopy system, for instance, is “literally an adapted Xbox controller,” said Eric Gantwerker, MD, MSc, MS, a pediatric otolaryngologist at Northwell Health’s Cohen Children’s Hospital in New York and former medical director of Level Ex, a medical video game studio.

Why? The video game industry is an extremely profitable sector. Consumers spent $57 billion on video games in 2023 (Entertainment Software Association. https://tinyurl.com/yjctstc5), with additional growth anticipated. By 2030, the global video gaming market is expected to be more than $665 billion (Fortune Business Insights, https://tinyurl.com/45cxsy9h). There are powerful financial incentives to develop engaging games and robust, intuitive, and ergonomic technology. As a result, the industry includes “the most tech-forward people in the world” who are “using the most advanced technologies, the fastest GPUs and fastest CPUs, to really push the envelope,” Dr. Gantwerker said.

Indeed, GPUs (graphic processing units) were originally developed for video games; they are responsible for the massive leap forward in gaming graphics, from chunky pixelated images to realistic, movie-quality renderings. Because GPUs are much faster and more powerful than central processing units (CPUs), they’re now being used in the digital neural networks that underpin the artificial intelligence that researchers and others are using to create AI tools (ENTtoday. https://tinyurl.com/bdeuw52d) that may soon help otolaryngologists diagnose head and neck cancers, otitis media, sinusitis, and vestibular disorders.

Medical device companies now regularly incorporate technology and designs that originated in the gaming industry as well. Physicians who grew up playing Xbox will have no trouble learning how to use the Auris Monarch interventional bronchoscopy system because they’re already familiar with the feel and layout of the controller. Leveraging proven, familiar tools and interfaces into novel medical devices can decrease training time and may even shrink reaction times, giving physicians extra time to manage unexpected challenges (Medical Product Outsourcing. https://tinyurl.com/fa6ks5er).

Current learners seek immediate feedback. Medical education based on 21st century adult learning principles needs to be interactive and engaging, challenging, stimulating, and directly applicable to practice. Those requirements can be provided by a gaming-type environment. —Jeffrey P. Simons, MD, MMM

Today’s gamers increasingly play in interactive digital spaces, collaborating in real time to accomplish a shared goal. The same headsets, high-speed internet connections, and immersive technologies that enable coordinated gameplay across continents are also laying the groundwork for remote collaboration in medicine, where teams separated by oceans might not just communicate, but eventually operate together in real time. Already, surgeons and clinicians in Hong Kong and Switzerland have demonstrated the feasibility of teleoperated robotic surgery, with a surgeon in Switzerland using an off-the-shelf joystick controller (PlayStation 3 Move Navigation Controller, Sony) to perform an in vivo gastroscopy on a pig in Hong Kong (Adv Intell Syst. https://doi.org/10.1002/aisy.202400522).

Will otolaryngologists located in the U.S. one day lead operations in rural Africa? It’s technologically possible.

Simulation: Practice Makes Safer

Simulation, in some form, has been part of medical education for decades. But although the use—and sophistication—of simulation in medicine has grown in recent years, it’s not yet a universal part of medical training. Physicians are not yet required to demonstrate their competence in digital environments before performing complex procedures on human patients. In contrast, pilots are required to achieve a degree of proficiency in simulated environments before they are allowed to control a plane.

“When I was a Navy flight surgeon, we used static simulators to go through our different procedures for routine flight and flight emergencies before getting into an aircraft,” said Brian J. McKinnon, MD, MBA, MPH, professor of otolaryngology-head and neck surgery at the University of Texas Medical Branch. To prepare to fly different aircraft, pilots use immersive simulators that include a cockpit, controls, and a responsive digital reproduction of critical flight systems.

“Those things move like a Disney ride,” Dr. McKinnon said. “They pitch and roll and simulate movement with heads-up displays within the simulator, showing changes in altitude and lighting.”

Just as gaming technology helps prepare pilots, so too can it help otolaryngologists develop proficiency. A 2024 scoping review of virtual reality-based airway management training programs—all of them based on technology that originated in the video game industry—noted benefits of applying virtual reality to airway training, with most showing improved skill acquisition compared to traditional methods of learning (Anesth Analg. https://doi.org/10.1213/ANE.0000000000006611).

Sinus simulators and temporal bone simulators are now available to help would-be otolaryngologists develop their surgical skills and learn to mitigate challenges. If a trainee’s movements, for instance, traumatize a virtual blood vessel, they must respond to the subsequent virtual bleeding. And if they fail, they can simply restart the simulation as easily as a child can reboot a video game, trying as many times as necessary to accomplish their designated task.

Unfortunately, not all medical schools have access to high-tech simulators. Cost is a major factor; simulators are expensive. Perhaps medicine can learn from the example of the United States’ Defense Innovation Unit, which is leveraging commercial gaming technologies, including extended reality (XR) headsets and personal computers, to train pilots “better, faster, and cheaper than ever before” (Defense Innovation Unit. https://tinyurl.com/9zwh8efh).

Gamification in Medical Education: Learning via Play

Video games’ ability to retain users’ attention is legendary. And intentional. Designers create engaging environments and titrate the level of challenge and rewards in games to keep players “in game” as long as possible. Such deliberate design certainly benefits video game companies, who have a financial incentive to maintain gamers’ interests. But the same strategies can be adapted by physicians and medical educators to enhance focus, motivation, and retention in clinical learning.

“Traditional medical education modalities fail to meet the needs of modern learners,” said Jeffrey P. Simons, MD, MMM, professor of otolaryngology at the University of Pittsburgh School of Medicine and education coordinator for the Triological Society. “Current learners seek immediate feedback. Medical education based on 21st-century adult learning principles needs to be interactive and engaging, challenging, stimulating, and directly applicable to practice. Those requirements can be provided by a gaming-type environment.”

Dr. Gantwerker demonstrated the power of game-based learning with an escape room experience at the two most recent annual meetings of the American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS). The timed experience featured six stations; teams worked together to solve challenges, and the teams who completed the course most quickly were awarded first, second, and third place. The 2024 game, A Puzzling Night on Call: The Otolaryngology Escape Room Experience, was “so popular that we had to turn a lot of people away,” Dr. Gantwerker said. The 2025 annual meeting will likely include another interactive game.

A 2021 review of gamification in otolaryngology, co-authored by Zack Westenhaver, MD, Robert Africa, MD, René Zimmerer, MD, and Brian McKinnon, MD, MBA, MPH, noted both “significant progress in research focusing on the use of serious games to teach surgical and laparoscopic skills” and the fact that “the medical community has not fully embraced gamification within residency education” (Laryngoscope Investig Otolaryngol. https://doi.org/10.1002/lio2.707).

The COVID-19 pandemic spurred the AAO-HNS (and other medical societies) to move toward virtual and gamified educational experiences. “We already had some online cases with branching decision points, but they didn’t include feedback and entertainment components,” said Dr. Simons, who served as the AAO-HNS education coordinator at the time. “So, we developed a few simple early-version online games to manage challenging cases.” xSim, available through AAO-HHS’s FLEX (Focused Lifelong Education Xperience. https://www.entnet.org/education/flex/), now includes otolaryngology games to support surgical training and help physicians prepare for scenarios that may arise.