From Video Game Controllers to ORs: The Surprising Role of Gaming in Modern Medical Practices

World of Warcraft (WOW) helped Alfred Jay Iloreta, MD, become a better head and neck surgeon.

Like many current physicians (and med students), Dr. Iloreta grew up playing video games. First-person shooter games were among his favorites. He also loved complex strategy games, like WOW, a massive multiplayer online role-playing video game, and Civilization, a turn-based adventure where players lead a civilization from ancient times to the modern era through exploration, diplomacy, and conquest.

He knew those games challenged his brain, but at the time, he didn’t know that he was also building essential surgical skills like hand–eye coordination, visual–spatial perception, situational awareness, and decision-making under pressure. He also didn’t know that he’d one day use modified video game controllers to perform minimally invasive procedures—or don a virtual reality gaming headset to plan surgeries.

“The crossover skillset I got from video games is invaluable,” said Dr. Iloreta, an assistant professor of otolaryngology, neurosurgery, artificial intelligence, and human health at the Icahn School of Medicine at Mount Sinai in New York. “I’m an endoscopic skull-based surgeon; I operate nearly universally on a camera and screen, in a very complex three-dimensional space, using instruments with joystick-like controllers.”

A few decades ago (and in many homes even today), video games were considered amusements, at best. More often, they’re regarded as an unhealthy waste of time. But even as some people continue to debate the effects of video games on human minds, gaming and gaming technology have seeped into every aspect of our lives and society. We’ve come a long way from 8-bit games and the Atari 2600. Today, three-quarters of gamers are adults (USA Today. https://tinyurl.com/yc8zefk2). The average age of a gamer is 36; 29 percent of video game players are aged 50 or older (Entertainment Software Association. https://tinyurl.com/yjctstc5). Robotic, joystick-controlled surgeries are common. Artificial intelligence—a disruptive technology powered by computer chips initially created for video games—is quickly advancing both medical research and clinical practice (ENTtoday. https://www.enttoday.org/article/everything-ai-and-its-effect-on-otolaryngology/). Additional video game-fueled technologies are likely to shape the future of both otolaryngology and medical education, so physicians who ignore this evolution risk falling behind. Those who understand the attraction, power, and impact of video gaming will be able to harness it to advance the profession.

The Technology Pipeline: From Xbox to OR

The controller for the Auris Monarch interventional bronchoscopy system, for instance, is “literally an adapted Xbox controller,” said Eric Gantwerker, MD, MSc, MS, a pediatric otolaryngologist at Northwell Health’s Cohen Children’s Hospital in New York and former medical director of Level Ex, a medical video game studio.

Why? The video game industry is an extremely profitable sector. Consumers spent $57 billion on video games in 2023 (Entertainment Software Association. https://tinyurl.com/yjctstc5), with additional growth anticipated. By 2030, the global video gaming market is expected to be more than $665 billion (Fortune Business Insights, https://tinyurl.com/45cxsy9h). There are powerful financial incentives to develop engaging games and robust, intuitive, and ergonomic technology. As a result, the industry includes “the most tech-forward people in the world” who are “using the most advanced technologies, the fastest GPUs and fastest CPUs, to really push the envelope,” Dr. Gantwerker said.

Indeed, GPUs (graphic processing units) were originally developed for video games; they are responsible for the massive leap forward in gaming graphics, from chunky pixelated images to realistic, movie-quality renderings. Because GPUs are much faster and more powerful than central processing units (CPUs), they’re now being used in the digital neural networks that underpin the artificial intelligence that researchers and others are using to create AI tools (ENTtoday. https://tinyurl.com/bdeuw52d) that may soon help otolaryngologists diagnose head and neck cancers, otitis media, sinusitis, and vestibular disorders.

Medical device companies now regularly incorporate technology and designs that originated in the gaming industry as well. Physicians who grew up playing Xbox will have no trouble learning how to use the Auris Monarch interventional bronchoscopy system because they’re already familiar with the feel and layout of the controller. Leveraging proven, familiar tools and interfaces into novel medical devices can decrease training time and may even shrink reaction times, giving physicians extra time to manage unexpected challenges (Medical Product Outsourcing. https://tinyurl.com/fa6ks5er).

Current learners seek immediate feedback. Medical education based on 21st century adult learning principles needs to be interactive and engaging, challenging, stimulating, and directly applicable to practice. Those requirements can be provided by a gaming-type environment. —Jeffrey P. Simons, MD, MMM

Today’s gamers increasingly play in interactive digital spaces, collaborating in real time to accomplish a shared goal. The same headsets, high-speed internet connections, and immersive technologies that enable coordinated gameplay across continents are also laying the groundwork for remote collaboration in medicine, where teams separated by oceans might not just communicate, but eventually operate together in real time. Already, surgeons and clinicians in Hong Kong and Switzerland have demonstrated the feasibility of teleoperated robotic surgery, with a surgeon in Switzerland using an off-the-shelf joystick controller (PlayStation 3 Move Navigation Controller, Sony) to perform an in vivo gastroscopy on a pig in Hong Kong (Adv Intell Syst. https://doi.org/10.1002/aisy.202400522).

Will otolaryngologists located in the U.S. one day lead operations in rural Africa? It’s technologically possible.

Simulation: Practice Makes Safer

Simulation, in some form, has been part of medical education for decades. But although the use—and sophistication—of simulation in medicine has grown in recent years, it’s not yet a universal part of medical training. Physicians are not yet required to demonstrate their competence in digital environments before performing complex procedures on human patients. In contrast, pilots are required to achieve a degree of proficiency in simulated environments before they are allowed to control a plane.

“When I was a Navy flight surgeon, we used static simulators to go through our different procedures for routine flight and flight emergencies before getting into an aircraft,” said Brian J. McKinnon, MD, MBA, MPH, professor of otolaryngology-head and neck surgery at the University of Texas Medical Branch. To prepare to fly different aircraft, pilots use immersive simulators that include a cockpit, controls, and a responsive digital reproduction of critical flight systems.

“Those things move like a Disney ride,” Dr. McKinnon said. “They pitch and roll and simulate movement with heads-up displays within the simulator, showing changes in altitude and lighting.”

Just as gaming technology helps prepare pilots, so too can it help otolaryngologists develop proficiency. A 2024 scoping review of virtual reality-based airway management training programs—all of them based on technology that originated in the video game industry—noted benefits of applying virtual reality to airway training, with most showing improved skill acquisition compared to traditional methods of learning (Anesth Analg. https://doi.org/10.1213/ANE.0000000000006611).

Sinus simulators and temporal bone simulators are now available to help would-be otolaryngologists develop their surgical skills and learn to mitigate challenges. If a trainee’s movements, for instance, traumatize a virtual blood vessel, they must respond to the subsequent virtual bleeding. And if they fail, they can simply restart the simulation as easily as a child can reboot a video game, trying as many times as necessary to accomplish their designated task.

Unfortunately, not all medical schools have access to high-tech simulators. Cost is a major factor; simulators are expensive. Perhaps medicine can learn from the example of the United States’ Defense Innovation Unit, which is leveraging commercial gaming technologies, including extended reality (XR) headsets and personal computers, to train pilots “better, faster, and cheaper than ever before” (Defense Innovation Unit. https://tinyurl.com/9zwh8efh).

Gamification in Medical Education: Learning via Play

Video games’ ability to retain users’ attention is legendary. And intentional. Designers create engaging environments and titrate the level of challenge and rewards in games to keep players “in game” as long as possible. Such deliberate design certainly benefits video game companies, who have a financial incentive to maintain gamers’ interests. But the same strategies can be adapted by physicians and medical educators to enhance focus, motivation, and retention in clinical learning.

“Traditional medical education modalities fail to meet the needs of modern learners,” said Jeffrey P. Simons, MD, MMM, professor of otolaryngology at the University of Pittsburgh School of Medicine and education coordinator for the Triological Society. “Current learners seek immediate feedback. Medical education based on 21st-century adult learning principles needs to be interactive and engaging, challenging, stimulating, and directly applicable to practice. Those requirements can be provided by a gaming-type environment.”

Dr. Gantwerker demonstrated the power of game-based learning with an escape room experience at the two most recent annual meetings of the American Academy of Otolaryngology–Head and Neck Surgery (AAO-HNS). The timed experience featured six stations; teams worked together to solve challenges, and the teams who completed the course most quickly were awarded first, second, and third place. The 2024 game, A Puzzling Night on Call: The Otolaryngology Escape Room Experience, was “so popular that we had to turn a lot of people away,” Dr. Gantwerker said. The 2025 annual meeting will likely include another interactive game.

A 2021 review of gamification in otolaryngology, co-authored by Zack Westenhaver, MD, Robert Africa, MD, René Zimmerer, MD, and Brian McKinnon, MD, MBA, MPH, noted both “significant progress in research focusing on the use of serious games to teach surgical and laparoscopic skills” and the fact that “the medical community has not fully embraced gamification within residency education” (Laryngoscope Investig Otolaryngol. https://doi.org/10.1002/lio2.707).

The COVID-19 pandemic spurred the AAO-HNS (and other medical societies) to move toward virtual and gamified educational experiences. “We already had some online cases with branching decision points, but they didn’t include feedback and entertainment components,” said Dr. Simons, who served as the AAO-HNS education coordinator at the time. “So, we developed a few simple early-version online games to manage challenging cases.” xSim, available through AAO-HHS’s FLEX (Focused Lifelong Education Xperience. https://www.entnet.org/education/flex/), now includes otolaryngology games to support surgical training and help physicians prepare for scenarios that may arise.

Transferable Skills: From Console to Clinic

Playing video games 30 minutes per day for five days may improve medical residents’ fiberoptic intubation skills—and, perhaps, other essential skills.

That’s the perhaps surprising conclusion of a 2019 research study that divided 36 junior anesthesiologists—none of whom had ever performed fiberoptic intubation (FOI)—into two groups. Both groups received similar instructions. Members of the experimental group also played a video game for 30 minutes per day for five days; members of the control group had no video game experience. All residents then performed their first nasal FOI on an anesthetized surgical patient, under supervision. Intubation times were shorter, and first attempts were more likely to be successful for the experimental group compared to the control (Anaesth Crit Care Pain Med. https://doi.org/10.1016/j.accpm.2018.11.012).

Of course, that’s just one small study. A 2021 meta-analysis of 16 studies, though, appears to confirm the utility of video gaming in the development of surgical skills, noting that “a history of gaming and video-game-based training were associated with improved metrics in robotic surgery and laparoscopy” (Surgery. https://doi.org/10.1016/j.surg.2020.11.034).

Video gaming can also foster non-technical skills that are critical in surgical settings. “Most people don’t realize how much situational awareness comes from playing video games with a massive playing field. Somebody who’s playing World of Warcraft or Modern Warfare knows everything that’s going on everywhere. It is tremendous situational awareness that we wish we could teach in healthcare,” Dr. Iloreta said.

Success in multi-player games requires cooperation, communication, leadership, and strategic decision making, and those skills also translate to medical practice. As noted in a 2024 study published in Human Resource Development International, gamers develop conflict resolution, coaching, self-confidence, and problem-solving skills through play, and these skills “spill over to and enrich the work domain” (Hum Resour Dev Int. https://doi.org/10.1080/13678868.2024.2404818).

Clinical Applications: Visualization, Planning, and Patient Involvement

Before a recent surgery, Dr. Iloreta “flew” around a 3-D image of his patient’s brain, carefully plotting his surgical path. He noted any potential obstacles as well as possible workarounds. He created backup plans—a Plan B, C, and even a Plan D—so he could quickly shift his approach during surgery, if needed, instead of wasting valuable surgical time. On this particular day, he used a computer screen and mouse to manipulate and view his patient’s brain. Other days, he dons an Oculus Rift—a now-discontinued virtual reality headset usually used for video gaming—to prepare for surgery.

“With this technology, I can really explore the target pathology and plan my trajectory. Because I’ve modeled the surgery in advance, I can move much faster—and safer—through that case,” Dr. Iloreta said.

While these tools are not yet available to all otolaryngologists, Dr. Iloreta believes they have the potential to improve both clinical outcomes and physician quality of life. The system he uses, for instance, automatically coalesces 2-D CT and MRI images into a 3-D image, allowing him to devote more mental energy to surgical planning.

“Our cognitive capacity is not infinite. When you use AI tools, you can preserve and optimize the fuel in your tank so you can be a better surgeon and better human,” Dr. Iloreta said.

Video game-based technology can also facilitate patient care. Dr. Iloreta uses some of the same technology he uses to plan surgery to educate and empower his patients. “Instead of them seeing a CT scan, they see an entire representation they can recognize. It’s almost a digital twin,” he said, noting that it’s much easier for patients to understand their pathology and treatment options when they can see where a tumor is located or how their sinuses are obstructed.

Most people don’t realize how much situational awareness comes from playing video games with a massive playing field. Somebody who’s playing World of Warcraft or Modern Warfare knows everything that’s going on everywhere. It is tremendous situational awareness that we wish we could teach in healthcare. —Alfred Jay Iloreta, MD

Research by Dr. Iloreta and others also suggests that video games and virtual reality can effectively distract patients and reduce pain during medical procedures. A randomized controlled trial published in Laryngoscope demonstrated that virtual reality significantly reduced pain in pediatric patients undergoing in-office nasal endoscopies, without disrupting procedural efficiency or workflow (Laryngoscope. https://doi.org/10.1002/lary.29148).

The Road Ahead: Barriers and Breakthroughs

Cost is perhaps the most significant barrier to the broader integration of video game technology in otolaryngology. Developing sophisticated, high-quality simulation and gamification modules requires significant investment, and the otolaryngology market is small compared to other medical specialties.

“It’s a challenging business model,” Dr. Simons said.

Hospitals and healthcare systems are also reluctant to invest in expensive tools without a proven return on investment. “They want data to show that this makes surgery faster or safer, but getting that data is fairly hard because of the complexity and heterogeneity of our surgeries,” Dr. Iloreta said. Researchers may need to use different metrics—including biometric data gathered from physicians during surgery— to make the case for high-tech tools.

“When we can show that these tools reduce burnout, improve physician quality of life, improve surgeon ergonomics, and reduce stress, I think hospitals and programs will realize they have value,” Dr. Iloreta said.

To move the field forward, otolaryngologists will need to collaborate with game designers. “They know how to use the tools and push the limits, and they have access to software and technology that we in healthcare don’t,” Dr. Gantwerker said. But game designers don’t know otolaryngology. By contributing their clinical experience and vision, physicians can help build smarter, more effective tools that not only reflect the realities of surgical practice but also shape the future of medical education.

Jennifer Fink is a freelance medical writer based in Wisconsin.