How Crowdsourcing Is Maximizing the Impact of Medical Research

The Delphi Method Steps into the 21st Century

Dr. Balakrishnan also conducts studies using the Delphi Method, a research process developed in the 1950s to predict how technology might impact war. The method uses groups of experts to anonymously answer surveys, gather statistics on the group response, and reduce the range of responses to arrive at a consensus when searching for answers to complex issues.

An example of the use of this method is a situation in which there was no data to make any kind of conclusion or recommendation. In real time, during the Cold War, researchers wanted to know the likelihood that Russia would launch a missile attack and determine how experts would predict this. “It’s hard to even think about what factors you would put together to reach some sort of conclusion,” said Dr. Balakrishnan, adding that the idea is that in the absence of data you can use expert opinion and expert experience. The beauty of the Delphi Method, he added, is that it corrects for the fact that if you ask 20 experts about something, you might get 20 different answers, and everyone thinks they are right. “And that’s certainly true in medicine,” he said. “We all like to think we’re right.”

The Delphi Method has been used worldwide for more than half a century, but advances in technology mean that, rather than gathering experts in a room at a conference and handing out paper surveys, it can now all be done online, making use of experts around the world. Dr. Balakrishnan used it in a study he conducted on pediatric laryngotracheal reconstruction (Laryngoscope. Published August 27, 2018 online ahead of print. doi: 10.1002/lary.27445), as well as in a Mayo Clinic-led study on pediatric aerodigestive programs he co-authored (Pediatrics. 2018;141(3). pii: e20171701), and he has several other similar studies in progress. He added that, while he has probably used the Delphi Method more than many otolaryngology investigators, it is gaining in popularity among researchers in the specialty.

Erynne A. Faucett, MD, a pediatric otolaryngologist for the division of otolaryngology at Phoenix Children’s Hospital in Arizona, said that she likes the fact that the Delphi Method, in contrast to other data gathering techniques, uses a multiple-iteration design to develop an agreement. “This feedback process may encourage participants to reassess their initial judgments about the information provided in previous versions,” she said, adding that it can prevent group pressure for conformity and the influence of dominant personalities, as well as allowing respondents to remain anonymous, if needed.

Dr. Faucett, whose research has been in educating medical students and residents, was introduced to crowdsourcing via the Delphi Method in fellowship when she and her colleagues were trying to figure out how to create an assessment tool for particular surgical procedures in pediatric otolaryngology. “We used the Delphi Method to bring experts in our field … to a consensus regarding creating assessment tools for particular surgical procedures,” she said.

She has also recently been involved in a project supported by the American Academy of Otolaryngology–Women in Otolaryngology Endowment grant, which involved a modified Delphi consensus study to identify high priority areas and associated gaps in current leadership and professional development resources for women in otolaryngology and surgery. “To increase our likelihood for successful identification of these priority areas, we utilized a Delphi consensus approach, which has been seen more in health care to achieve agreement/consensus among recognized experts on particular topics,” she said. With the topics identified via this method, Dr. Faucett and her colleagues will introduce a coaching curriculum to address these high priority needs and create an educational audio podcast, along with associated print materials.