New Initiatives Aim to Improve Residency Application, Program Evaluation

Every year, applicants to otolaryngology-head and neck surgery residency programs are simply outstanding, said Mark A. Zacharek, MD, an associate professor and associate program director at the University of Michigan Health Systems Department of Otorhinolaryngology, the director of the Michigan Sinus Center and the current president of the Otolaryngology Program Director’s Organization (OPDO). “There is a continuous flow of tremendous candidates,” he added.

Dr. Zacharek has reviewed candidates for Michigan’s program for the past two years and was program director at Henry Ford Hospital in Detroit for four years. “They have 4.0 grade point averages in high school, they have scored in the 99th percentile of their board exams and by the way, they’re also Eagle Scouts and Olympic athletes,” he said. “These people are unbelievable.” Dr. Zacharek added that while not every candidate is a valedictorian or has high test scores, many applicants are in the top 10 to 20 percent of their medical school classes.

They are also competing for highly sought-after otolaryngology residency spots. For the program directors, who may sort through hundreds of applications each year for a handful of openings, it’s a complex process that carries an enormous responsibility—ensuring that its residents and, ultimately, its otolaryngologists, can do the job well. To that end, new initiatives such as a standardized residency recommendation letter form, a new residency program review process and enhanced professional standards instituted by the Accreditation Council for Graduate Medical Education (ACGME) aim to streamline the admissions process and help uphold physician excellence.

A Standardized Recommendation Form

Candidates applying to otolaryngology residency programs usually submit three or four letters of recommendation as part of their application. “A typical otolaryngology program with 250 applicants will receive nearly 1,000 letters to review,” said Anna Messner, MD, past president of OPDO, a professor of otolaryngology (head and neck surgery) and pediatrics and the otolaryngology residency program director at Stanford University School of Medicine. She has reviewed applicants to Stanford’s program for the past 11 years. “Each letter takes at least 60 to 90 seconds to read, and it takes a tremendous amount of time,” she said. “The reality is that at many programs, not all letters are read.”

“Medical education is going through a revolution right now to ensure patient safety and quality.”

—Mark A. Zacharek, MD

Even when they are read, the letters don’t always give a full picture of the candidate, Dr. Messner said. “The traditional letter of recommendation is sometimes helpful, but often it is not. They oftentimes duplicate what is in the rest of the application. Some letters will talk about a candidate’s interest in research, while others will not. We just don’t get consistent information,” she said.

To address some of these issues, OPDO introduced a standard letter of recommendation during the 2012–2013 application season, and it was used by an estimated 20 to 30 percent of all applicants. While the old format of the letters was “subjective and written in a style that is often flowery and uses superlatives,” this new standardized format “is an objective measurement of qualities in candidates,” said Dr. Zacharek.

The standardized letter uses a five-point scale to measure a candidate’s qualifications, specifically:

• Patient care;
• Medical knowledge;
• Professionalism;
• Interpersonal and communications skills;
• Procedural skills;
• Research;
• Initiative and drive;
• Commitment to the field of otolaryngology–head and neck surgery;
• Commitment to academic medicine; and
• Match potential.

It also asks for up to 10 lines of written commentary meant to address the candidate’s unique qualities that are not already covered elsewhere in the application.

OPDO is still evaluating how the new recommendation form was used, said Dr. Messner. “We are creating a survey to all program directors and chairs to see how useful it was, and to see how it can be improved,” she said.

“The public and others are demanding medical training that is outcome based and not just based on the duration of training.”

—Sugki S. Choi, MD

The Next Accreditation System

Beginning July 1, 2014, all otolaryngology residency programs in the United States will be evaluated by a new process known as the Next Accreditation System (NAS), which will replace the current residency review process. The goals set by the ACGME for the NAS are to accredit programs based on outcome measures rather than process measures that constitute the current system. The NAS will reduce the burden of accreditation by eliminating the Program Information Form (PIF) that took so long for a program to complete.

The roots of the NAS date back to 1998, when the ACGME established the Outcome Project, which served as a precursor. The Outcome Project helped determine core competencies for all residents and then made those competencies goals for residents to master. By 2009, the ACGME was working with dozens of medical specialty organizations and fully reviewing the changing health care landscape to help residents stay current and well trained.

“I think the impetus behind NAS is the public demand for a more accountable use of the GME dollars and medical education based on the competency outcomes,” said Sukgi S. Choi, MD, chair of the otolaryngology residency review committee and director of the voice clinic in the otolaryngology department at the Children’s National Medical Center in Washington, DC. “The public and others are demanding medical training that is outcome based and not just based on the duration of training.”

Once NAS begins, all graduating otolaryngology (and other medical specialty) residents will need to show competency in 16 areas, known as milestones. Such milestones are developmentally based, specialty-specific achievements that residents are expected to demonstrate in established intervals throughout their training.

“The milestone project is the next step in the movement from process-based accreditation to outcomes-based accreditation,” said John Nylen, MBA, the chief financial officer and a senior vice president at the ACGME. The American Board of Medical Specialties and the ACGME are working together with residents and resident review committees, including OPDO, to create specific language for the otolaryngology milestones. They will “allow evaluators to determine the level of growth of the resident within that competency,” said Nylen.

Twenty U.S. otolaryngology residency programs will begin using the milestones on a trial basis in 2013. When the new milestones are fully implemented, a process that is planned to begin with the 2014–2015 academic year, all otolaryngology programs, along with all residency programs in all medical specialties, will evaluate their residents—and, consequently, the training program—using these milestones.

Milestones proposed by the otolaryngology RRC include 16 objective measurements, said Dr. Zacharek. Patient care objectives include topics relating to aerodigestive tract lesions, salivary disease lesions, sleep-disordered breathing/sleep medicine, facial trauma, rhinosinusitis, chronic ear disease and pediatric otitis media. Medical knowledge areas will cover upper aerodigestive tract malignancies, hearing loss, dysphasia and dysphonia and inhalant allergy. Other milestones reflect the ACGME’s existing core competencies of interpersonal and communication skills, practice-based learning and improvement, professionalism, systems-based practice for patient safety and systems-based practice for resource utilization.

The ACGME will continue to administer resident surveys and will add faculty surveys for the core faculty beginning in 2014. Both faculty and resident surveys will ask questions related to program resources, faculty involvement in education, the evaluation process, patient safety and duty hours.

“These surveys are anonymous, and the RRC sees only aggregate data,” said Dr. Choi. “Resident surveys in particular can highlight problems that the residents in a specific program may be having, and the RRC can then help the program address these issues.” Resident surveys have typically been administered in January or February of each year, and it’s likely the faculty surveys will be distributed around the same time.

Under the NAS, all residency programs will also be evaluated yearly, rather than the current frequency of every three to five years. Increasing the frequency will make it easier to identify underperforming programs earlier. “If the annual data indicates that the program may be struggling, the ACGME will schedule a focused or diagnostic site visit, which can be done on short notice, and help the program diagnose the problem and bring them back on track.”

The NAS will also track rolling board pass rates and will administer self-study visits every 10 years for all otolaryngology residency programs.

A “Huge” Responsibility

Such changes show that “medical education is going through a revolution right now to ensure patient safety and quality,” said Dr. Zacharek. “How do we ensure that residents finish their program [being]competent for the public? It’s no small task. This process is supposed to help that happen.”

Will evaluating each resident by a new set of standards take a lot more of program directors’ time? Yes, said Dr. Zacharek. “It’s a constant issue with regards to how a program director is organizing their time,” he said. But doing a thorough evaluation is critical. “The program directors have a huge responsibility and ownership in making sure their residency programs are meeting ACGME guidelines,” he said. “The responsibility to sign off on a resident is a big deal: We’re saying they are competent and capable of taking care of the public on their own. The priority is to train the most competent otolaryngologists who will focus on patient safety and who will always seek ways to improve the care they provide.”

Standard Recommendation Letter Measurements

The new standardized recommendation form introduced by the Otolaryngology Program Director’s Organization uses a five-point scale to measure a candidate on:

• Patient care
• Medical knowledge
• Professionalism
• Interpersional and communication skills
• Procedural skills
• Research
• Initiative and drive
• Commitment to the field of otolaryngology-head and neck surgery
• Commitement to academic medicine
• Match potential