AI in Otolaryngology Journals: Many Policies, Little Consensus

This commentary could have been written, edited, or even conceived of by a large language model (LLM). The extent of the LLM’s contributions that would require disclosure, however, varies drastically depending on which otolaryngology journal received it. The lack of consensus among top otolaryngology journals reflects how quickly LLMs have outpaced the policies meant to govern them. Publicly accessible LLMs can produce images, clinical trial protocols, research study designs, and text nearly indistinguishable from human writing. They also enhance clarity and efficiency in scientific communication, making them increasingly valuable for research and publication. Important questions remain regarding the reliability, reproducibility, and ethical use of these tools, however.

Key issues include the accuracy of AI-generated information, the validity of its reasoning, and its ability to produce original, relevant content. Mirzadeh et al. demonstrated that LLMs do not engage in true logical reasoning but instead replicate reasoning patterns from their training data (arXiv. https://doi.org/10.48550/arXiv.2410.05229). This raises the risk of unintentional plagiarism, the use of copyrighted material, and the amplification of biases from unreliable sources (The 16th Asian Conference on Machine Learning (Conference Track). https://tinyurl.com/4e3fh2e6).

One need not reason far beyond this principle to arrive at a scenario in which patients are at risk. For example, as otolaryngologists routinely adopt operative steps from the literature, an AI-generated passage that reads authoritatively but is subtly wrong can pose a tangible threat to patient safety. In response, many academic journals have adopted policies that either restrict AI use or require disclosure. As these frameworks evolve, there is a growing need for clear and consistent guidance, especially in fields such as otolaryngology, where research findings often inform clinical decisions.

To gauge discrepancies across guidelines, we reviewed the posted AI policies of the 20 highest-impact otolaryngology journals using Google Scholar Metrics (2019–2023) based on h5-index and h5-median, noting rules on disclosure and use. What we found was an incongruous set of standards that did not appreciably follow any guiding principles.

AI as an Author

Authorship was the lone dimension we analyzed in which we found unanimity. Specifically, all otolaryngology journals reviewed prohibit attributing authorship to LLMs or citing LLMs as sources. This aligns with Lund and Naheem’s broader study of 300 academic journals, which demonstrated a universal consensus against LLM authorship (Learned Publishing. https://doi.org/10.1002/leap.1582).

The International Committee of Medical Journal Editors defines authorship based on four criteria: significant contributions to study design, data collection, analysis, and interpretation; writing or critical review of the manuscript; final approval for publication; and responsibility for the work’s integrity (International Committee of Medical Journal Editors. https://tinyurl.com/mwnp8hdr). Since AI tools cannot meet these standards, they are excluded from authorship under current guidelines.

As LLMs become more advanced, journals may need to reconsider how AI contributions are acknowledged, particularly when they significantly influence the manuscript’s structure or wording, without equating this to authorship.

AI Use for Editing and Rephrasing

AI-assisted editing, especially for grammar and clarity, is now common and generally does not alter a manuscript’s scientific integrity. It is particularly helpful for non-native English speakers and reduces language-related barriers (Sci Ed. https://doi.org/10.36591/SE-D-4702-01).

LLMs allow the quality of an idea to be disentangled from the fluency of its expression, which, in principle, can only strengthen the literature. All journals permitted the use of AI for spelling and grammar, and most did not require disclosure for these applications. However, about one in five journals, particularly those published by Lippincott or Cambridge University Press, did require disclosure of AI use for readability. Another three of the 20 journals did not specifically address this use case. As AI editing tools become more sophisticated and offer content-related suggestions, journals may need to define more precise boundaries between acceptable editing and substantive content changes to maintain transparency and credibility.

Generative AI Use

Generative AI tools that create original text, data, or images raise additional concerns about accuracy, bias, and scientific validity. LLMs may generate hallucinated citations, fabricate or omit information, and produce incomplete summaries of scientific literature. When used for data analysis or figure generation, AI tools may apply inappropriate analytical methods, misinterpret results, or create misleading visualizations. Such errors could affect the reporting of clinical outcomes or treatment recommendations.

In this review, four of the journals advised against using generative AI for content creation. Among journals that allowed it, 12 required authors to provide detailed documentation, including input prompts, AI outputs, software version, and date of use; however, there was inconsistency in where and how this information should be reported in manuscripts.

While some may view documentation requirements, such as raw prompt/output logs, as excessive, bioethicist David Resnik, JD, PhD, emphasizes that transparency is critical to preserving scholarly integrity. He argues that AI should be treated like any other part of the methods: Authors should record when and how it was used and keep the prompts and outputs, just as they would retain raw data for an experiment (National Institute of Environmental Health Sciences. https://tinyurl.com/mvd492fw).

To promote consistency and transparency, we posit that journals should adopt a standardized approach for reporting the use of generative AI. Including AI-related disclosures in dedicated sections of the manuscript would clarify expectations for both authors and reviewers. Going to the extent that Dr. Resnik suggests, listing the raw inputs and outputs from LLMs, would introduce an onerous step that may not be warranted.

AI in Peer Review

As the volume of manuscript submissions grows, interest in using AI tools in the peer review process is increasing. Some studies suggest AI-generated feedback aligns well with that of human reviewers and may even provide valuable suggestions (NEJM AI. https://doi.org/10.1056/AIoa2400196).

The use of AI in peer review also raises concerns about data confidentiality, accountability, and the potential for misuse of unpublished material, however. The risk of premature exposure of unpublished material that may occur in uploading manuscripts to a chatbot is self-evident. A more subtle concern is the possibility that unpublished ideas may be incorporated into future AI training datasets and later reemerge without attribution to the author or context. Such a risk conflicts with the confidentiality and protection of intellectual contributions that are central to the peer review process.

Perhaps reflecting these concerns, three journals explicitly prohibited the use of AI in the peer review process, including for editing or summarizing feedback. An additional six permitted AI-assisted editing of reviewer comments but prohibited AI-generated content. The remaining journals lacked clear policies.

As AI tools become increasingly common in editorial work, establishing clear expectations and boundaries for their use during peer review will be necessary for maintaining ethical standards. And, as these technologies are further integrated into research, journal policies regarding LLM use in research publications continue to evolve.

Among the top otolaryngology journals analyzed, current policies prohibit AI from being credited as an author and generally require disclosure when generative AI is used for content creation. In contrast, the use of AI solely for editing or improving readability is widely accepted and often does not require disclosure. Policies on AI use in peer review, however, vary significantly, and many journals are still refining their guidance.

Perhaps the patchwork of policies is a microcosm of the scientific community’s uncertainty regarding how much scholarly judgment the field is willing to delegate. Considering this uncertainty, we argue that the field’s imperative is to deliberately define the boundaries of automation before the increasingly capable tools do so for us. Handled with intention, AI’s proliferation is not necessarily a threat, but an occasion to articulate the most valuable aspects of scientific discovery.    

Dr. Sturm is a resident in the Northwell Health otolaryngology program in New York. Ms. Wong and Mr. Singer are medical students at the Zucker School of Medicine at Hofstra/Northwell in Hempstead, N.Y. Dr. Gantwerker is a pediatric otolaryngologist at Cohen Children’s Medical Center, Northwell Health, in New Hyde Park, N.Y.