Study Design Flaws Can Create Result Interpretation Problems

by Amy E. Hamaker •

What are the most common pitfalls observed in scientific research derived from national cancer registries, predominantly the Surveillance, Epidemiology, and End Results (SEER) Program and the National Cancer Database (NCDB)?

Bottom line: Pitfalls were identified in two major areas: design and data analysis. These inherent design flaws can raise considerable problems with interpretation.

Comment: This is a timely article that combined a literature review with structured interviews with journal editors to identify common pitfalls in database research in head and neck oncology. The authors identified design, analysis, and statistical shortcomings that were common among database studies submitted for peer review. This article is a valuable read for otolaryngologists who find database research challenging to interpret and sheds light on some of the design and statistical nuances that limit actionable conclusions in this type of research. —Andres Bur, MD

BACKGROUND: National cancer registries allow researchers to analyze large cohorts of populations with malignancies to examine patterns of cancer care delivery and outcomes. This can be valuable in our understanding of cancer epidemiology and biology, and its treatment. The vast amount of data, however, also sets the stage for flawed design, analysis, and statistics. 

STUDY DESIGN: Literature review.

SETTING: PubMed, The National Cancer Institute’s SEER website, and the American College of Surgeons’ NCDB website.

SYNOPSIS: An a priori plan documentation through pre-data analyses is often evidence of a well-thought-out study; without it, false or meaningless associations may arise. To avoid stretching associations, defined hypotheses and appropriate statistical modeling should be established before analyzing data. Pitfalls of study design can occur in selecting patient cohorts (e.g., unbalanced demographics, multiple primaries from the same individual, cases diagnosed only at autopsy), survival analysis (e.g., uncontrolled tumor or treatment factors, treatment imbalance), and bias introduction from tumor biology. There are multiple strategies to address these problems, including stringent, detailed inclusion criteria, matched analysis, propensity score–based weights, Bonferroni correction, and regression adjustment. An additional flaw is not being selective in the pathology groupings for survival analysis. For example, some studies that evaluate head and neck squamous cell carcinoma do not control for the effects of human papillomavirus (HPV) oropharynx cancer. Cases missing key variables needed for analysis should be dropped from the study so the cohort is uniform. Inadequate database capture of chemotherapy, radiation, or multiple treatments can skew treatment outcomes and treatment delay implications.

CITATION: Jones EA, Shuman AG, Egleston BL, et al. Common pitfalls of head and neck research using cancer registries. Otolaryngol Head Neck Surg. 2019;161:245–250.