Rigorous Methodology Key to Comparative Effectiveness Research

Rigorous Methodology Key to Comparative Effectiveness Research

Although comparative effectiveness research (CER) has been conducted for decades in the health care community, it has recently begun to receive an increasing amount of funding and attention. This has resulted, in part, from the passing of the American Recovery and Reinvestment Act of 2009, which provided $1.1 billion in funding to CER.

The Saturday, May 30, Education Session “Introduction to Methods in Comparative Effectiveness Research,” chaired by Sharon H. Giordano, MD, MPH, of The University of Texas MD Anderson Cancer Center, provided an overview of a variety of approaches to CER.

“A lot more people are starting to conduct research in this area,” Dr. Giordano said. “There has been a general realization that, although it would be ideal to do a randomized clinical trial for every situation in every population, it is not realistic, and because of those gaps in knowledge, people are interested in using comparative effectiveness research to help determine the best way to treat their patients.”

The Institute of Medicine’s (IOM) definition of CER, published in 2009, is among the more commonly used, according to Dr. Giordano. IOM defines CER as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care,” and the purpose of CER is “to assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.”

With these characteristics in mind, CER can include a wide variety of trial types, such as observational studies, randomized trials (including pragmatic clinical trials), and research synthesis (including the use of decision analysis and modeling), many of which were addressed during the session.

Observational Study Design

Dr. Giordano opened the session with an overview of CER using observational study design.

Although randomized controlled trials remain the gold standard for research, these trials can be expensive and time consuming and are often limited to a very select population of patients. In contrast, observational studies are inexpensive, can be conducted relatively quickly, and study real-world populations. However, Dr. Giordano said that observational studies also have the potential for bias, including selection bias, performance bias, detection bias, and in some cases, outcome-reporting bias.

Researchers interested in undertaking observational CER studies have a wide variety of information sources from which to pull data.

“Cancer researchers have the advantage of using population-based cancer registries to track incidence and outcomes, and there are more and more publicly available data or licensable databases that capture a large amount of information on patient treatment and outcomes,” Dr. Giordano said, continuing on to discuss several of the more commonly used data sources.

One widely used source of cancer data is the Surveillance, Epidemiology, and End Results registry, a population-based registry that represents approximately 28% of the U.S. population. This database is designed for cancer surveillance and includes information on incidence, stage, survival, and first course of treatment. The National Cancer Data Base is another valuable resource that collects data from hospital registries of Commission on Cancer–accredited facilities, but Dr. Giordano noted that because these data are from accredited facilities, which tend to be high-quality centers, they may not be representative of the population as a whole.

Researchers can also turn to national claims databases such as those that track commercial claims through health insurers and Medicare, which is the primary health insurer for 97% of the U.S. population aged 65 and older.

None of these data sources is without issue, Dr. Giordano said, but all provide a starting point for observational trials that can fill gaps in knowledge left from randomized controlled trials.

Table 1


Traditional Randomized Controlled Trial

Pragmatic Clinical Trial


Estimate maximum effect of intervention

Inform choices between reasonable alternatives

Eligibility Criteria

Strict, specified, most likely to benefit and be compliant


No exclusion criteria


Many subspecialists

Typical provider in typical practice

Level of Analysis


Patient or cluster


Often requires careful training and assessment

Easily and objectively measured without central adjudication


Very stringent

Relaxed criteria

Pragmatic Clinical Trials

Caprice Christian Greenberg, MD, MPH, of the University of Wisconsin, provided an overview of pragmatic clinical trials in CER, including the incorporation of patient-centered outcomes.

The majority of the health care community is very familiar with efficacy trials, Dr. Greenberg said. These trials are rigorous and comprehensive, test a consistent intervention, provide clear indications for eligibility, and typically yield a clear answer (Table 1). However, they are also time consuming and expensive and may have limited generalizability.

Effectiveness trials, or pragmatic clinical trials, can be executed rapidly and can evaluate large cohorts of patients, including special subgroups, but they are subject to confounding and may lack detail about key parameters of interest.

To illustrate these differences, Dr. Greenberg provided an example of a trial that would try to reduce re-excision rates in patients with stage I/II breast cancer. The trial would be designed with a primary outcome of rate of reoperation and a secondary outcome of margin status.

“We are interested in this because the re-excision rate, in general, is higher than it should be,” Dr. Greenberg said, pointing to variation ranging from 18.4%-26.5% according to geographic location, and variation among surgeons from 0%-70%.

A traditional efficacy trial addressing this research question would likely be limited to women aged 40-65 with ductal carcinoma in situ (DCIS) with no comorbidities or neoadjuvant chemotherapy. Study providers would be fellowship-trained surgeons who have performed at least 50 breast surgery cases per year, and outcomes would be examined with central review. In contrast, an effectiveness or pragmatic trial would include all patients with DCIS or stage I-III invasive breast cancer undergoing breast-conserving surgery, would include any surgeon who performs this surgery at a participating institution, and would examine outcomes by looking at margin status pulled from pathology reports at the participating institutions.

“Switching from efficacy trials to pragmatic trials has the potential to increase the efficiency in our research,” Dr. Greenberg said.

Dr. Greenberg also discussed patient-centered outcomes as they relate to CER.

“There has slowly been recognition that, up until now, there was a lot of focus on a provider-centric approach to health care, looking at things that are important to providers, such as morbidity, mortality, or complications,” Dr. Greenberg said. “Slowly, we are starting to recognize the importance of ‘patient centeredness’ and finding ways to measure outcomes that are meaningful to patients.”

According to Dr. Greenberg, patient centeredness is a critical part of CER, focusing on the idea of engaging patients and other stakeholders in research that can help to define outcomes, as well as other aspects that will change the quality of care as the patient experiences it.

Looking back at her example of the re-excision rate study, Dr. Greenberg pointed out that the primary outcome of reoperation status is a patient-centered outcome, but that margin status is not.

“Patients don’t care about that number; they care about whether they need another operation,” she said.

Increased incorporation of patient-centered outcomes are critical to CER going forward, and pragmatic approaches to clinical trial design, like other comparative effectiveness methodologies, have the potential to increase efficiency and decrease the resource intensity of research.

Disease Simulation

The final speaker of the session, Natasha K. Stout, PhD, of Harvard Medical School and Harvard Pilgrim Health Care Institute, discussed the use of disease simulation modeling in CER.

Disease simulation modeling is an IOM-endorsed methodology that helps to fill evidence gaps left by clinical trials or observation studies. According to Dr. Stout, models can often be used in cases in which a decision needs to be made immediately but for which traditional research would take years to produce an outcome of interest.

Modeling can project both near- and long-term outcomes in a timely manner, and, in some cases, can project the value of conducting more research in a given area.

To illustrate the use of modeling, Dr. Stout walked the audience through several examples. In one, she discussed the use of modeling for evaluating the treatment of DCIS, a disease with a heterogeneous progression and high potential for overdiagnosis and overtreatment. The treatment options for DCIS, which can include mastectomy or lumpectomy with or without radiation or tamoxifen, are largely equivalent in terms of overall survival but differ in terms of recurrence risk and quality of life, Dr. Stout said.

Population-level modeling can incorporate heterogeneity and provide outcomes useful for patient-level decision making. DCIS models can combine both trial data and observational data and can project benefit and risk information across multiple treatment alternatives simultaneously, allowing clinicians to make more informed patient-level decisions. There is currently an online decision tool for DCIS at OnlineDeCISion.org, where clinicians can input patient characteristics and get near- and long-term outcomes on recurrence, breast preservation, and survival.

Dr. Stout acknowledged that modeling is not without its inherent challenges, including that models often use indirect evidence and extrapolate to situations where there are few to no data, but disease simulation is an important method in the CER toolbox and complements other research methods and real-world studies.  

Watch the session, visit the ASCO Virtual Meeting website.