HiQuiPs

Designing a QI project – a Comparison of Quality Improvement and Traditional Research Paradigms

In Education & Quality Improvement, HiQuiPs by Shawn MondouxLeave a Comment

You are attending your local ED monthly rounds where a journal club is being held discussing a Quality Improvement (QI) published paper. As the rounds progress, your presenter critiques the article and discusses perceived weaknesses including a small sample size, lack of  randomization and blinding, and lists many different possible biases in this project. You reflect back on your previous journal clubs and wonder whether it makes sense to apply a research-style critical appraisal to this QI paper. You wonder what differences between traditional research and QI paradigms are relevant in answering this question.

Many healthcare institutions in both community and academic settings have adopted QI strategies leading to the training of providers and the recruitment of experts in QI methodology. As these activities continue to grow, both investigators and institutions are being asked to reconcile traditional research with emerging QI agendas.  An approach to this problem requires an understanding of the similarities and differences between traditional research (hereafter referred to as Research) and QI. In this article, we contrast Research and QI and propose opportunities for the productive combination of these research teams and agendas. This is by no means an exhaustive description and comparison.  Instead, it should be used to create and foster discussion around increased collaboration between these two fields.

A comparison of purposes

The traditional teaching is that Research seeks to engage in clinical problem-solving in order to create generalizable knowledge. This knowledge can then act as the fundamental truth to guide how we manage a specific patient population. When Research findings are applied to this same population in a similar setting, similar outcomes should be achieved.  In this way, many major Research trials look to establish new treatments and medications and are largely focused on WHAT treatments are being delivered. 

On the surface, QI seems to steer clear of these questions and focus much more on HOW interventions and treatments are deployed. For instance, as a research and clinical improvement methodology, QI looks to identify areas where clinical performance is suboptimal and deploy interventions to increase the quality of care and improve patient outcomes. 

Although this paradigm is helpful, it falls short in describing the full breadth of Research or QI studies. On closer consideration, we see that the underlying goal of both QI and Research is ensuring patients receive the best treatment for their condition.  There are good examples of this common goal in the literature that blur the line between these two fields. For example, the landmark Rivers trial focused on a method of treatment (early goal-directed therapy in sepsis) rather than novel treatments, yet remains a Research study at its core.​1​ Moreover, statistical tools used in Research such as intention-to-treat analyses may help us predict the effects of a particular therapy outside the controlled environment of a research study.​2​ Therefore the distinction between knowledge generation and its real-world implementation may not be as clear as previously suggested.

Research as a component of QI: the Domains of Quality

Another way to compare these two approaches is to reflect on them through the lens of the Domains of Quality.​3​ It could be stated that an underlying assumption within the Research community is that clinicians would make better care decisions at the bedside if they simply had better medicines, better rules and better diagnostics.​4​  These all fit within the realm of the Effectiveness domain. Although these are laudable goals, they understate the full scope of possibilities to improve care. Safer care may, in some cases, constitute withholding a “best practice” treatment.  Timeliness and flow are rarely the focus of Research. Experiments of efficiency are rarely seen in the literature. Equity may actively be infringed upon in the development of high-cost treatments and the patient voice is rarely considered when data is fed back on a clinician’s adherence to clinical practice guidelines.  

As QI continues to grow and Research continues to expand in their focus on equity and patient-centeredness, the lines of “purpose” seem to blur between QI and Research.  And to be clear, looking for well-defined lines is not in our best interest. Increasingly, we will engage in mixed methodologies where differentiation is especially counter-productive. 

PICO for research and QI

The PICO paradigm has long been used in evidence based medicine.​5​  Population, Intervention, Control and Outcome (PICO) are the building blocks of any good Research study, but similarly applies to QI.  Let us compare these individually:

  • Population. Well-designed QI interventions will include a clear definition of patient populations to which the intervention will be applied.  This population is often determined via a root cause analysis for a specific clinical problem and is included in the aim statement for a project.  Similarly, clear definition of the population of interest helps readers appraise the generalizability of research findings.
  • Intervention. The intervention in a Research study is typically based on previous findings of the research group, published literature, a biological mechanism or a plausible cause.  In contrast, the intervention in a QI study is based on a root cause analysis and addresses possible drivers of a suboptimal outcome. Furthermore, Research study interventions are typically fixed over the entire study period whereas, in QI, interventions can be adapted in successive stages where we apply what we learned from each stage to the next.  The key difference here is that QI acknowledges that the initial intervention may not affect the root cause as planned and thus allows for modification of this intervention. Despite this, both methods should have well described and cogent interventions designed to effect change.
  • Control. Ideally, Research studies use control groups to observe differences in outcomes and performance.  QI studies may compare performance to a control group although they may also use historical trends in data for comparison (see below). This is analogous to a before-and-after Research study design.  Regardless, both methodologies use a comparative population to determine the effect of the intervention.
  • Outcome. Research studies should prospectively identify the outcomes they hope to change before deploying the intervention.  Ideally, Research studies should assess the adverse outcomes associated with the intervention. However, they may lack a comprehensive analysis of the potential negative consequences of deploying a given intervention in the real world. In this area, QI has a small advantage in that it mandates the use of a family of measures which include process measures (to inform adherence to change) and balancing measures, to assess the negative impact of the intervention.  Despite this, well-defined outcomes are essential in both QI and Research.

Protocols and pragmatism

A significant difference between the two methods is the iterative design and deployment of the intervention in QI.  Traditional Research establishes an intervention, estimates its effect and the difference that would be required to have external validity in order to calculate a sample size.  Research protocols are then deployed and kept running until the sample size is reached. Large studies may conduct interim analyses to ensure patient safety, although multiple interim analyses can introduce bias.  Similarly, modification of the intervention partway through a study can make it harder to interpret the results. As such, even interventions demonstrating little to no effect at a preliminary analysis often continue to run to the trial end without assessing reasons for non-compliance or treatment effect. 

In contrast, QI looks to ensure first that a protocol for intervention has good adherence before broader dissemination.  This is done through stakeholder engagement. The intervention is often changed multiple times to achieve compliance, while learning from every change.  The outcome measures in the target population are then evaluated and the intervention is modified accordingly. This process occurs iteratively to achieve the desired outcome. This provides the freedom of developing a highly adaptable protocol that is optimized before broad dissemination. As discussed previously, pragmatic determination of sample size means the minimum number of subjects are required to identify meaningful change or establish a baseline for variation. 

Data Analysis 

The statistical testing used in QI is fundamentally different than that used Research.  A comprehensive description of the statistical tools can be found in a previous post.  For the purposes of this discussion, there exists 2 main differences between data analysis techniques:

  1. Aggregate data vs. Continuous data. It is a common practice in Research studies to combine pre- and post-intervention data into aggregate statistics (mean, median, standard deviation) and compare these using statistical tools (t-test, Chi-square, etc).  QI looks to plot data over time without reducing pre- and post-intervention data to aggregate statistics. The effect in QI studies is demonstrated using statistical tools deployed in the manufacturing environment such as run charts or Statistical Process Control (SPC) Charts (see future HiQuiPs posts!). While both Run charts and SPC charts may use statistical techniques to highlight changes such as the use of probability based tests, the visual plot itself is also helpful to understand the effect of an intervention. 
  2. Analysis between processes vs. within processes. Whereas Research often compares performance of one group to another, QI seeks to compare the variability within and performance of a fixed process both before and after each intervention. QI uses the premise that, if a process is observed long enough, we can statistically determine its “range of production”. Variations in this range after the intervention determines whether the intervention had a significant benefit.

A comparison of Research and QI methodologies is summarized in Figure 1. 

Figure 1. Differences between Research and Quality Improvement ​6​

The future of healthcare problem solving

In a market-minded environment, healthcare improvement “contractors” may be sought to solve healthcare delivery issues.  In this open market, modalities of improvement would compete on the traditional metrics of timeline, budget, complexity, magnitude and results (imagine a healthcare Dragon’s Den). New clinical decision rules, therapies, devices or bundles may be best served by Research paradigms. Flow optimization, reductions of clinic no-shows, adherence to best practices and issues of healthcare equity may be best served by QI paradigms.  

In addition, it is likely that the number of “contractors” at the table will expand.  We should expect to see advocates of Artificial Intelligence, Design Thinking, Experience Based Co-Design and many others join the bid to solve the most appropriate healthcare problems for their field.  Not only should we expect them to take part, we should welcome their presence at the table. It would be shortsighted to think that any one paradigm could solve all of healthcare’s woes and each will find their niche.  Yet perhaps the panacea is not how we circumscribe the territory of each paradigm, but how we seek to blur it in collective models of problem solving. 

Now with a better understanding of differences and similarities between Research and QI you are ready for future journal clubs discussing QI papers. Stay tuned for a future post that will help you prepare a QI project for publication. Finally, join us next month where we discuss our next post on performance score cards in the ED.

Senior Editor: Ahmed Taher

Junior Editor and copyediting: Ed Mason

**UPDATE (November 2020): The HiQuiPs Team is looking for your feedback! Please take 1 minute to answer these three questions – we appreciate the support!**

  1. 1.
    Rivers E, Nguyen B, Havstad S, et al. Early goal-directed therapy in the treatment of severe sepsis and septic shock. N Engl J Med. 2001;345(19):1368-1377. doi:10.1056/NEJMoa010307
  2. 2.
    Montori V, Guyatt G. Intention-to-treat principle. CMAJ. 2001;165(10):1339-1341. https://www.ncbi.nlm.nih.gov/pubmed/11760981.
  3. 3.
    AHRQ. Six Domains of Health Care Quality. Agency for Healthcare Research and Quality. https://www.ahrq.gov/talkingquality/measures/six-domains.html. Accessed December 31, 2019.
  4. 4.
    Mondoux S, Shojania K. Evidence-based medicine: A cornerstone for clinical care but not for quality improvement. J Eval Clin Pract. 2019;25(3):363-368. doi:10.1111/jep.13135
  5. 5.
    Schardt C, Adams M, Owens T, Keitz S, Fontelo P. Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007;7:16. doi:10.1186/1472-6947-7-16
  6. 6.
    Solberg L, Mosser G, McDonald S. The three faces of performance measurement: improvement, accountability, and research. Jt Comm J Qual Improv. 1997;23(3):135-147. doi:10.1016/s1070-3241(16)30305-4
Shawn Mondoux

Shawn Mondoux

Dr. Shawn Mondoux is an emergency physician at Hamilton Health Sciences (HHS) and faculty at McMaster University in Hamilton, Ontario. He obtained a masters of Quality Improvement and Patients Safety (QIPS) at the University of Toronto and serves as QI advisor to all projects within the ED. He has a strong interest in clinical QI work as well as the education of PGME learners in QI principles.