Making the Most of Chart Reviews

In Education & Quality Improvement, Mentorship by Nadim LalaniLeave a Comment

A few months back Dr Eddy Lang (@EddyLang1) [Co-editor of the Royal College Research Guide] graced us with his kind and friendly personality and dropped some pearls on retrospective chart reviews.

Medical Record Review [MRR] Research in General aka Chart Reviews

“Chart reviews don’t get  the respect they may deserve” Dr Lang

Dr Lang lamented the fact that MRR doesn’t get the street cred it deserves. This is large part because of a historical pattern of:

  • Wrong questions
  • Poor methods
  • Action/Documentation Divide = what happened vs what was documented
  • Missing Data
  • Case identification

Gilbert et al., in their article “Chart Reviews in Emergency Medicine: Where are the Methods?” showed that 25% of EM publications between 1988-1995 relied on chart reviews1. However, although inclusion criteria were present 98% of the time, important data regarding methodology was generally absent:

  • Abstractor training, 18% (95% CI, 13% to 23%)
  • Standardized abstraction forms, 11% (95% CI, 7% to 15%)
  • Periodic abstractor monitoring, 4% (95% CI, 2% to 7%)
  • Abstractor blinding to study hypotheses, 3% (95% CI, 1% to 6%)
  • Inter-rater reliability was mentioned in 5% (95% Cl, 3% to 9%) and tested statistically in 0.4%

In their article – Gilbert et al. lay out their solutions1.

The 7 Key Ingredients of good MRR:

 1. Abstractor Training: Need to convince the reader that the people pulling the charts

  • Describe the Qualifications and Training procedure for the data abstractors
  • Before the study begins pull some Trial charts to Test the data abstraction process

2. Case Selection: Needs to be explicit and well described

  • Administrative codes is a start but has flaws
    • Often this can lead to a substudy [i.e do the ultimate codes reflect the Dx?]
  • Clear inclusion/exclusion criteria
  • Screening procedures must be solid

3. Definition of the variables: Need to be done well

  • Dictionary – define things e.g. vitals signs … at triage? by the Emergency Physician? on reassessment?
  • Timing and Source of the info needs to be described
  • Adjudication – how are you going to categorize contradictions and inconsistencies?

4. Data Abstraction Tool: Make it good

  • Need to have a standardized data abstraction tool – use your research staff here
  • Need to have a uniform process of handling missing data  – need to think about what to do with missing or unclear data
  • Consider using software to manage data [e.g. Using Redcap Software]

5. Blinding:

  • Are the abstractors unaware of the study hypothesis? – consider quizzing them afterwards to see!

6. Quality Control

  • Regular meetings to ensure standard process
  • Need to monitor the abstractors work – consider audits
  • Resolution of conflicting assessments

7. Inter-rater reliability: Report inter-rater reliability – it’s eKspected …get it?

  • Reported on a sample of charts reviewed by another [blinded] reviewer

Criteria to Follow for Chart Reviews

The next paper Dr. Lang discussed was a landmark article by Jansen et al who developed a guideline on how to conduct chart reviews.2 They are outlined below for your perusal.

Screen Shot 2014-11-13 at 11.44.07 PM

Dr. Lang also outlined some questions that chart reviews can help to answer:

  • How are we doing? [care practices, quality of care e.g. Look at time to analgesia after intro of new acute pain protocol for say… renal colic]
  • What does this condition look like? [e.g looking for key word search “Rugby” … pull charts associated with rugby injuries]
  • Derivations Models  [e.g. Risk Factors for Hospitalization after Dog Bite injury5]

He also provided some comments on ethics in this type of research, noting that a full ethics process is often not necessary for QI initiatives which can often be approved after “expedited review.” The ARECCI Ethics Screening Tool can guide potential researchers as to what type of review is ethically required for the work that they plan on doing.

Instructive Examples – Chart Reviews CAN change practice!

Dr Lang finished by giving us some examples of chart reviews that changed practice.

  • In his 1995 publication on the prognostic value of amylase in the evaluation of the abdominal pain patient3 he pulled lab results, reviewed charts, and showed that there was no difference in the outcomes between patients with intemediate levels of amylase and normal patients. This impacted the way that his group treated patients with these results at the local level.
  • The Canadian Adverse Events study was a chart review that had an impact on global practice.4 A retrospective review of 3500 charts from 5 provinces in Canada, it showed an adverse event rate of 7.5% of hospitalizations and shone a powerful spotlight on issues of patient safety.

Note: This post was originally published on the ERMentor Blog on November 14, 2014. It was revised by Rob Carey (@_RobCarey),  Sean Nugent (@sfnugent), and Brent Thoma (@Brent_Thoma) before being reposted on CanadiEM in August of 2016.


Gilbert E, Lowenstein S, Koziol-McLain J, Barta D, Steiner J. Chart reviews in emergency medicine research: Where are the methods? Ann Emerg Med. 1996;27(3):305-308.
Jansen A, van A-C, Hutten B, Büller H, Kastelein J, Prins M. Guidelines were developed for data collection from medical records for use in retrospective analyses. J Clin Epidemiol. 2005;58(3):269-274.
Lang E, Afilalo M, Dankoff J, Colacone A, Tselios C, Guttman A. The prognostic significance of moderate hyperamylasemia in the evaluation of the emergency department patient. J Emerg Med. 1995;13(1):107-112.
Baker G, Norton P, Flintoft V, et al. The Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada. CMAJ. 2004;170(11):1678-1686.
Rhea S, Weber D, Poole C, Cairns C. Risk factors for hospitalization after dog bite injury: a case-cohort study of emergency department visits. Acad Emerg Med. 2014;21(2):196-203.
Nadim is an emergency physician at the South Health Campus in Calgary, Alberta. He is passionate about online learning and recently made a transition into human performance coaching. He is currently working on introducing the coaching model into medical education.