Using quality appraisal techniques to find trustworthy content in the FOAM universe

In Editorial, Featured, Knowledge Translation by Daniel TingLeave a Comment

Update June 16, 2020: For a rapid review on this topic, check out the following paper: Ting DK, Boreskie P, Luckett-Gatopoulos S, Gysel L, Lanktree MB, Chan TM. Quality appraisal and assurance techniques for Free Open Access Medical education (FOAM) resources: a rapid review. Semin Nephrol 40(3):309-319.​1​

Introduction

The COVID-19 Pandemic has highlighted the vast amount of information exchange that happens over the internet for healthcare providers. From January to March 2020, almost 400 million Tweets about COVID-19 had been shared, ranging from high-quality content from world experts to panic, over-reaction, or just plain misinformation.​2​ With the vast amount of content and the range in content quality, any consumer of Free Open Access Medical Education (FOAM) must have tools in assessing quality to apply valid knowledge at the patient bedside.   

Quality Appraisal in FOAM

Unfortunately, simply reading an article and providing a gestalt impression of its quality has been shown to be unreliable, since different individuals will see and value different things.​3,4​ In recent years, educators and researchers have developed quality appraisal techniques to augment the reader’s ability to judge online content. Broadly speaking, these tools can be sorted into two categories that rate either an entire FOAM website or an individual online educational resource. 

Quality markers – Website

For website-level metrics, one of the most simple techniques to understand is the Social Media Index (SMi). The SMi is developed from a mathematical formula that considers website views and social media followers/fans. The best way to think about the SMi is comparing it to the impact factor in traditional journals. There have been studies showing that the SMi correlates to both impact​5​ and quality.​6​

Table 1: Top 10 FOAM websites ranked by the Social Media Index (as of 2020).

1. Life in the Fast Lane
2. EMCrit
3. Academic Life in Emergency Medicine
4. Rebel EM
5. EM Docs
6. Emergency Medicine Cases
7. First10 EM
8. CanadiEM
9. Dr Smith’s ECG Blog
10. EMS 12 Lead

Readers can use the SMi to find websites that have strong reputations for producing quality scholarship. Overall, this may be the simplest strategy, especially for the healthcare worker who is relatively new to FOAM. On the other hand, a major drawback is that even the best website has a variance in the quality of the articles published, and “blind faith” in the website can lead the reader astray when there is a poor-quality article. The conventional literature has many examples of this, despite having a more formalized peer-review system.

Other tools that assess websites for quality:

  • Medical Educational Website Quality Evaluation Tool (MEWQET),​7​ developed for Pathology.
  • Modified Education in Otolaryngology Website Assessment Tool (MEOW),​8​ developed for Otolaryngology.
  • Critical Care Medical Education Website Quality Evaluation Tool (CCMEWQET),​9​ developed for Critical Care.

Quality markers – Individual online educational resources

Other tools that have been developed focus on individual resources, such as blog posts. Two prominent tools that have been developed include the rMETRIQ score​10​ and the ALiEM Air score.​11,12​ One of the benefits of these scores is that users can learn specific critical appraisal skills for the medium of the internet. At the same time, the scores take a bit more effort to use and become comfortable with. 

The rMETRIQ score was developed in multiple phases and is intended to be used by readers of all levels of training.   

rMETRIQ FOAM quality
Figure 1: the rMETRIQ score for quality in individual online educational resources.​10​

The ALiEM AIR score was developed for medical educators to rate blog content.

AliEM Air FOAM Quality
Figure 2: the AliEM AIR score for quality in individual online educational resources.​12​

Resources that score the highest using this score receive an “Approved Instructional Resources (AIR)” certification that is displayed on the resource so that readers can easily recognize that the particular online resource has been vetted. Resources that have all received AIR certification have been organized into a curriculum that can count for academic credit. 

ALiEM Air FOAM Quality
Figure 3: The AliEM “Approved Instructional Resources” icon.

Other tools that assess individual resources:

  • Quality checklist for blogs and podcasts.​13​

Conclusion

In summary, knowing how to judge quality in FOAM is increasingly important as FOAM becomes more common among trainees and staff physicians, training programs, and research teams. We recommend starting off with the Social Media Index to become familiar with established sites, and then start to use the rMETRIQ and/or ALiEM AIR scores to build a skill set to assess individual resources.   

How do you judge quality when you use FOAM? Do you have an example of when a poor-quality FOAM resource has led to an adverse patient event or a near-miss?

  1. 1.
    Ting DK, Boreskie P, Luckett-Gatopoulos S, Gysel L, Lanktree MB, Chan TM. Quality Appraisal and Assurance Techniques for Free Open Access Medical Education (FOAM) Resources: A Rapid Review. Seminars in Nephrology. Published online May 2020:309-319. doi:10.1016/j.semnephrol.2020.04.011
  2. 2.
    Rosenberg H, Syed S, Rezaie S. The twitter pandemic: The critical role of twitter in the dissemination of medical information and misinformation during the COVID-19 Pandemic. CJEM. Published online April 6, 2020:1-7. doi:10.1017/cem.2020.361
  3. 3.
    Thoma B, Sebok-Syer SS, Krishnan K, et al. Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study. Annals of Emergency Medicine. Published online September 2017:394-401. doi:10.1016/j.annemergmed.2016.12.025
  4. 4.
    Krishnan K, Thoma B, Trueger NS, Lin M, Chan TM. Gestalt assessment of online educational resources may not be sufficiently reliable and consistent. Perspect Med Educ. Published online February 27, 2017:91-98. doi:10.1007/s40037-017-0343-3
  5. 5.
    Thoma B, Sanders J, Lin M, Paterson Q, Steeg J, Chan T. The Social Media Index: Measuring the Impact of Emergency Medicine and Critical Care Websites. WestJEM. Published online March 23, 2015:242-249. doi:10.5811/westjem.2015.1.24860
  6. 6.
    Thoma B, Chan TM, Kapur P, et al. The Social Media Index as an Indicator of Quality for Emergency Medicine Blogs: A METRIQ Study. Annals of Emergency Medicine. Published online December 2018:696-702. doi:10.1016/j.annemergmed.2018.05.003
  7. 7.
    Alyusuf R, Prasad K, Abdel Satir A, Abalkhail A, Arora R. Development and validation of a tool to evaluate the quality of medical education websites in pathology. J Pathol Inform. Published online 2013:29. doi:10.4103/2153-3539.120729
  8. 8.
    Yang N, Hosseini S, Mascarella MA, et al. Identifying high quality medical education websites in Otolaryngology: a guide for medical students and residents. J of Otolaryngol – Head & Neck Surg. Published online May 25, 2017. doi:10.1186/s40463-017-0220-4
  9. 9.
    Wolbrink TA, Rubin L, Burns JP, Markovitz B. The Top Ten Websites in Critical Care Medicine Education Today. J Intensive Care Med. Published online March 8, 2018:3-16. doi:10.1177/0885066618759287
  10. 10.
    Colmers‐Gray IN, Krishnan K, Chan TM, et al. The Revised            METRIQ            Score: A Quality Evaluation Tool for Online Educational Resources. Yarris LM, ed. AEM Education and Training. Published online July 30, 2019:387-392. doi:10.1002/aet2.10376
  11. 11.
    Lin M, Joshi N, Grock A, et al. Approved Instructional Resources Series: A National Initiative to Identify Quality Emergency Medicine Blog and Podcast Content for Resident Education. J Grad Med Educ. 2016;8(2):219-225. doi:10.4300/JGME-D-15-00388.1
  12. 12.
    Chan TM-Y, Grock A, Paddock M, Kulasegaram K, Yarris LM, Lin M. Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources. Annals of Emergency Medicine. Published online December 2016:729-735. doi:10.1016/j.annemergmed.2016.02.018
  13. 13.
    Colmers IN, Paterson QS, Lin M, et al. The quality checklists for medical education blogs and podcasts. The Winnower. doi:10.15200/winn.144720.08769

Daniel Ting

Daniel Ting is an Emergency Physician and Clinical Assistant Professor at the University of British Columbia, based in Vancouver. He is the Editor-in-Chief of CanadiEM and a Decision Editor at the Canadian Journal of Emergency Medicine. He completed the CanadiEM Digital Scholarship Fellowship in 2017-18. No conflicts of interest (COI).

Brent Thoma

Editor at CanadiEM
Dr. Brent Thoma is a medical educator, blogging geek, and trauma/emergency physician who works at the University of Saskatchewan College of Medicine. He founded BoringEM and is the CEO of CanadiEM.

Teresa Chan

Senior Editor at CanadiEM
Emergency Physician. Medical Educator. #FOAMed Supporter, Producer and Researcher. Chief Strategy Officer of CanadiEM. Associate Professor, Division of Emergency Medicine, Department of Medicine, McMaster University.