Educational Scholarship: Measuring the Impact of FOAM

In Knowledge Translation by Brent Thoma8 Comments

I recently asked a prominent EM researcher “Does FOAM (Free Open Access Meducation) have academic value?” I don’t need to paraphrase his answer as it was quite concise: “No.” While he was elegant in his bluntness, you will probably not be surprised to hear that I disagree. This response led thinking and discussion that culminated at CAEP13 after I attended the Consensus Conference on Educational Scholarship.

What is Educational Scholarship?

The Consensus Conference was held to provide feedback on a group of educators’ definition of and vision for educational scholarship in EM in Canada. Its tenets were consistent with those in the literature (thanks to both of my reviewers, Teresa Chan and Michelle Lin, for pointing me to these references) and focused on three pillars: public availability, peer review and how well others are able to use it. For more literature on this topic, consider reviewing Boyer’s (1990) model of scholarship. Some prominent definitions include:

“We develop a scholarship of teaching when our work as teachers becomes public, peer-reviewed and critiqued, and exchanged with other members of our professional communities so they, in turn, can build on our work.” (Shulman, 2000)

“Educational scholarship refers to any material, product or resource originally developed to fulfill a specific educational purpose that has been successfully peer-reviewed and is subsequently made public through appropriate dissemination for use by others.” (AAMC, 2009)

“The systematic investigation of an issue which results in the creation of a product presented in a form that can be reviewed by peers for quality, and publicly disseminated for others to learn from and build upon.” (CAME, 2011)

FOAM clearly meets two of the three pillars with post-publication peer-review being the arguable criteria that makes or breaks the case for FOAM as educational scholarship.

Why does Educational Scholarship matter?

I started this blog primarily to meet my own educational objectives. However, over the last 7 months (time flies!) it has morphed from a curious educational hobby to something that I have become passionate enough about to spend several hundred hours on. I’d like to think that I am creating something that has worth beyond its intrinsic value as a learning tool for me.

I recall twitter discussions from many moons ago outlining the academic acknowledgement (or lack thereof) of the big producers of FOAM. If nothing has changed since, I believe Michelle Lin of ALIEM is the only person receiving substantial acknowledgement for her work (she holds the UCSF Academy Endowed Chair in Emergency Medicine Education). Am I better off plugging away in clinical research or systematic reviews?

Why should FOAM be considered Educational Scholarship?

I think FOAM should be considered educational scholarship because it is a novel method of knowledge translation that incorporates edutainment, engages a broad audience and allows for robust post-publication peer-review. Research and knowledge translation are two sides of the same coin: while FOAM would be nothing without the research it is based on, research is useless if it is not put into practice. Lauren Westafer and Scott Weingart delved into FOAM as knowledge translation  in much more depth in (Don’t) Mind the Gap and EMCrit Wee -Tacit Knowledge and Medical Podcasting, respectively.

As mentioned above, FOAM meets all of the criteria with the arguable exception of peer review. I know I’ve beaten that topic to death elsewhere (see previous posts A Commitment to Pre-publication Peer Review, Arguments for a Journal of FOAM, FOAM: A Market of Ideas and Crowdsourced Instantaneous Review: The Peer Review of FOAM) but I need to open this can of worms once more because it led me to consider the use of impact and reputation as markers/surrogates of peer review.

During the consensus conference at CAEP13 online publication on websites like this one was dismissed from consideration as educational scholarship because “anyone” could post “anything” and it would be remiss to let it stand as scholarship with no publication standard. Leaving the problems associated with classical peer review aside, I think this is a fair point. Certainly, there is great variability in the quality of information found on FOAM sites and someone publishing poor quality work that nobody reads should not be lauded for their “scholarly activities.” However, there was also acknowledgement at the Consensus Conference that post-publication peer review was an important, but often absent, component of educational scholarship.

Following the conference I spoke to some of the attendees about the substantial post-publication peer review that many of the popular sites and podcasts enjoy. Scott Weingart addressed this on an EMCrit Wee -Tacit Knowledge and Medical Podcasting saying:

“If I get anything wrong or say something that (listeners) disagree with they write me or put show comments on and so I actually have a more robust peer review on my podcast than I ever would on any publication I would place in conventional journals.”

He noted that this robust review is primarily a result of his large and vocal audience. Other FOAM resources vary in the size of their audience on a spectrum of One (themselves) to Weingart (who doesn’t listen to EMCrit?). Why does he have such a big audience? Because his peers have spoken, deemed that he publishes good stuff and spread the word every time they end a statement saying “I heard about it on EMCrit.” With everybody listening to his words, they are subject to heavy scrutiny. The impact that his podcast has results in its robust peer review and because his work has survived this scrutiny his reputation has grown.

Presuming (perhaps prematurely) that a robust post-publication peer review process is sufficient to qualify as educational scholarship, a FOAM resource’s value as educational scholarship could be determined with surrogate measures like impact and author reputation.

How can we measure the Impact of FOAM?

The idea of measuring impact to determine the value of someone’s scholarly output is not new. The h-index and related metrics are used in academia to for a similar purpose: to quantify the impact of an academic’s publications rather than just their productivity. The h-index as defined by Wikipedia is:

“a scholar with an index of h has published h papers each of which has been cited in other papers at least h times. Thus, the h-index reflects both the number of publications and the number of citations per publication.”

While these metrics are arguably a big improvement over systems looking solely at the number of publications, they are still criticized for many reasons. In particular, defining a single metric may result in too much emphasis being put on a single measure and there is a potential for ‘gaming’ the system by taking advantage of how it works.

As FOAM is criticized for its ease of publication, impact and/or reputation metrics would help to quantify the value of FOAM contributions. Greater impact/reputation would presumably result from contributions with greater scholarly value. While there would be no easy way to develop such a system, the idea has been discussed as a new model of peer review for the classic scientific literature and there are models that are functionally related. For example, a reputation system is used by Wikipedia that can accurately predict the quality of the contributions of its editors and Alexa web analytic software is able to rank websites based on their traffic and number of inbound links (citations?). Additionally, as I outlined in FOAM: A Market of Ideas, I think there is an argument to be made that the prominence of FOAM serves as a surrogate measure of their quality with the best work being shared the most often in social media and attracting more readers/listeners.

While there isn’t a perfect measure of impact in classical academia, creating a measure will be even more difficult in the multi-modal world of FOAM. How could podcasts be compared with blogs? Should they be? What about apps?  There is huge heterogeneity in this subset of the academic world – thanks to Michelle Lin for raising some of these questions in her review of this article.

Conclusion

Where do we go from here? Stay tuned this week for an experiment that I anticipate will be widely criticized, I will attempt to quantify the impact of some popular FOAM sites. Good idea? Bad idea? If you have any thoughts (or if you disagree with me completely regarding all of this), please comment, e-mail or tweet!

I’d like to thank Rob Woods, Brian Rowe, Jonathan Sherbino, Teresa Chan, Joseph Bednarczyk and Ken Milne because this post represents the cyclone of discussions that I have had with them.

Thanks as well to Michelle Lin and Teresa Chan for reviewing and providing feedback on this post.

Brent Thoma

Brent Thoma

Editor at CanadiEM
Dr. Brent Thoma is a medical educator, blogging geek, and trauma/emergency physician who works at the University of Saskatchewan College of Medicine. He founded BoringEM and is the CEO of CanadiEM.
Brent Thoma