The Social Media Index (SM-Index): A Pilot Project

In Knowledge Translation by Brent Thoma7 Comments

As promised last week, I have put together a pilot Social Media Index (SM-Index) for free open-access medical education (FOAM) websites. There were 25 sites included within this analysis. They were selected from the sites that I frequent as well as anyone that volunteered their site when I asked on twitter. This article discusses the thinking behind the creation of the SM-Index and the methods used to calculate it. The index itself is located here: SM-Index.

Note that this is not a novel concept. There has been a social media index created for blogs/people (here and here) as well as companies. However, to my knowledge, nobody has done this for our particular area of medical education or for the same reasons.

As I anticipated the index being somewhat controversial, I have done my best to keep the methodology as transparent as possible. If you e-mail me I will even send you the Excel file data used to calculate the values.

Why create a SM-Index?

but-why-meme-generator-but-why-84103d

Three reasons (elaborated on in posts A Market of Ideas and Educational Scholarship: Measuring the Impact of FOAM):

  • Open-access medical education needs a way to measure output relative to others for promotion/tenure/recognition/etc in the same way that the classical academia measures theirs through publication numbers, their h-index, etc. Florida State University was one of the first to call for called for these contributions to be recognized in the report of their 2010 Open Education Resources Task Force (see page 26). Thanks to Javier Benitez for the reference.
  • I suspect that post-publication peer review correlates directly with the number of people that read and listen to content. If this is true, then it follows that the popularity of a resource should correlate with its credibility as legitimate academic scholarship. See this piece for a discussion of the requirements for academic scholarship.
  • Newcomers to open access medical education often do not know where to start. Assuming popularity correlates with quality, reviewing the SM-Index would help them to find the most credible content in their area of interest.

Is this just a giant popularity contest?

memes-popularity-contest.jpg.w300h183

Notwithstanding the above meme, Iprefer to think of it as a reputation contest. When I repeatedly visit a website or follow someone on Twitter/Google+/Facebook it is an acknowledgement that they are producing good content that I want to hear about. If a lot of people feel the same it is likely that they have a strong reputation for being a credible source.

What values were used to make up the SM-Index?

The index is a composite value derived from two composite website traffic indices (Alexa and PageRank) and three social media modalities (Twitter, Facebook and Google+). I decided to use the composite indices such as Alexa and PageRank rather than primary website traffic data for ease of access (acquiring Google Analytics data for each site would be very time consuming), transparency (everyone can see these values) and credibility (I doubt I can analyze it as appropriately as they do). The social media modalities were selected for the frequency of their use. The way the SM-index was calculated resulted in more frequently used social media sites (every site on the current index had a Twitter feed, Alexa data and a PageRank) and this resulted in them being weighted more heavily than less frequently used social media sites (fewer Google+ Communities and Facebook Pages).

The values used from each modality were:

Alexa Ranking – Alexa is a website ranking service with methodology that they have not made public. The lack of transparency of their ranking system is criticized by some, but it decreases the ability of a site to “game” the system (although it still occurs). There are toolbars you can install for your browser that will allow you to see the Alexa ranking of the sites that you visit.

PageRank – PageRank is the value that google uses to determine the relative importance of a website based on the number of other sites linking to it and their importance. There are toolbars you can install for your browser that will allow you to see the PageRank of the sites that you visit.

Twitter – Followers, specifically the number of followers of the most-followed author/podcaster on the site (ie – Simon Carley‘s follower number was used rather than St Emlyn’s because he is an author has more followers than the site’s handle).

Facebook Page – Likes

Google+ Community – Followers

How was the SM-Index calculated?

 

chair-math_o_130887

Each site was given a rank for each modality

For example, Twitter ranks ranged from 1-25 because all of the sites had a twitter feed. Because only 7 sites had active Google+ communities this modality was ranked from 1-8 with 8 being the score for sites that have no community. Similarly, the Facebook indices went from 1-11.

Each website’s rank in each modality was averaged

For example, Emergency Medicine Ireland ranked 12/7/11/6/4 for Alexa/Twitter/Facebook/Google+/PageRank, respectively. This gave it a mean ranking of 8.0.

The SM-Index was calculated

The average rank (RANK) for each site was inverted and multiplied by a number (k) equal to 1000 times the average rank of the top website (RANKh):

SM-Index = k (1 / RANK) where k = 1000 x RANKh

In this case, LITFL’s average rank was 1.4 across the 6 modalities and k equaled 1400. If the top site was ever ranked #1 in every modality k would equal 1000.

Sorry for making it complicated. This was done to create a SM-Index with a theoretical range of 1 to 1000 and was relative to all of the other sites included in the SM-Index. In this way, if I repeat the SM-Index in the future the top ranked website will still have a SM-Index of 1000 and all of the sites will still to be ranked relative to each other.

Analysis

Reviewing the results of this experiment, I find the rankings to be fairly consistent with my expectations for the sites. The top sites (LITFL, EMCrit and ALiEM) ranked consistently high across all modalities. Facebook and Google+ were less robust indicators than Alexa/Twitter/PageRank because not all sites used them.  However, because they were ranked only from 1-11 (Facebook) and 1-8 (Google+), respectively, they had correspondingly less effect on the SM-Index than the more robust indicators that ranked sites from 1-25. I’ll leave a more robust analysis to anyone reviewing the index when it is posted.

Conclusion

This was a pilot and the response to it will determine if the SM-Index should be modified, expanded, repeated (monthly, quarterly, annually?) or trashed. Check out the Social Media Index for more details. Please message me on twitter, leave a comment on this post or write a critique of what I have done on your own website. That is, after all, how scholarship works.

Thanks to Michelle Lin, Eve Purdy, Danica Kindrachuk, Jon Sherbino, Javier Benitez and Damien Roland for discussing this topic with me.

Brent Thoma

Editor at CanadiEM
Dr. Brent Thoma is a medical educator, blogging geek, and trauma/emergency physician who works at the University of Saskatchewan College of Medicine. He founded BoringEM and is the CEO of CanadiEM.