SGEMXtra: Enter the METRIQ

Subscribe: RSS

[display_podcast]

Date: March 23rd, 2016

Guest Skeptic: Dr. Brent Thoma. Brent is an emergency physician, trauma team leader, Clinical Assistant Professor and the Director of EM research at the University of Saskatoon in Saskatchewan. As of July 1^st, he will become the EM Program Director.

Brent was last on SGEM#56: BEEM Me Up (Impact Factor in the Age of Social Media). He was discussing the new idea of social media, impact factor and medical education with Dr. Chris Carpenter. Chris had just published a new study in AEM looking at how the BEEM rater score correlated with traditional journal impact metrics.

This is a SGEMXtra so there will be no critical review. We will instead be discussing the following three things:

Healthy Skepticism
Free Open Access to Medical Education (FOAMed)
METRIQ Study.

Healthy Skepticism:

People often think skeptics are close minded, cynical and don’t believe anything. However, Wikipedia defines Skepticism as a “questioning attitude towards unempirical knowledge or opinions/beliefs stated as facts, or doubt regarding claims that are taken for granted elsewhere.” To me it means I want to see some good evidence before I believe something is true.

It is an especially good idea to be skeptical of the medical literature. There was an amazing paper by Dr. John Ioannidis titled Why Most Published Research Findings Are False published over ten years ago in PLoS.

However, we need to avoid nihilism and not conclude that nothing works. What we are trying to achieve is healthy skepticism. To remain open minded. We need to be ready to reject the null hypothesis when presented with convincing evidence to the contrary.

Free Open Access to Medical Education (FOAMed):

The term FOAMed was coined over a pint of beer in Dublin Ireland at ICEM back in 2012. It was to be medical education for anyone, anywhere, anytime.

The creators of FOAMed define it not as a “teaching philosophy or strategy but as a globally accessible crowd-sourced educational adjunct providing inline (contextual) and offline (asynchronous) content to augment traditional educational principles.”

FOAMed movement has just one objective; to make the world a better place. This reminds me of Bill Nye the Science Guy. He encouraged us in science and made us believe that we could change the world. Another example could be Steve Jobs from Apple and his famous ‘Think Different” commercial.

Brent started in the FOAMed world as a resident with his blog BoringEM.org. He has now launched a new site with a team of individuals called CanadiEM.org on January 1st, 2016. The mission of this new national site is “to improve emergency care in Canada by building an online community of practice for healthcare practitioners and providing them with high quality, freely available educational resources.”

One of the most popular posting on CanadiEM has been the info graphic of the Top Five Changes from the new AHA Guidelines. That post has been viewed over 25,000 times on the sites. In addition, the graphics have been shared more than 30,000 times on social media and even translated into a few other languages.

FOAMed has really taken off and become popular for asynchronous learning. CanadiEM adds a distinctly Canadian resource to the global FOAMed community like LITFL, ALiEM, St. Emlyn’s and many others .

Residents have certainly been early adaptors to podcasts and blogs as a resource for medical education. Research has show greater than 80% of residents in Canada and the US are regularly accessing the ~200 English-language blogs and podcasts in EM (Purdy et al CJEM 2015).

So the FOAMed movement has reached a big global audience but that does not make it a good way to learn. It could be a logical fallacy, an argument from popularity (Bandwagon Fallacy). Just because something popular and lots of people are doing it does not make it true.

We need to turn our skeptical eye even on our own stuff like FOAMed. Here is some eminence as evidence from Dr. Joe Lex. Some people consider him the Godfather FOAMed. Dr. Lex has a famous line about medical education. It was recorded in his best Marlon Brando impression from the movie The Godfather.

If you want to know how we practiced medicine 5 years ago, read a textbook.
If you want to know how we practiced medicine 2 years ago, read a journal.
If you want to know how we practice medicine last year, go to a (good) conference.
If you want to know how we practice medicine now and in the future, listen to the conversations in the hallway and use #FOAMed.

But we need more evidence, better evidence, stronger evidence that FOAMed can improve knowledge translation. It has been said many times on the SGEM that it can take over ten years for high quality, clinically relevant information to reach the patients’ bedside. This was illustrated in the leaky pipe model published in the AEM paper by Diner et al almost 10 years ago.

So we know the traditional model takes too long and that social media and FOAMed is fast. But the question that always comes up about FOAMed is what about quality? However, nobody seems to question the quality of a textbook and few people seem to question the quality of peer reviewed publications. What about lectures, where does the structured critical review of this teaching format come into play?

The METRIQ Study:

This stands for Medical Education Translational Resources: Impact and Quality (METRIQ). The translational resources that we are referring to are blogs and podcasts. A few years ago we published the Social Media Index. The idea was to quantify the relative impact of blogs and podcasts so that trainees could find them and producers could get credit for them. Our findings were published WJEM and we received a lot of criticism from FOAM luminaries like Scott Weingart and Simon Carley, in part because it did not measure quality.

Some of that criticism was answered with the Delphi studies looking at quality indicators for blogs and podcasts. In this series of studies we came up with a bunch of quality indicators that would help evaluate blogs and podcasts. But they didn’t really provide an easy way for the readers and listeners to evaluate these resources critically.

Hypothesis for METRIQ: There are several hypotheses that we are examining with this data, but the primary one is that we can develop a tool for junior trainees like medical students that will help them to reliably evaluate the quality of blogs and podcasts.

What is your primary outcome? Interestingly, METRIQ is designed and powered to be an equivalence study. That’s not a study design that is very common, but we are hoping to prove that at least one of these scoring systems will closely approximate the mean gestalt quality evaluation of attending EM physicians. Basically, we want a medical student with a tool to come to the same conclusion about a resource’s quality that a room full of EM attendings would!

What methods are you using? To do that, we are recruiting medical students, EM residents, and EM attendings to rate a bunch of blog posts! To get enough data to power our calculations each participant needs to rate 20 separate posts.

How many people do you need to complete the METRIQ? We are hoping to recruit 64 collaborators from each group – so 64 medical students, EM residents, and EM attendings.

What are your numbers like as of today? We are off to a great start. So far as of recording the show today, we have had a total of 220 people express interest in the study. Of those, 113 have started rating posts and 42 have completed all of their rating. We’re hoping that the responses will continue to roll in and we are still recruiting to ensure that 64 people from each group finishes their rating.

Now it is time for the SGEMers to pay back to the FOAMed community for all the great free medical education. Please sign up today and finish the METRIQ Study. Go to METRIQstudy.org to find out all the details.

There is no keener contest this week but last weeks winner was Dhruba Tamuli an EM resident from Hull Royal Infirmary in the United Kingdom. They knew another clinical indication for mannitol other than raised intracranial pressure is for reduction of elevated intraocular pressure.

The SGEM will be back next week with another critical appraisal trying to cut that knowledge translation window down from over ten years to less than one year.