[display_podcast]

Date:  December 4th, 2013

Guest Skeptic: Dr. Chris Carpenter and Dr. Brent Thoma

Case Scenario: An newly graduated EM resident gets her research paper published in an emergency medicine journal.

Question: How can she know what kind of impact her paper will have on the profession?

Background: Best Evidence in Emergency Medicine (BEEM) is a knowledge translation and dissemination project started at McMaster University by Dr. Andrew Worster. The BEEM mission is to provide Emergency Medicine practitioners with the best clinical evidence to optimize patient care.

There are close to 5,000 articles published every day. BEEM has a validated and reliable way of screening this mountain of information to separate the signal from the noise. BEEM Process

BEEM via Health Information Research Unit at McMaster University screens the electronic databases of about 200 journals each month. While most articles do not make it past this point, 10-20 articles are emergency medicine related. These articles are then organized in Survey Monkey.

BEEM docs

The survey includes the title of the paper and author’s conclusions. Articles are sent to over 100 BEEM raters. These are front line emergency doctors just like you.Raters are asked to assume that the results of this article are valid. They are then asked to rate clinically how important the paper is to their own practice on a seven point Likert scale. Only those highly rated articles are appraised by the BEEM faculty. Standardized EBM tools are used to create a critical appraisal and BEEM bottom line.

BEEM rater scoreBEEM has the only validated audience rating tool in emergency medicine continuing medical education. Worster et al. Consensus Conference Follow-up: Inter-rater Reliability Assessment of the Best Evidence in Emergency Medicine (BEEM) Rater Scale, a Medical Literature Rating Tool for Emergency Physicians.Acad Emerg Med Nov 2011.

Reference: Carpenter CR et al. Best Evidence in Emergency Medicine (BEEM) Rater Scores Correlate With Publications’ Future Citations. Acad Emerg Med. 2013; 20:1004–1012

Objective: To validate the BEEM rater score as a predictor of literature citation, using a bibliometric construct of clinical relevance to EM.

Methods: Approximately 200 EPs from around the world voluntarily reviewed the titles and conclusions of published EM-related studies from 2007-2012 identified by BEEM process. Using the seven-point BEEM instrument, raters independently assigned a scores to approximately 10 to 20 articles each month. Two investigators independently abstracted the bibliometric indices for these articles. A citation rate for each article was calculated by dividing the Thomson Reuters Web of Science (WoS) total citation count by the number of years in publication. BEEM rater scores were correlated with the citation rate using Spearman’s rho. The performance of the BEEM rater score was assessed for each article using negative binomial regression with composite citation count as the criterion standard, while controlling for other independent bibliometric variables in three models.

Result: A total of 605 articles were reviewed by BEEM raters giving a mean score of 3.84 and a median score of 3.85. The citation rate and BEEM rater score correlated positively (0.144), while the BEEM rater score and the Journal Citation Report (JCR) impact factor score were minimally correlated (0.053). In the first model, the BEEM rater score significantly predicted WoS citation rate (p < 0.0001) with an odds ratio (OR) of 1.24 (95% confidence interval [CI] = 1.106 to 1.402). In subsequent models adjusting for the JCR impact factor score, the h-indices of the first and last authors, number of authors, and study design, the BEEM rater score was not significant (p = 0.08).

Dr. Chris Carpenter

Dr. Chris Carpenter

Authors Conclusion: “To the best of our knowledge, the BEEM rater score is the only known measure of clinical relevance. It has a high interrater reliability and face validity and correlates with future citations. Future research should assess this instrument against alternative constructs of clinical relevance.“

 

 

Limitations: 

  • Surrogate marker of impact
  • Citation rate manipulation (gaming the system)
  • Selection bias of recruiting BEEM raters
  • English language only restrictions

Some Definitions:

  • SoMe: “Social media refers to interaction among people in which they create, share, and/or exchange information and ideas in virtual communities and networks”.
  • FOAM:  Free Open Access to Meducation
  • Altmetrics: The creation and study of new metrics based on the Social Web for analyzing, and informing scholarship.

Screen Shot 2013-12-05 at 10.56.30 AM

Discussion Between Chris and Brent:

  1. Information Overload
  2. Peer Review/Quality
  3. Sustainability/You get what you pay for
  4. Holy Grail/Changing practice and improving patient oriented outcomes

Case Resolution: There were be a number of ways to measure her impact factor both traditional and via social media altmetrics.

Additional Resource: 

KEENER KONTEST: There were lots of people who correctly identified the Hawthorn effect “whereby subjects improve or modify an aspect of their behavior, which is being experimentally measured, in response to the fact that they know that they are being studied”. The first person was Dr. Dree Daugherty from Michigan.

Listen to this weeks podcast for the Keener question. If you know the answer then send your answer to TheSGEM@gmail.com with keener kontest in the subject line. Be the first one with the correct answer and I will send you a cool SGEM skeptical prize.

The BEEM Dream Team is just finishing al the critical reviews for this years conferences. Check out the website and book into SkiBEEM, SteelBEEM, PrairieBEEM or SweetBEEM.

Remember to be skeptical of anything you learn, even if you heard it on The Skeptics’ Guide to Emergency Medicine.