Pages Navigation Menu

Meet 'em, greet 'em, treat 'em and street 'em

SGEM#127: Suspicious Minds vs. Clinical Prediction Rule in Children with Trauma

SGEM#127: Suspicious Minds vs. Clinical Prediction Rule in Children with Trauma

Podcast Link: SGEM127
Date: September 8th, 2015

Guest Skeptic: Dr. Anthony Crocco is a Pediatric Emergency Physician and is the Medical Director & Division Head of the Division of Pediatric Emergency at McMaster’s Children’s Hospital. I often refer to him as a PedsEM Super Hero but the SGEMers probably know him for his RANThony’s on YouTube.

Season#3 finished with an SGEM#126 Hot Off the Press featuring Dr. Jeff Kline discussing outpatient management of venous thromboembolism. We are going to kick off Season#4 with another SGEM HOP from Academic Emergency Medicine.

SGEM HOP SAEM logoSGEM Hot Off the Press Process:

  1. We select a paper that has been accepted by the editors at Academic Emergency Medicine that is about to be published.
  2. The SGEM then put a skeptical eye upon the manuscript using a modified BEEM critical appraisal tool.
  3. One of the authors is then invited to discuss their work on the SGEM.
  4. We podcast the review on iTunes and on the SGEM blog shortly after publication inviting the SGEMers to respond on the blog, via Twitter or Facebook.
  5. A summary of the review including the top social media feedback is then published in a subsequent issue of AEM.

Case: 10 year old girl is a restrained rear-seat passenger in a motor vehicle collision.  The driver of the other car was texting and ran a red light at low city speeds, “T-boning” her vehicle on her side.  There was minimal intrusion into the vehicle space.

The girl presents to the ED via EMS with a c-spine collar and on a spine board.  She is alert, appropriate and in no distress.  Her exam is normal, including no evidence of abdominal wall trauma, tenderness or nausea/vomiting.  The remainder of her exam is normal.

Given the mechanism of the injury you are concerned about blunt abdominal trauma.  You wonder if you should do a CT abdomen, but are also concerned about the risk of radiation.  You heard of a study in 2013 that developed a clinical decision instrument for this very question, but don’t know if you should use that, or trust your own ‘gut feeling‘.

Background: The leading cause of death in children is trauma. Blunt torso trauma makes up a significant portion of the mortality statistics. CT scanning has become a routine method for identifying children with intra-abdominal injuries. The amount of radiation being delivered to these vulnerable patients and the risks of developing radiation-induced malignancy has raised concern. There is evidence that clinical suspicion is not accurate in identifying children with intra-abdominal injuries. In addition, there is wide variation on the use of abdominal CT and over utilization in children with low pre-test probability of clinically important injuries.


Clinical Question: How well do clinicians identify children suffering from blunt torso trauma who have intra-abdominal injuries compared to an established clinical prediction rule?


Mahajan, Prashant-2010

Dr. Prashant Mahajan

Reference: Mahajan P et al. Comparison of clinical suspicion versus a clinical prediction rule in identifying children at risk for intra-abdominal injuries after blunt torso trauma.  Sept 2015 AEM .

  • Population: Children <18 years old with blunt torso trauma presenting to an Emergency Department.
    • Exclusion criteria: “injury occurring >24 hours prior to presentation, penetrating trauma, pre-existing neurological disorders preventing reliable examination, known pregnancy, or transfer from another hospital with prior abdominal CT scanning or diagnostic peritoneal lavage.” Patients were also excluded if the clinician did not document their level of clinical suspicion of intra-abdominal injury.
  • Intervention: Clinical suspicion of intra-abdominal injury as per ordinal scale (<1%, 1-5%, 6-10%, 11-50%, >50%)
  • Comparison: Intra-abdominal injury undergoing acute intervention.
  • Outcome: Descriptive statistics using clinician gauged risk <1% as cut-off.
    • Sensitivity, specificity, negative predictive value, positive predictive value, positive and negative likelihood ratios were calculated. Results were compared to an un-validated clinical prediction rule metrics.

Authors’ Conclusions: “A clinical prediction rule had a significantly higher sensitivity than clinician suspicion for identifying intra-abdominal injury undergoing acute intervention, but a lower specificity. The higher specificity of clinician suspicion, however, did not translate into clinical practice, as clinicians frequently obtained abdominal computed tomography scans in patients they considered to be at very low risk. If validated, this clinical prediction rule can assist in clinical decision-making around computed tomography use after blunt abdominal trauma in children by limiting computed tomography scan use in low-risk patients.”

checklist-cartoonQuality Checklist for A Diagnostic Study:

  1. The clinical problem is well defined. Yes. Whether or not to expose children to ionizing radiation is an ongoing concern and minimizing this exposure while not missing significant injuries is a concern for clinicians in the ED.
  2. The study population represents the target population that would normally be tested for the condition (ie no spectrum bias) Yes
  3. The study population included or focused on those in the ED. Yes
  4. The study patients were recruited consecutively (ie no selection bias). Yes
  5. The diagnostic evaluation was sufficiently comprehensive and applied equally to all patients (ie no evidence of verification bias). No
  6. All diagnostic criteria were explicit, valid and reproducible (ie no incorporation bias). Yes
  7. The reference standard was appropriate (id no imperfect gold-standard bias). Yes
  8. All undiagnosed patients underwent sufficiently long and comprehensive follow-up (ie no double gold-standard bias). Yes
  9. The likelihood ratio(s) of the test(s) in question is presented or can be calculated from the information provided. Yes
  10. The precision of the measure of diagnostic performance is satisfactory. Yes

Key Results: Clinicians predicted 168/203 (83%) intra-abdominal injuries requiring intervention and predicted 9217/11716 (79%) negative patients.


Sensitivity= 82.8% (76.9 – 87.7)  Specificity= 78.7% (77.9 – 79.4)

PPV= 6.3% (5.4 – 7.3) NPV= 99.6 % (99.5 – 99.7)

+LR= 3.9 (3.6 – 4.2)  -LR= 0.22 (0.16 – 0.30)


Screen Shot 2015-04-25 at 3.11.12 PMFive questions for Dr. Mahajan about his research study. Listen to the podcast for his responses.

  1. There usually is some backstory about how a research project got started. So what was the inspiration for doing this project?
  2. Your title does not specify that the standard which you are comparing the physicians performance metrics is an un-validated instrument which may mislead the reader into using this instrument prematurely.  You mention the fact the instrument is unvalidated later in the text, but I worry that the message is lost.
  3. The literature has many examples where the results of a derivation study were not echoed in the validation studies. Are there plans to take this clinical prediction instrument and attempt to validated it in a future study?
    • Screen Shot 2015-06-05 at 10.58.52 AMWe have to remember that before a CDI is ready for prime-time, it should go through three steps.  The first is derivation – where the researchers try to see amongst a number of various variables, which ones appear to have predictive value.  Unfortunately, just by chance alone, some variables may appear significant when they are not.  This is why the second stage, validation, is so important.  At this stage the researchers work to confirm the significance of the predictive variables from the derivation study.  There are examples of seemingly awesome derived CDIs that, when it came time to validation, did very poorly.  Recently the CATCH CT head rule was developed, and the initial results were promising.  People were using the CDI and it apparently even showed up on a national licensing exam!  The results from the validation studies for the CATCH rule, however, were not nearly as good and, as such, it should not be used.  A final step, if a CDI passes validation, is impact analysis.  This is where researchers see if the rule actually does what it’s supposed to in the real world.
    • For a more detailed discussion of these three steps of developing a CDI, SGEMers should check out Anthony’s amazing website sketchyebm.com! He has these great five minute videos using whiteboards to explain CDIs and other EBM concepts. I also really love your tag line…Always draw your own conclusions.
  4. The rates of CT scans based on physician behavior was 5318/11919 = 45%.  The rates of CT using the un-validated decision instrument would have been 7004/12044 = 58%.  The un-validated instrument appears to significantly increase CT rates.
  5. We like to see patient oriented outcomes as the primary outcome. Although the study title describes the outcome being “children at risk for intra-abdominal injuries”, really the results are describing children with intra-abdominal injuries requiring intervention.

Comment on authors’ conclusion compared to SGEM Conclusion: Until the decision instrument is validated, there is limited value in comparing clinicians’ judgment to it.


SGEM Bottom Line: Clinicians appear to do reasonably well at predicting which children with blunt torso trauma require investigations for intra-abdominal injuries requiring intervention. Nothing can be said at this time about the derived clinical decision instrument.


Case Resolution: Trauma labs and routine xrays were performed including chest and c-spine – all of which were normal.  The c-spine collar was removed and the 10 year old was observed for a couple of hours, during which time she tolerated oral intake and was able to ambulate.  She continued to be pain-free and is discharged home.

Clinically Application: No change for now.

What do I tell my patient? I’m a bit worried that your child has a significant abdominal injury. We will need to do some more tests to find out if there are internal injuries. One the tests that can help us identify important abdominal injuries is a CT scan.

Keener Kontest – Listen to the podcast for this weeks’ Keener question. If you know the answer then send me an email TheSGEM@gmail.com with keener in the subject line. The first correct answer will receive a cool skeptical prize.

There you have it the first episode of Season#4 of the SGEM. Don’t forget to give us feedback on Twitter, Facebook or the blog. The best comments will be published in Academic Emergency Medicine. Make sure you also check out SketchyEBM for a number of great EBM videos.


Remember to be skeptical of anything you learn, even if you heard it on the Skeptics’ Guide to Emergency Medicine.


Conferences:

Screen Shot 2015-06-28 at 9.33.29 AMSt._Paul_s_Header_2015ACEP-2015-logoThe Teaching Course

 

 

  • SAEM EBM IG

    Deriving, validating, and testing the real-world impact of Clinical Decision Instruments (i.e. decision aids, clinical decision rules) is labor-intensive, time-consuming, and quite expensive. In EM the methodological standards for this process were described >15 years ago (http://pmid.us/10092723), but researchers and clinicians often seek to accelerate the process leading to a viable instrument ready for widespread application. The SGEM commentators @TheSGem:disqus and @SketchyEBM appropriately note (with suspicious minds) that the pediatric blunt trauma clinical decision instrument is NOT ready for widespread use because it lacks external validation and an impact analysis. Clinical decision instruments often provide less impressive results in validation trials than in derivation studies (for example: http://pmid.us/3689067) and sometimes fail validation (http://pmid.us/19181957) so there are no shortcuts. Nonetheless, over-testing and over-diagnosis are real problems in emergency medicine (see the open access 2015 “Overdiagnosis” Consensus Conference proceedings in the December 2015 issue of Academic Emergency Medicine that will be available at http://onlinelibrary.wiley.com/journal/10.1111/%28ISSN%291553-2712/homepage/consensus_conference_proceedings.htm), so efforts to safely & efficiently reduce CT ordering rates are essential. Kudos to Dr. Mahajan’s research team and PECARN for studying this challenging problem!

  • TheSGem

    “The value of any imaging CDR is the ability to be highly sensitive to rule OUT any significant disease so that imaging is not required. This rule does not meet this requirement with a Sensitivity lower 95%CI interval of 76%, which misses 1 in 4 cases of significant abdo injury. Increased utilization of CT scans with the CDR is also problematic…” Dr. Suneel Upadhye BEEM Group McMaster University

    • SAEM EBM IG

      Focusing on sensitivity alone is problematic. Remember the definitions for LR+ is sensitivity/(1-specificity) and for LR- is (1-sensitivity)/specificity. If a clinical decision instrument has a 95% sensitivity but a 5% specificity both the LR+ and the LR- are 1 and do not alter the post-test probability of the outcome at all. This is why it is much more helpful to think in terms of LR+ and LR- (and for continuous data interval likelihood ratios) rather than sensitivity or specificity in isolation. Excellent reviews on this topic at http://pmid.us/12883522, http://pmid.us/10216335, and http://pmid.us/12883521.

  • Kirsty Challen

    For people like me who think in pictures.

    • TheSGem

      Great work Kristy. Thanks for creating this info graphic of the paper and the #SGEMHOP episode on the topic.

  • Mike Falk

    Looking at the data and the study, I think we are still left with clinical judgement as the best option. I often do serial exams of the abdomen and use FAST to examine the belly and then make a decision. If it’s late at night, keeping the patient till the am and then reasessing with surgery is always a great option. But if my clinical judgement is low and the FAST is negative, then I am comfortable with those types of plans.
    I would vote for “suspicious minds”.

    • TheSGem

      I too think we are still left with clinical judgement as the best option at this time.

  • Jeff Perry

    The authors of this study are to be congratulated for their incredible work. This is a very large, well conducted derivation study for a serious, but relatively rare clinical problem. Unfortunately the sensitivity of their proposed rule is 97% (94-99%). This sensitivity is better than using the predictions by clinicians (with >=1% being considered high risk -sensitivity of 83% (77-88%)). However, physicians did not practice using this 1% cut point. They still investigated patients with a sensitivity of <1%. Therefore, it is doubtful that physicians will adopt a rule with 97% sensitivity, even if subsequently successfully validated. I hope that the authors are able to continue studying this problem to identify a more sensitive rule without drastically lowering the specificity. Otherwise, it is unlikely clinicians will use even a validated version of their rule.

    • Kurt Eifling

      Yet the search should continue. The PECARN work that led to the derivation and validation of the use of head CTs after blunt head trauma… that has turned a common clinical scenario from muddy and spooky to a conversation about known risks, clear prognosis, and deliberate choices. I agree that blunt abdominal trauma is less common, and of such complexity that any signal in the data will likely be vague compared to the head trauma data. But if it eventually helps us bring an informed, understandable conversation to the bedside in this scenario, the grind of these authors’ hunt will be worthwhile.

    • TheSGem

      Thank you for the feedback Dr. Perry. Agreed that we hope the authors continue to study the problem. Front line care providers of injured children need good studies to help give them good information so they can provide the best care to these patients.

  • Prashant Mahajan

    Great discussion. I just want to clarify regarding #4 – the intent of the clinical decision rule that was derived from the parent study (Holmes et al in AEM) was to assist the clinician’s in identifying those who are at low risk for intra-abdominal injury requiring intervention (IAIAI). The parent study clearly states that presence of any variable in the prediction rule does not mean that it is an indication for clinicians to order a CT. The study further provides risk estimates for IAIAI based on presence of individual variables of the rule. It is possible that presence of certain variables can suggest observation rather than obtaining a CT and thus further reducing the risks associated with radiation. Indeed, we do not want clinicians to conclude what was mentioned in #4. We are suggesting that if this rule is validated, then this can be used in the clinical context to aid clinicians in whom they do not need to obtain a CT – which is a majority of the patients who unfortunately are getting a CT despite low clinician suspicion.

    • TheSGem

      Thank you for the clarification. We always get a better understanding of the literature when engaging with the lead author.

  • TheSGem

    #SGEMHOP now published and available #FOAMed http://onlinelibrary.wiley.com/doi/10.1111/acem.12851/abstract

  • Pingback: AIR Series: Trauma Module (2017) – Global Intensive Care()

  • Ken Milne
  • Ken Milne