Pages Navigation Menu

Meet 'em, greet 'em, treat 'em and street 'em

SGEM#153: Simulation for Ultrasound Education

SGEM#153: Simulation for Ultrasound Education

Podcast Link: SGEM153

Date: May 2nd, 2016

Guest Skeptic: Dr. Chris Bond. Chris is an emergency physician and clinical lecturer at the University of Calgary. He is currently the host of CAEP Casts, which highlight educational innovations from emergency medicine residency programs across Canada. Chris also has his own #FOAMed blog called Standing on the Corner Minding My Own Business (SOCMOB).

Case: An emergency medicine resident in your institution has learned some basics of ultrasound training, but feels uncomfortable performing ultrasound in a crashing patient. They come to ask you how they can learn to use their ultrasound skills in critically ill patients, other than doing it in real time.

Background: Point of care ultrasound (PoCUS) can be broken down into two main categories – diagnostic applications and procedural applications. There are simulators and trainers for both types.

  • Diagnostic Applications: When it comes to diagnostic ultrasound simulation the case is most compelling for the more invasive applications. These include things like transesophageal echo in shock and arrest states or transvaginal ultrasound in first trimester bleeding.
  • Procedural Applications: The procedural side of ultrasound simulation is where we find the most compelling evidence. In one study by Barsuk et al, trainees who were trained to mastery on Central Venous Line trainers with the use of ultrasound and performed far better on real patients than those trained with the more traditional approach of go read about it, then see one – do one.

There is now a movement towards hybridized simulation experiences where either diagnostic or procedural trainers are introduced into broader simulation environments. This type of bridging from simple task training to clinical integration (in a simulated setting) is now being explored more and more, especially as we move towards competency based training.

Assessing competency is not a one-time event. Seeing a trainee perform a skill like PoCUS in one instance is insufficient to say they have achieved competence. Assessment of competence requires thoughtful assessment of several instances spaced over time and ranging in complexity. It is here where these types of hybrid simulation offer a window for assessment.

Seeing when the trainee decides to reach for the transducer, how they hold it, how long they allow themselves to scan, how they interpret and then integrate the findings into care. This can serve as one of many touch points making the case for that trainee’s PoCUS competence. Eventually, with multiple assessment points over the duration of their training with the formative feedback that is associated with it, they will achieve competence in PoCUS.

Resuscitative PoCUS can be divided into three parts:

  1. Firm Grasp of Indications – Knowing when and where to scan
  2. Image Generation with Interpretation – Generating the images and knowing when the image is adequate for interpretation and data extraction
  3. Integration into Clinical Care – Making decisions based on the all the data (history, physical and PoCUS images).

High-fidelity simulation (HFS) can then be combined with PoCUS to teach residents.

  • The purpose of this study was to evaluate two comparable ultrasound simulation interventions as used during HFS. Comparing two somewhat similar but distinct interventions allowed the study team to assess and isolate for the potential value of basic probe handling and other logistical aspects associated with the use of either intervention.

Clinical Question: What ultrasound simulation method is preferred by trainees and instructors in high fidelity simulation?


Reference: Olszynski et al. Ultrasound during Critical Care Simulation: A Randomized Crossover Study. CJEM 2016

  • Population: Emergency Medicine residency program trainees and 8 instructors (5 staff physicians, 3 senior emergency medicine trainees with significant U/S experience)
  • Intervention: 1) edus2 PoCUS simulator, which is comprised of a modified ultrasound probe and laptop and 2) Laptop with ultrasound videos placed on an audiovisual cart
  • Comparison: Previous PoCUS with HFS
  • Outcome: Trainee and instructor preference

Authors Conclusions: The edus2 was identified as being a superior teaching intervention, as it allowed for greater functional integration of PoCUS into critical care, better assessment of trainee skills and had greater impact on session debriefing and formative feedback.

Quality Checklist for Observational Trials:

  1. checklistThe study population focused on those in ED. Yes, it focused on EM trainees who would normally be performing the intervention.
  2. The groups were adequately randomized. No, it was not a random allocation method, as groups were “every other” trainee based on order of arrival.
  3. The randomization process was concealed. No.
  4. The study participants were analyzed in the groups to which they were randomized. Yes, but this is a crossover study, so all participants completed both interventions.
  5. The study participants were recruited consecutively. Yes.
  6. The patients in both groups were similar with respect to prognostic factors. Yes.
  7. All participants were unaware of group allocation. No.
  8. All groups were treated equally. Yes.
  9. Follow-up was complete (>80% both groups).Yes.
  10. All important outcomes were considered. Yes.
  11. The treatment effect was large enough and precise enough to be clinically significant. Unsure.

Key Results: 25 trainees with an average pre-intervention multiple choice question exam of 72% based on the American College of Emergency Physician’s EMSONO online exam. This means they were familiar with PoCUS

Twenty-one out of twenty-five already had their level I PoCUS course or equivalent. However, they reported their HFS to date had been poor (3.26 out of 10)


Trainees and instructors rated these two studied interventions superior over-previous critical care ultrasound simulation with edus2 being the prefer method of the two.


Sim ResultsTrainees rated edus2 and laptop as a quantitatively better experience than previous experience on a Likert scale in five categories. There was also no significant difference between the two teaching interventions.

Two weeks after the course the trainees completed a qualitative assessment. All the respondents indicated they preferred the edus2. Reasons included primarily the themes of real-time handling of the U/S probe with the edus2 and hands-on use. The laptop alone was felt to be a more artificial intervention.

The instructors were also asked about intervention. Both the quantitative and qualitative data support a preference for the edus2 over the laptop intervention as well as previous experiences.

Screen Shot 2015-04-25 at 3.11.12 PM

The findings of this study seem to support the integration of PoCUS whether by a simple laptop model or an edus2 model.

Dr. Paul Olszynski

Dr. Paul Olszynski

To get a deeper understanding we have a few questions for Dr. Paul Olszynski. Paul is a Clinical Assistant Professor and Director of Emergency Department Ultrasound in the Saskatoon Health Region, Paul has recently been appointed Director of Undergraduate Ultrasound Education at the College of Medicine at the University of Saskatchewan (listen to the podcast for his full responses):

  1. Randomization: You just allocated every other trainee based on arrival
    • The challenge with randomization was that we never really knew who was coming on a given day. So we decided that we would use their order of arrival to the simulation center at Whipps Cross Hospital as part of our randomization. As they walked into the center, we provided them with an envelope that included their group allocation. We had arranged our envelopes to alternate from group A to B to A and so forth as a means of ensuring that we would have balanced groups in terms of size. It was somewhat reassuring to see that our process did result in 2 generally similar groups. In the end there were no statistically significant differences between the two groups in terms of POCUS and simulation experience so I guess our randomization worked?
  2. Quantitative vs. Qualitative Data. You used both tools. Why and what do you think they tell us about your study?
    • When it comes to education research, I think combining the two forms of data really enriches the study. It adds insight to the numbers. In our study, it also highlighted how tricky research on students and instructors can be. For example, the trainees initially rated the two interventions very similarly. Yet on the exit survey, every one of them stated they preferred the edus2 to the laptop intervention.
  3. Personal Bias: You were the lead author but also were involved in the teaching sessions. How did you try to minimize this bias?
    • Early on we realized I could not act as a teacher/instructor with any of the groups. In the end, my job was to briefly introduce the trainees to the two interventions (explaining how they work and how they would be able to activate the clips they desired during the scenarios). I was also the voice of the patient for all scenarios.
  4. Resuscitative PoCUS Competence Model. You give a conceptual framework illustrating the key concepts supporting the use of PoCUS simulation in HFS. Can you take us through that model?Resuscitation
    • Bloom’s cognitive, affective psychomotor domains:
    • Miller’s Framework:
      • In my mind it represents the earliest work around assessment of competency in Medicine. First the trainee is simply asked to write it down but as they move along in their training, we expect them to describe how to do it and show us how they would do it. Until finally, near the end of their residency we observe them do it completely independently. The hybrid simulation experience we studied represents the shows how in Millers framework
    • Kirkpatrick’s Hierarchy:
      • This is about the meaningfulness of outcomes as they relate to an educational intervention. Ideally, studies like mine would show clinically significant outcomes (better patient outcomes). But to get there, we often have to first make our way up the pyramid. I think introducing PoCUS into resuscitation simulation provides us with the opportunity to “glimpse” at transfer of knowledge into practice (albeit in a simulated setting) where we can see things like improved diagnostic accuracy or shorter time to diagnosis.
    • Zone of Proximal Development (ZPD):
      • Going back to those residents of ours – the ones who are nervous about reaching for the transducer when facing a critically ill patient: You could think of that “reach” as representing their zone of proximal development – that space between what they know how to do (for example: scanning a stable trauma patient) and what they have not done yet (but have some of the basic skills already) like performing resuscitative PoCUS. The ZPD is the space the trainee could move through IF they have the proper guidance. Practicing resuscitative POCUS in HFS offers a safe environment for them to reach into their ZPD prior to doing so in real life.
  5. Patient Oriented Outcomes: Do you have any plans for demonstrating this PoCUS HFS has a positive impact on patient oriented outcomes?
    • It keeps coming up but it would have to be multi-centered. If anyone is interested, please get a hold of me.

Is there anything else you wanted to mention Paul?

  • The CAEP Emergency Ultrasound Committee’s Education Working Group is working hard to establish some national objectives and milestones for PoCUS EM in Canadian EM residency. We’re also hoping to introducing CAEP first ever SONOGAMES to CAEP17! We’ll be meeting at CAEP16 to review our progress thus far while also setting our compass for the coming year. Check us out at CAEPultrasound.ca to get in touch with us whether you are interested in being involved or just interested in being kept up to date!

Comment on authors conclusion compared to SGEM Conclusion: We agree with the authors’ conclusions.


SGEM Bottom Line: Consider integrating PoCUS into your high fidelity simulation program.


Case Resolution: You suggest that PoCUS be integrated into a high fidelity simulation case, either by creating an edus2 type trainer with an ultrasound probe and laptop, or simply putting a laptop with ultrasound clips at the bedside.

Clinically Application: PoCUS skills are useful at the bedside in real patients and this integration into high fidelity simulation is the last step prior to using them in real patient encounters.

What do I tell my learners? We know that use of PoCUS during simulation is beneficial for your learning and will help develop your knowledge of PoCUS indications, and your skills of image interpretation, clinical integration of ultrasound images and resuscitation choreography.

Keener Kontest The last winner was Eric Schneider. He knew the tallest building in the world is Burj Khalifa in Dubai, United Arab Emirates. It is 829.8m and 160 stories tall

Listen to the podcast to hear this weeks’ question. Send your answer to TheSGEM@gmail.com The first correct answer will win a cool skeptical prize.

Screen Shot 2015-11-29 at 3.19.26 PMSkeptics’ Guide to Emergency Medicine Hot Off the Press: Now it is time for the SGEMers to join the conversation. What do you think about this SGEMHOP episode? What questions do you have for Paul and his team on PoCUS and high fidelity simulation? Reach out to us on Twitter (#SGEMHOP), Facebook or the SGEM blog. The best social media feedback will be published in Canadian Journal of Emergency Medicine.


Remember to be skeptical of anything you learn, even if you heard it on the Skeptics’ Guide to Emergency Medicine.


 Screen Shot 2016-03-27 at 3.32.09 PM

smaccdub-2-1

  • Chris Hicks

    Hats off to you fellas, a really interesting study and education intervention.

    Couple o’ thoughts:

    1. Now that you know what’s preferred, could you examine what is most effective? Two training streams using the technique you described, standardized/blinded assessment of a) indications, b) image acquisition, c) integration …

    2. Regarding c), one could imagine a hybrid simulation mock up for both training and assessment, using your training module plus either a mannequin or or standardized patient. Nice way to see how the rubber meets the road (ie integration into critical care) without botheringing actual patients …

    Thanks for the read and the review. Great work.

    • Paul Olszynski

      Thanks for sharing your thoughts Chris!
      A second (smaller and subsequently edited out) part of the study involved 4 POCUS educators here in Canada reviewing clips/recordings of all the sessions with standardized evaluation forms. They were tasked with assessing the trainees’ use of ultrasound during resuscitation (through either intervention). They were then asked which intervention allowed for a better assessment of skills (knowledge of indications, image generation and interpretation, and clinical integration). All 4 rated the edus2 more favorably – some suggested that they could even comment on image generation skills by where and by the way the trainees held the simulated probe (which was also brought up in the main findings of the published paper).
      There have also been two abstracts presented on clinical integration since the time of article submission – see below. They suggest improved clinical integration and performance.
      Hoping to build on all of this cool work in the coming years!
      cheers,

      Paul O

  • Simon Carley

    Like Chris I’m interested in the difference between preference and success and also the proximity of assessment.

    I’m reading ‘Making it Stick’ at the moment and in that there are many studies quoted that contradict the idea that preference = performance. In fact the opposite appears to be true.

    Also 2 weeks is quite a short period of time from intervention to assessment (though better than many studies). I’d be very interested to see how this intervention relates to usage in 2-12 months post intervention.

    Thanks once again for some fabulousness (yes it is a word) this Tuesday evening.

    vb

    S

    • Paul Olszynski

      Thanks Simon,
      I agree – preference does not (by itself) equal performance! With this in mind, we designed the study to include both trainee and the instructor ratings. Sure the trainees find it engaging and realistic – but do instructors feel it adds anything to the assessment of the trainee’s skills? Does it help then address observed deficiencies during sim debrief?
      The instructors indicated that both interventions offered insights into various aspects of trainees’ POCUS skill-set. That said, they indicated the edus2 offered a better “view” of just how the trainee intends to use Resuscitative POCUS – whether that be during trauma care (fairly straight forward), or a shock assessment (getting a little more complex) or lastly in arrest (quite the tricky resusc choreography!). The two studies i alluded to in my response to Chris certainly add more evidence to support the performance side – wonder what it will take to get those authors to jump in on the conversation 😉
      cheers,
      Paul O

  • Great episode. We had some experience with EDUS2, a colleague built one from the open source instructions and RFID tags available online. It was trialled for some time on the existing ED registrar sim program. There was quite mixed uptake due, probably, to the varied levels of skill and experience with POCUS. My take – EDUS2 provided a useful step in integrating the cognitive components of POCUS, with a small component of psychomotor realism (probe only has to be in close range of the anatomically placed RFID tag to generat image on laptop (sim scanner). From a simulationist point of view I think the temporal reality added to an immersive sim was excellent, ie. Learners had to get the scanner, place the probe at anatomical positions and evaluate image and integrate findings into diagnosis/treatment plan in real time.
    I’m really one rested in the future planned evaluations and particularly curious as to the optimal time/skill level to integrate EDUS2 without having potentially deleterious effect on the psychomotor skills required for high quality image acquisition (running before walking). Thanks again.

    • Paul Olszynski

      Thanks Jesse,
      Your experience rings true – some novice learners find resusc POCUS integration overwhelming while advanced trainees seem frustrated by the inability to demonstrate their image generation prowess.
      Where, when and how to integrate POCUS into critical care simulation will depend on the nature and structure of the overall training program – but i do have some suggestions (and would welcome feedback)
      NOVICE: ultrasound simulation as a means of opening their minds to the range of POCUS applications in shock/resuscitation. The goal is motivation and awareness – driving them to develop their POCUS image generation skills (on patient volunteers, at scanning days, on shift on stable patients). One way of allowing them to start thinking about its use on their own would be to do something like the edus2 workout. (https://sasksonic.com/education/ )
      INTERMEDIATE: These trainees are now confident in their ability to generate the images on most patients – now they want to start putting the pieces together and are interested in developing a POCUS choreography that fits into their broader resusc choreography. Just like any critical care simulation should have airway equipment, catheters, a defibrillator/AED, etc… there should always be a simulated ultrasound device (which may of course go unused, but the option should be there just like it is in actual emergency care).
      EXPERT: here’s where it gets fun – we can add more challenging cases (both in terms of image interpenetration but also in terms of logistical challenges with use). Examples here include RV collapse in the setting of suspected tamponade – possibly also introducing a pericardiocentesis trainer, POCUS for primary survey in trauma, or introduce some of the limitations of POCUS such as when the patient has significant subcutaneoous emphysema from PTX rendering some views indeterminate, etc..
      With the above there are lots of opportunities for formative assessment and feedback leading towards competence.
      When it comes to assessing competence, with summative assessment moving to include simulation, I suppose another question is whether critical care ultrasound sim could be used for standardized assessment. Anybody?
      Cheers,
      Paul O

      • I think that a “lesson plan” guide that clarifies your insights above would be a great opportunity to help educators immediately translate your findings into their practice.

        Paul, perhaps this is something you can do in conjunction with folks like the EMSimCases.com crew to KT your work out?

        I think lesson guides that help build in new knowledge about our teaching methods are important for changing educator practice. Similarly, I think it is important for our researcher colleagues to also provide us suggested order sets and clinical pathways that help us to fold their new ideas into our clinical practice.

        Thanks for some great work Paul!

        • Paul Olszynski

          Thanks for the great advice Teresa – i will follow up with EMSimCases!

  • Rob Woods

    Congrats Paul on getting this published. Our residency program has been using the EDUS2 since its initial development as part of our regular SIM curriculum and I believe it has dramatically enhanced the fidelity of the scenarios. It is so easy to break the fiction contract when you get to the point of the SIM where you need a diagnostic test result. Instead now, the residents have to put the probe on the area of interest, interpret the results and act on them. Prior to this, we would simply have the microphone from above or a confederate in the room give them the answer. At this point, the next step in management became obvious as the undoubtedly correct answer to the US image led them to the next step. Having the residents actually interpret the US image themselves keeps the fiction contract alive, and maintains the real personal doubt of interpreting an image in front of peers/colleagues. EDUS2 over a laptop maintains the time sequence reality of image generation. A laptop also takes you away from the patient (we often see ALL the learners leave the patient and look at the laptop screen), whereas the EDUS2 has you remain in intimate contact with the patient, maintaining fidelity.

    • Paul Olszynski

      Having spent every other Monday afternoon running sims together (and over a 2 year stretch at that!) – you and I bounced A LOT of ideas back and forth, ultimately bringing about this project. Thanks Rob!

  • Pingback: SGEM-HOP Ultrasound in High Fidelity Simulation - SOCMOB Blog()

  • More to come in this. We are finishing up TraumaSim looking at how introducing edus2 into trauma sims improves diagnostic accuracy and confidence- it does!

    • Paul Olszynski

      Great to hear from you Paul – looking forward to reading the TraumaSim paper!
      You and your team out in St. John having been doing all kinds of cool work with hybrid simulation for quite some time now using various ultrasound simulators within the context of a resusc simulation case (for those interested – see response to Chris Hicks below to see Ultra-sim poster abstract).
      Having used a range of simulators, including task trainers that offer image generation opportunity and others that don’t, any words of wisdom to other programs?
      Cheers,
      Paul O

  • Will Sanderson

    Congrats on the study. We’ve not incorporated this into our program yet, but I definitely see the appeal. Would love to see a lesson plan!

    Piggybacking on Simon’s comment, my biggest concern with learner preference as primary outcome is that preferences don’t seem to have a positive correlation with learner performance (and in fact may have a negative correlation.) It would be interesting to see how to incorporate disfluency interventions here.

    Keep up the great work!

    • Paul Olszynski

      Thanks Will – lesson plan will be forthcoming, but first i must say you’ve got me interested in disfluency! Though i know little about it, I am going to take a stab at it using an example from the study.
      In our trauma case (high speed MVA), the first FAST clips were normal (no free fluid). This despite the fact that the simulated patient was complaining of some pain on the RUQ associated with inspiration. As the scenario plays on, the patient becomes unstable and then on repeat FAST the trainee would encounter free fluid in the RUQ.
      Of the eight times we ran that trauma scenario, on three occasions the trainee leading the case convinced him/herself that the first FAST scan was positive for free fluid (despite it clearly being normal). We as a study team found this very interesting! Though our sample was too small to draw any major conclusions, we did come up with a couple hypotheses:
      1. The trainee lacked sufficient image interpretation skill to properly assess the image (this alone seemed unlikely given the level of knowledge demonstrated by the entrance exam)
      2. The high pre-test probability (combined with mid-level knowledge) led the trainee to convince themselves that there was free fluid (a form of cognitive error?).
      Since the study, we have been using “unexpected normal” findings in cases (here and there) as a means of addressing the importance of objective image interpretation.
      Finally – while I appreciate the concerns raised about learner preference, i think we have to keep in mind that simulation based training is a bit unique. If you ask trainees during a sim session how they feel about doing it, they will tell you its stressful, they are anxious, that its hard. Afterwards you find them gushing over how great it was to have the opportunity to test their metal in the relatively safe environment of the sim suite. Similarly, just because they preferred a given sim intervention, it doesn’t necessarily translate that is was easier on them – but perhaps that they felt they learned more. The instructor feedback and the other studies mentioned below would support that position.
      Cheers,
      Paul O
      Would the above count as a form of disfluency?

  • TheSGem

    Excited to have this episode selected by Life in the Fast Lane as this weeks’ Best of Medical Education and Social Media http://lifeinthefastlane.com/litfl-review-231/

  • Kirsty Challen

    Congrats on the study and the podcast. My attempt to get all your info into a single #paperinapic.

  • Erica Beatty

    Hi Dr. Olszynski, Erica Beatty here, resident in the FRCPC program at ULaval. I am interested in attending your meeting at CAEP this coming week if possible. I know my message is a little last minute but I am a bit behind in my podcasts. I believe PoCUS ultrasound simulation would be a great project to bring to our residency program here. Unfortunately the link provided for CAEPultrasound.ca requires a password. Could you please pass the meeting details along to me via email. Thank you so much. See you at CAEP.

  • Pingback: in situ simulation Workshop #smaccDUB | FOAMina()

  • Pingback: in situ simulation workshop at #smaccDUB – foam-europe.org()

  • Pingback: Global Intensive Care | in situ simulation workshop at #smaccDUB()

  • Pingback: SGEM#153: Simulation for Ultrasound Education | Online Medical Assistant Training Programs()

  • Pingback: LITFL Review 231 • LITFL: Life in the Fast Lane Medical Blog()