Podcast: Play in new window | Download

Subscribe: RSS

Date: December 21st, 2021

Guest Skeptic: Dr. Howard “Howie” Mell began his career as a firefighter / paramedic in Chicago. He became double board certified in Emergency Medicine (EM) and Emergency Medical Services (EMS). Howie also has a Master of Public Health.

Reference: Grotta JC et al. Prospective, multicenter, controlled trial of mobile stroke units. NEJM 2021

Case: The Mayor of your community consults you as an expert in public health, EMS and as an EM physician on whether they should purchase a mobile stroke unit (MSU) ambulance.

Background: No one who has listened to the SGEM will be surprised we are covering another paper looking at stroke. We have often discussed the use of thrombolysis for acute ischemic stroke (AIS) with or without endovascular therapy (EVT). However, the SGEM has also looked at secondary stroke prevention on previous episodes (SGEM#24, SGEM#303).

The SGEM has looked at pre-hospital stroke care using early administration of nitroglycerin by paramedics to see if it would improve neurologic outcome in patients with a presumed acute stroke (SGEM#269). The results from the RIGHT-2 trial reported no statistical difference in functional outcome as measured by the modified Rankin Scale (mRS) score at 90 days.

The SGEM bottom line was that very early application of transdermal nitroglycerin by paramedics in the pre-hospital setting cannot be recommended at this time in patients with a suspected stroke.

Mobile Stroke Unit

The issue of having a MSU has also been discussed on  SGEM#330. A systematic review and meta-analysis which included seven randomized controlled trials and four observational studies including 21,297 patients was critically appraised.  The primary outcomes reported better neurologic outcome at seven days but not at one day post treatment by a MSU compared to conventional care (Fatima et al Int J Stroke 2020).

The SGEM bottom line from that episode was we cannot recommend the use of MSU based on the available evidence.

Clinical Question: Should mobile stroke units be purchased and deployed in your community?

Reference: Grotta JC et al. Prospective, multicenter, controlled trial of mobile stroke units. NEJM 2021

  • Population: Patients calling EMS with a history and physical/neurological examination consistent with acute stroke who is last seen normal (LSN) possibly within 4 hours and 30 minutes and who had no definite tPA exclusions per guidelines, prior to CT scan or baseline labs. Daytime hours and mostly weekdays.
  • Intervention: Care by a mobile stroke unit (MSU)
  • Comparison: Care by traditional EMS referred to as standard management (SM)
  • Outcome:
    • Primary Outcome: Score on the utility-weighted modified Rankin scale (uw-mRS) at 90 days in patients who were adjudicated to be eligible to receive tPA on the basis of subsequent blinded review
    • Secondary Outcomes: There were twelve secondary endpoints in their final protocol listed in hierarchical sequence of importance
      • Agreement between on-board vascular neurologists (VN) and the remote VN
      • Exploratory cost-effectiveness analysis (CEA)
      • Outcomes comparing patients found eligible for tPA on MSU weeks compared to patients on SM weeks
        • Ordinal (shift) analysis of mRS at 90 days, and
        • Proportion of patients achieving 90 day mRS 0,1 vs 2-6
        • 30% improvement from baseline to 24hr NIHSS
      • Outcomes comparing all patients treated with tPA (whether or not adjudicated as tPA eligible) on MSU weeks compared to patients on SM weeks.
      • Uw-mRS at 90 days
        • Ordinal (shift) analysis of mRS at 90 days, and
        • Proportion of patients achieving 90 day mRS 0,1 vs 2-630%
        • Improvement from baseline to 24hr NIHSS
      • Outcomes of those treated within 60 min LSN compared to those treated from 61 to 270 minutes
        • Change in uw-mRS from baseline at 90 days
        • Ordinal shift analysis of MRS at 90 days
        • Proportion of patients achieving 90 day mRS 0,1 vs 2-6
        • 30% improvement from baseline to 24hr NIHSS
      • Outcomes all patients treated with IAT (separate analyses for those adjudicated as tPA eligible, all tPA treated, or all IAT with or without tPA) on MSU weeks compared to patients on SM weeks.
      • Uw-mRS at 90 days
        • Ordinal (shift) analysis of mRS at 90 days, and
        • Proportion of patients achieving 90 day mRS 0,1 vs 2-6
        • 30% improvement from baseline to 24hr NIHSS
      • The time from LSN to tPA treatment on all patients treated within 4.5 hours of LSN on MSU weeks compared to similarly eligible patients on SM weeks
      • Proportion of patients treated within 60 minutes of LSN on MSU weeks vs SM weeks.
      • The time from LSN and from ED arrival to start of endovascular procedure on MSU vs SM weeks
      • Proportion of all tPA-eligible patients having EVT on MSU vs SM weeks
      • The median/mean time from LSN to tPA therapy decision on all patients considered for treatment within 4.5 hours of LSN on MSU weeks compared to SM weeks
      • Time between 911 call and onset of etiology-specific BP management on MSU vs SM weeks.
    • Safety Endpoints:
      • Incidence of symptomatic intracranial hemorrhage (sICH) in enrolled tPA treated patients on MSU weeks compared to SM weeks. sICH was defined as any intracranial blood accumulation associated with a clinical deterioration of 4 points of the NIHSS for which the hemorrhage has been identified as the dominating cause of the neurologic deterioration)
      • Mortality up to one year
      • Incidence of stroke mimics and transient ischemic attacks (TIAs) in tPA treated patients on MSU weeks compared to SM weeks.
  • Trial: Prospective cohort study with cluster randomized deployment weeks and blinded assessment of both trial entry and clinical outcomes. Cluster randomization can have both strengths and weaknesses just like any study design. For those less familiar with this methodology Taljaard and Grimshaw wrote a good article the topic in 2014.

Authors’ Conclusions: “In patients with acute stroke who were eligible for t-PA, utility-weighted disability outcomes at 90 days were better with MSUs than with EMS.”

Quality Checklist for Observational Studies:

  1. Did the study address a clearly focused issue? Yes
  2. Did the authors use an appropriate method to answer their question? No
  3. Was the cohort recruited in an acceptable way? No
  4. Was the exposure accurately measured to minimize bias? Yes
  5. Was the outcome accurately measured to minimize bias? No
  6. Have the authors identified all-important confounding factors? No
  7. Was the follow up of subjects complete enough? Yes
  8. How precise are the results? Unsure
  9. Do you believe the results? No
  10. Can the results be applied to the local population? No
  11. Do the results of this study fit with other available evidence? Yes
  12. Did the study have no conflicts of interest. No

Key Results: This prospective observational study screened 10,443 patients and enrolled 1,515 patients (58.5% MSU vs 41.5% SM). Fourteen percent overall were not eligible for tPA due to intracranial blood seen on CT scan. Two-thirds in both groups (1,047 total) were decided post-hoc to be eligible for tPA. Of the tPA eligible patients, 97% in the MSU group received tPA compared to 79.5% in the SM group.

This results section was a real struggle. It was unclear which primary and secondary outcomes we should highlight in the review. Should it be those published in the NEJM or do we discuss the original ClinicalTrials.gov outcomes, the current ClinicalTrials.gov outcomes or pre-specified published protocol outcomes (Yamal et al Int J Stroke 2018)?

At the end of the day, we decided to provide the published primary outcome, mention the secondary outcomes and give a few of the safety outcomes.

Key Result: Patients treated with a mobile stroke unit had better 90 day outcomes.

  • Primary Outcome (NEJM): Score on the uw-mRS at 90 days in patients who were adjudicated to be eligible to receive tPA on the basis of subsequent (post-hoc) blinded review
    • 0.72 in the MSU group and 0.66 in the SM group
    • Adjusted Odds Ratio (aOR) ≥0.91, 2.43 (95% CI, 1.75 to 3.36; P<0.001).
  • Secondary Outcomes: Among the patients eligible for tPA, 55.0% in the MSU group and 44.4% in the SM group had a score of 0 or 1 on the mRS at 90 days. Among all enrolled patients, the mean score on the uw-mRS at discharge was 0.57 in the MSU group and 0.51 in the SM group (aOR for a score of ≥0.91, 1.82; 95% CI, 1.39 to 2.37; P<0.001). For more secondary outcomes see the NEJM publication.
  • Safety Endpoints:
    • sICH in ~2% of patients who received tPA in each group and none of the patients considered to be stroke mimics.
    • Mortality at 90 days was 8.9% in the MSU group vs 11.9% SM group.

1. Houston, We Have a Problem: They changed their protocol at least four times over the course of the study. These changes were described in the PDF of their protocol. Sometimes the changes were minor and other times they were major. You can also see how their primary outcomes changed on ClinicalTrials.gov, in their pre-published protocol and through to their published manuscript in the NEJM.

We were unable to find the any data in the manuscript or supplemental material on the other three “original” or “current” primary outcomes. This included the kappa value for the agreement between on scene vascular neurologist and remote vascular neurologist, cost effectiveness or the change in uw-mRS from baseline at 90 days. We have reached out to the lead author Dr. Grotta and will update the blog if this information becomes available.

UPDATE: The Cohen kappa was published by Wu et al in 2017 with a value of 0.73 which is considered moderate inter rater reliability according to McHugh 2012. This does not explain why this outcome was considered a primary outcome when the trial was registered in 2014 and in the update in August 2018 and then considered a secondary endpoint in the 2021 published manuscript supplementary protocol material dated 2015.

These multiple changes and selective reporting make me skeptical of the publication. This position is based upon studies by Chan et al JAMA 2004, Chan et al 2004 CMAJ, Dwan et al PLoS 2013, Hartung Annals Int Med 2014, and Chen et al JAMA Network Open 2019.

Here are the details of the changes to the primary outcome over time:

  • Original Primary Outcomes (2014 ClinicalTrials.gov):
    • Time LSN to tPA treatment
    • Agreement between on scene VN and remote VN
    • Cost effectiveness
  • Current Primary Outcomes (2018 ClinicalTrials.gov):
    • Uw-mRS score change from baseline to 90 days
    • Agreement between on scene VN and remote VN
    • Cost effectiveness
  • Published Protocol Co-Primary Outcome (2018):
    • Score on the uw-mRS at 90 days
    • Cost-effectiveness based on two measures
  • Published Primary Outcome (NEJM 2021):
    • Score on the uw-mRS at 90 days
  • Summary of protocol changes, Original protocol, Final protocol (2021)

2. Hours of Operation: The seven different sites operated daytime only and not on Sundays. Most patients came from Houston (77%) which operated from 8 am to 6 pm Monday through Saturday. These restricted hours and days of operation could have contributed to only 2.4 patients/week at the Houston site and 2.4 patients/month at the six other non-Houston sites. We will talk more about these low numbers in another nerdy point. However, it is unclear if these limited times and days can be extrapolated to evenings, nights, and Sundays.

It is also unclear if this data has external validity to a Canadian community setting beyond just the limited hours of operation.

3. Secondary Outcomes: This refers to the first nerdy point. The secondary outcomes were also changed. The original 2014 protocol had just two secondary outcomes. This was updated to a total of six in 2016 on ClinicalTrials.gov. The published protocol with the manuscript lists twelve secondary outcomes plus three safety outcomes for a grand total of fifteen non-primary outcomes (NEJM 2021).

  • Original Secondary Outcomes (2014 ClinicalTrials.gov): Two secondary outcomes
    • mRS at 90d 0,1 vs 2-6 of patients treated with tPA within 60min on either MSU or SM weeks, compared to similar patients treated 61 to 270min and
    • mRS 0,1 vs 2-6 at 90 days of all patients meeting published guidelines for treatment with tPA within 4.5 hours of symptom onset on MSU vs SM weeks, adjusting for any imbalances in stroke severity (baseline NIHSS) between the groups at the time of treatment.
  • Current Secondary Outcomes (2016 ClinicalTrials.gov): Added four more secondary outcomes for a total of six
  • Published Protocol Secondary Outcomes (2018): 90-day mRS; time metrics including LSN, alert, scene arrival and departure, tPA decision, tPA bolus, ED arrival, and start of IAT; healthcare utilization during the first year after the stroke; QoL. Safety outcomes include mortality and symptomatic hemorrhage.
  • Published Secondary Outcomes (2021 NEJM): Lists twelve secondary outcomes plus three safety outcomes for a total of fifteen non-primary outcomes.

Secondary outcomes should usually be considered hypothesis generating. With all the changes made in the secondary outcomes it is hard to have much confidence in the results and their interpretations.

4. Observer’s Paradox or Hawthorne Effect: This study is the observer’s paradox personified. It was impossible to blind the ED staff or patients to the use of an MSU to transport the patient. No outcomes were reported based on the treatment provided (e.g., tPA). The only variable reported on was treatment by the MSU or SM. This introduces a plethora of variables from the Hawthorne Effect to the attention paid to a prehospital report given by a specialist physician or specialty team to the one given by a standard EMS provider.

Compounding this is the fact that ~77% came from one city (Houston), with one EMS system as the control. Without some breakdown as to why the patients treated in the MSUs did better, these data are very difficult to generalize. It is interesting that outcome data was not reported by the type of therapeutics provided (i.e., IV tPA, clot retrieval +/- tPA, IA tPA, or medical management only).

5. Outcome Measure: This study outcome of uw-mRS may be a bit more confusing to some readers. Most of us are familiar with the mRS and even the ordinal shift analysis that is used in some stroke studies.

The uw-mRS translates the seven levels (0 to 6) of the mRS to values between 0 and 1. The distances between the levels reflecting patient and societal valuation of each disability state (Chaisinanunkul et al Stroke 2015). A higher score indicates a better outcome. This is in contrast to the mRS where a higher score indicating more disability.

The authors posit two advantages of using the uw-mRS over the dichotomized mRS and ordinal shift approach. Dichotomizing and ordinal outcome results in loss of information in contrast to the uw-mRS which provides a way to use all points in the scale in a more patient-oriented way. Ordinal analyses also assume that each change between ordinals has the same clinical impact. The uw-mRS adjusts each ordinal to be more patient-oriented.

To make things more complected, different studies will assign different weights to each level making it even harder to compare studies.

6. Table S5: There is an amusing table included in the supplement (Table S5). Now what I find amusing here is that 16.9% of MSU treated patients had their strokes “reversed by tPA” whereas only 8.8% of EMS patients did, but somehow only 0.8% of the MSU patients had TIAs, whereas 4.0% of the SM patients did. It seems that in the post analysis, symptomatic patients seen on the MSU were cured by tPA, instead of the more likely explanation that they would’ve recovered regardless. The other thing this shows is that at least seven patients “ineligible” for tPA in the MSU group and two patients “ineligible” for tPA in the SM group received IV tPA (more on this below).

7. Protocol Violators: The authors report that 16.7% of those in the MSU group and 11.6% of those in the SM group were treated with tPA despite not being eligible for tPA. These are the numbers of patients being exposed to a treatment in the context of a prospective cluster cohort study that has potential harm (sICH). It would be likely the number of protocol violators outside a study would be larger and this would lead to more potential harm. This position is informed by the Cleveland area experience when tPA was first introduced (Katzan et al JAMA 2000). They had 50% of patients deviate from national treatment guidelines (protocol violations) with an in-hospital mortality rate three times higher than those who did not receive tPA (15% vs 5%).

8. Figure S3 Consort Diagram: This is an interesting figure that breaks down the 1,492 patients enrolled. Why were only 350 of the 430 (81.3%) “tPA eligible” patients in the SM arm given tPA or EVT direct when 599 of the 617 (97.1%) “tPA eligible” patients in the MSU arm were? While I do not accept that IV tPA represents the best treatment for these patients, it is certainly clear that the authors do. Given that, why the discrepancy? By the definitions of the protocol, these patients were within the timing window for administration, so why wasn’t tPA given? This provides further evidence that the ED staff responded differently to the MSU than they did to a standard SM unit. Eliminating these differences should be the focus of this paper, not the expansion of MSUs.

9. Change in uw-mRS Score: No data are provided regarding change in uw-mRS scores.  This was one of their three “primary” outcomes listed in the trial registry in 2018 ClinicalTrials.gov. Yet according to their supplemental material published in 2021, they changed the primary outcome on April 18, 2018. The new version changed the primary outcome from a change in mean uw-mRS from pre-stroke to 90 days to mean uw-mRS at 90 days. This is confusing because they published their protocol in 2018 with a primary outcome of mean uw-mRS at 90 days. This changing back and forth of the primary outcome without publishing the results of the change in uw-mRS increases my level of skepticism.

One secondary outcome that they did report and relates to change in function is if there was a 30% reduction in NIHSS score from baseline to 24 hours. They observed such an improvement in 75.0% of the patients eligible for tPA in the MSU group and in 67.8% of those in the SM group (aOR, 1.45 [95% CI, 1.09 to 1.91] with inverse probability weighting and 1.45 [95% CI, 1.10 to 1.93] without inverse-probability weighting). These data are interesting for two reasons: They demonstrate that the data regarding change and degree of change were collected at least for NIHSS score at 24 hours, but nothing is reported on the change in uw-mRS at 90 days.

The other interesting aspect is the improvement at 24 hours in this prospective observational study contrasts with the NINDS RCT which did not report an improvement in NIHSS score at 24 hours. This suggests something is confounding the data and tPA is not responsible for the observed differences between groups in BEST-MSU.

10. Jurassic Park: There is a great line from the early scenes of the 1993 film Jurassic Park: “Your scientists were so preoccupied with whether they could, they didn’t stop to think if they should.” Any conversation of mobile stroke units needs to include this issue in the discussion.

This study ran for five years to get 1,515 patients enrolled. That equals 303 patients per year that possibly could be aided by the intervention. Now let’s look at some of the data.

In 90-day mortality, MSU was 3% better (NNT ~33). Improvement at 24 hours favored MSUs as well with 7.2% more MSU patients improving (NNT ~14). This is interesting because NINDS 1995 did not show a statistically significant benefit with tPA at 24 hours over placebo for their primary outcome using the NIHSS score. The SGEM did a structured critical appraisal of the NINDS trial on SGEM#70.

If we look at the difference in patients with an mRS < 2 at 90 days, the MSU patients do better by 7.2% again (NNT ~14). Now that means that if all 303 patients enrolled annually had received care in an MSU, 22 of them would have better outcomes for it. So far so good. But what is the population protected by the MSUs in this study?

Patients were enrolled from Houston (807), Colorado (100), Memphis (54), New York City (28), Los Angeles (23) Sutter Health in Burlingame, CA (22), and Indianapolis (13). So how many people, if they called 9-1-1 for a stroke during the study period would’ve been eligible to be enrolled? The answer is impossible to know but given the number of sites and the size of the respective cities, using a million people as a benchmark seems more than fair (as the actual number is likely three to five million). If we accept that a million people were protected by these units, and 22 patients would have had improved outcomes (which I’m only accepting for the sake of this argument), then one outcome was improved for every 45,455 persons in the population protected per year.

Data I collected and reported at the 2020 Annual Meeting of the Society of Vascular and Interventional Neurology showed that in previous MSU studies with better defined populations, there was roughly one person treated for roughly every 12,000 – 14,000 persons protected (Berlin 1 per ~12,000 pop per year, Toledo 1 per ~14,000 pop per year, Cleveland 1 treated per ~14,000 pop per year). Those studies focused on tPA administration however, whereas BEST-MSU solely focused on the mere presence of the MSU. With a NNT in those studies of ~7 (if you accept their findings), one outcome was improved for every ~91,000 persons protected per year (which would fit the BEST-MSU data neatly if two million persons were protected by its participants).

Ok, but what does that mean? For the average US EMS agency, millions will have to be spent to improve one outcome a year. Even in high population cities, population density would require multiple MSUs, so again, millions would have to be spent to “save” one person. In most North American cities, putting two or three additional regular EMS units (which could be done for the cost of one MSU) would have much better outcomes in terms of overall population health. Can MSUs improve outcomes? Maybe (probably not). Is the juice worth the squeeze? Absolutely not.

Comment on Authors’ Conclusion Compared to SGEM Conclusion: We don’t disagree with the authors’ conclusion. We disagree with how that conclusion is interpreted in the paper. This paper doesn’t prove that MSUs should be in wider use, it probably proves that the handoff of a suspected stroke patient between EMS and ED staff can be improved on, and this would likely improve patient outcomes.

SGEM Bottom Line: We still cannot recommend the use of MSU even with the addition of the BEST-MSU publication.

Case Resolution: You tell the mayor that given the choice of spending lots of money on one MSU ambulance vs. two to three regular ambulances you would advise the latter. This would not just improve response times for stroke patients but for all patients who call 911.

Dr. Howie Mell

Clinical Application: We do not have high-quality evidence to support the use of MSUs and these initiatives should not be supported by EM or EMS physicians.

What Do I Tell the Mayor?  I would tell a mayor, EMS chief, or health system that MSU’s are an expensive boondoggle that won’t change the health of the population they serve. I would point out that this study suggests that there may be issues with the handoff between EMS and the ED regarding suspected stroke patients that could be improved on to increase favorable outcomes.

Keener Kontest: Last weeks’ winner was a resident from Florida Atlantic University. They knew the adult human body contains approximately 1 kg (2.2 lbs) of calcium.

Listen to the SGEM podcast for this weeks’ question. It is about this about this documentary from EMR called 24/7/365 on the history of emergency medicine. If you know, then send an email to thesgem@gmail.com with keener in the subject line. The first correct answer will receive a cool skeptical prize.

Other FOAMed:

Remember to be skeptical of anything you learn, even if you heard it on the Skeptics’ Guide to Emergency Medicine.