Podcast: Play in new window | Download

Subscribe: RSS

Date: July 1st, 2022

Guest Skeptic: Dr. Ravi Garg is a Neurologist in the Department of Neurology, Division of Neurocritical Care at Loyola University Chicago.

Reference: Garg R, Mickenautsch S. Risk of selection bias assessment in the NINDS rt-PA stroke study. BMC Med Res Methodol. 2022 Jun 15;22(1):172.

This is an SGEM Xtra episode. Dr. Garg saw some tweets about the NINDS trial and sent me his recent publication. I asked him to come on the SGEM and discuss the original NINDS trial, some of the reanalyses and share his analysis of the NINDS data.

One of the criticisms of Emergency Medicine physicians who have done FOAMed post publication reviews of the stroke literature like Dr. Justin Morgenstern, Dr. Ryan Radecki, Dr. Anand Swaminathan and Dr. Salim Razaie, is that we are not neurologists and specifically not stroke neurologists. While this is true, we are part of the team that diagnose and treat acute stroke patients.

The SGEM tries to include a wide variety of clinicians in this knowledge translation project. Great emergency care takes a team from the prehospital setting, emergency department, inpatient and outpatient all working together. That is why we have had paramedics, nurses, physiotherapists, pharmacists and a wide spectrum of physician specialists on the SGEM.

However, until now have we not had a neurologist on the SGEM who has a specialized interest in stroke neurology and published on thrombolysis as a guest skeptic. Dr. Garg sent me his analysis of the NINDS trial that he wrote with his co-author Dr. Steffen Mickenautsch. This new peer reviewed publication is the basis of this SGEM Xtra episode.

The NINDS trial was published back in 1995 and we did a structured critical appraisal of the classic paper with Dr. Anand Swaminathan on SGEM#70.  I was a resident at the time of publication and Dr. Garg was only eight years old.

Dr. Garg was asked a series of questions. You can listen to his responses on the SGEM podcast.

Thoughts on the NINDS Trial and Some of the Reanalyses

  • Dr. Ravi Garg

    Any general thoughts about NINDS trial?

  • One concern about the NINDS trial was the baseline differences in NIHSS score. This resulted in multiple reanalyzes attempting to control for these factors. NINDS commissioned an independent committee to investigate if any of these imbalances invalidated the entire trial. This committee’s findings supported the use of tPA in less than three hours (Ingall et al 2004). What are your thoughts on this commissioned report?
  • Another reanalysis was done by Kwiatkowski et al 2005 that also confirmed that the baseline imbalance in the NINDS trial did not account for the better outcome of tPA-treated patients. Any brief comments on this reanalysis?
  • Hoffman and Schrieger stirred things up a bit with their graphic reanalysis of the NINDS trial using the NIHSS score. They published their findings in Annals of EM 2009. The results questioned the effect of tPA for acute ischemic stroke in patients treated within three hours. The graphs created in the publication also failed to support the “time-is-brain” hypothesis. There are some criticisms of this graphic reanalysis. What are your thoughts on this contrarian view?
  • Saver et al responded to Hoffman and Schrieger’s graphic reanalysis in Academic Emergency Medicine 2010. They pointed out number concerns with the publications. Did Saver and colleagues make some sound arguments?

Ravi Garg and Steffen Mickenautsch BMC June 2022

The title of your paper is Risk of selection bias assessment in the NINDS rt-PA stroke study. It was published in BMC Medical Research Methodology, June 2022.

  • With all the other reanalyses, what motivated you to do this another reanalysis of the NINDS trial?
  • You were able to get patient level data for this review. Why is that important?
  • What tool did you use to assess the NINDS trial for risk of selection bias?
  • Can you walk us through the Cochrane Risk of Bias-2 (RoB-2) tool that address systematic error arising from the randomization process?
  • You did four sensitivity analyses based on the randomization process using participant level data. Briefly what were the four analyses?
  • What did you do to assess the potential effect of baseline imbalances on reported alteplase treatment effects?
  • What were the results of your study on the NINDS trial?
  • What did you discover with the four sensitivity analyses?
  • You adjusted for the differences found in the sensitivity analyses. How did that impact the results?
  • Why is unbiased randomization so important in RCTs?
  • What points do you want to highlight from your discussion
  • What do you think the limitations are to your study?

Conclusions to this New Analysis of the NINDS Trial Data

  • What conclusions did you draw from your assessment of the NINDS trial?
  • What does this high risk of selection bias due to your certainty about this data?
  • You conclude the imbalances seen in the NINDS trial were not noise (random error) in the data but rather an error in randomization. This can bias the results and move us away from the “truth” (the best point estimate of an observed effect size with a confidence interval around that effect size). So the results are fuzzier and less certain?
  • This error in randomization would then be passed along into any systematic review and meta-analysis (SRMA) done on this topic. Could this bias a SRMA even if it used individual patient data which is considered the “gold standard” by Cochrane?
  • Some of those convinced of the efficacy of tPA for acute ischemic stroke will say it is unethical to perform a placebo controlled RCT due to a lack of equipoise. How do you respond to that argument?
  • How should we apply your paper clinically?

Dr. Ravi Garg’s Bottom Line: I’m very skeptical about the results in the NINDS study and thrombolytic studies for stroke in general.

The SGEM will be back next episode doing a structured critical appraisal of a recent publication. Trying to cut the knowledge translation window down from over ten years to less than one year using the power of social media. So, patients get the best care, based on the best evidence.

Remember to be skeptical of anything you learn, even if you heard it on the Skeptics’ Guide to Emergency Medicine

Additional Reading:

  • Shinton R. Questions about authorisation of alteplase for ischaemic stroke. Lancet. 2014 Aug 23;384(9944):659-60. doi: 10.1016/S0140-6736(14)61385-4. PMID: 25152265.
  • Appelros P, Terént A. Thrombolysis in acute stroke. Lancet. 2015 Apr 11;385(9976):1394. doi: 10.1016/S0140-6736(15)60714-0. PMID: 25890417.
  • Mickenautsch S, Fu B, Gudehithlu S, Berger VW. Accuracy of the Berger-Exner test for detecting third-order selection bias in randomised controlled trials: a simulation-based investigation. BMC Med Res Methodol. 2014 Oct 6;14:114. doi: 10.1186/1471-2288-14-114. PMID: 25283963; PMCID: PMC4209086.
  • Austin PC, Tu JV. Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality. J Clin Epidemiol. 2004 Nov;57(11):1138-46. doi: 10.1016/j.jclinepi.2004.04.003. PMID: 15567629.
  • Goyal M, Menon BK, van Zwam WH, et al.  HERMES collaborators. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. Lancet. 2016 Apr 23;387(10029):1723-31. doi: 10.1016/S0140-6736(16)00163-X. Epub 2016 Feb 18. PMID: 26898852.
  • Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995 Feb 1;273(5):408-12. doi: 10.1001/jama.273.5.408. PMID: 7823387.
  • Johnstone C. Thrombolysis for acute ischemic stroke: does it work?–the con position. CJEM. 2015 Mar;17(2):180-3. doi: 10.1017/cem.2015.14. Erratum in: CJEM. 2015 Sep;17 (5):600. PMID: 26052969.
  • Berger VW, Exner DV. Detecting selection bias in randomized clinical trials. Control Clin Trials. 1999 Aug;20(4):319-27. doi: 10.1016/s0197-2456(99)00014-8. PMID: 10440559.
  • Estruch R, Ros E, Salas-Salvadó J, Covas MI, Corella D, Arós F, Gómez-Gracia E, Ruiz-Gutiérrez V, Fiol M, Lapetra J, Lamuela-Raventos RM, Serra-Majem L, Pintó X, Basora J, Muñoz MA, Sorlí JV, Martínez JA, Martínez-González MA; PREDIMED Study Investigators. Primary prevention of cardiovascular disease with a Mediterranean diet. N Engl J Med. 2013 Apr 4;368(14):1279-90. doi: 10.1056/NEJMoa1200303. Epub 2013 Feb 25. Retraction in: N Engl J Med. 2018 Jun 21;378(25):2441-2442. Erratum in: N Engl J Med. 2014 Feb 27;370(9):886. Corrected and republished in: N Engl J Med. 2018 Jun 21;378(25):e34. PMID: 23432189.
  • Estruch R, Ros E, Salas-Salvadó J, Covas MI, Corella D, Arós F, Gómez-Gracia E, Ruiz-Gutiérrez V, Fiol M, Lapetra J, Lamuela-Raventos RM, Serra-Majem L, Pintó X, Basora J, Muñoz MA, Sorlí JV, Martínez JA, Fitó M, Gea A, Hernán MA, Martínez-González MA; PREDIMED Study Investigators. Primary Prevention of Cardiovascular Disease with a Mediterranean Diet Supplemented with Extra-Virgin Olive Oil or Nuts. N Engl J Med. 2018 Jun 21;378(25):e34. doi: 10.1056/NEJMoa1800389. Epub 2018 Jun 13. PMID: 29897866.
  • Hicks A, Fairhurst C, Torgerson DJ. A simple technique investigating baseline heterogeneity helped to eliminate potential bias in meta-analyses. J Clin Epidemiol. 2018 Mar;95:55-62. doi: 10.1016/j.jclinepi.2017.10.001. Epub 2017 Oct 13. PMID: 29032245.