SGEM Xtra: EBM Master Class – McGill University Grand Rounds 2020

Subscribe: RSS

Date: January 4th, 2021

This is an SGEM Xtra episode. I had the honour of presenting at the McGill University Emergency Medicine Academic Grand rounds. They titled the talk “Evidence-Based Medicine Master Class”. The presentation is available to watch on YouTube, listen to on iTunes and all the slides can be downloaded (McGill 2020 Part 1 and McGill 2020 Part 2).

Five Objectives:

Look at the burden of proof and talk about what is science
Discuss EBM and give a five step process of critical appraisal
Talk about biases and logical fallacies
Do a check list for randomized control trials
Record a live episode of the SGEM

1) Who has the Burden of Proof and What is Science?

Those making the claim have the burden of proof. It is called a burden because it hard – not because it is easy. We start with the null hypothesis (no superiority). Evidence is presented to convince us to reject the null and accept there is superiority to their claim. If the evidence is convincing we should reject the null. If the evidence is not convincing we need to accept the null hypothesis.

It is a logical fallacy to shift the burden of proof onto those who say they do not accept the claim. They do not have to prove something wrong but rather not be convinced that the claim is valid/“true” and this is an important distinction in epistemology.

What is science? It is the most reliable method for exploring the natural world. There are a number of qualities of science: Iterative, falsifiable, self-correcting and proportional.

What science isn’t is “certain”. We can have confidence around a point estimate of an observed effect size and our confidence should be in part proportional to the strength of the evidence. Science also does not make “truth” claims. Scientists do make mistakes, are flawed and susceptible to cognitive biases.

Physicians took on the image of a scientist by co-opting the white coat. Traditionally, scientists wore beige and physicians wore black to signify the somber nature of their work (like the clergy). Then came along the germ theory of disease and other scientific knowledge.

It was the Flexner Report in 1910 that fundamentally changed medical education and improved standards. You could get a medical degree in only one year before the Flexner Report. The white coat was now a symbol of scientific rigour separating physicians from “snake oil salesman”.

Many medical schools still have white coat ceremonies. However, only 1 in 8 physicians still report wearing a white lab coat today (Globe and Mail).

Science is usually iterative. Sometimes science takes giants leaps forward, but usually it takes baby steps. You probably have heard the phrase “standing on the shoulders of giants”? In Greek mythology, the blind giant Orion carried his servant Cedalion on his shoulders to act as the giant’s eyes.

The more familiar expression is attributed to Sir Isaac Newton, “If I have seen further it is by standing on the shoulders of Giants.” It has been suggested that Newton may have been throwing shade at Robert Hooke.

Hooke was the first head of the Royal Society in England. Hooke was described as being a small man and not very attractive. The rivalry between Newton and Hooke is well documented. The comments about seeing farther because of being on the shoulders of giants was thought to be a dig at Hooke’s short stature. However, this seems to be gossip and has not been proven.

Science is also falsifiable. If it is not falsifiable it is outside the realm/dominion of science. This philosophy of science was put forth by Karl Popper in 1934. A great example of falsifiability was the claim that all swans are white. All it takes is one black swan to falsify the claim. There are some philosophers that refute Popper’s claim about falsifiability.

Science is self-correcting. Because science is iterative and falsifiable it is also self correcting. Science gets updated. We hopefully learn and get closer to the “truth” over time. Medical reversal is a thing and there is a great book and by Drs. Prasad and Cifu on this issue called Ending Medical Reversal: Improving Outcomes, Saving Lives.

The evidence required to accept a claim should be in part proportional to the claim itself. The classic example was given by the famous scientist Carl Sagan (astronomer, astrophysicist and science communicator). Did the TV series Cosmos and wrote a number of popular science books (The Dragons of Eden). Sagan made the claim that there was a “fire-breathing dragon that lives in his garage”.

The quality of evidence to convince you of something should be in part proportional to the claim being asserted. The summary is the famous quote by Carl Sagan that “extraordinary claims require extraordinary evidence”.

Science does not make claims about the truth. It gives an approximation of the the best point estimate of the observed effect. It’s the best known method for exploring the natural world. Science has no agency but rather it is a process. However, scientists are flawed individuals who make mistakes. As Blaise Pascal said: “There is not such thing as the truth, we can only deliver the best available evidence and calculate a probability”.

Real World Example:

Marik et al made the claim in 2016 that vitamin C cocktail (hydrocortisone, thiamine and vitamin C) could cure sepsis. He published a before and after observational study with 94 patients. The result was a 32% absolute decrease in mortality (NNT 3). We covered this study on SGEM#174: Don’t Believe the Hype – Vitamin C Cocktail for Sepsis. Dr. Jeremy Faust (FOAMCast) and I had eleven other skeptics comment on Dr. Marik’s study. Our bottom line was that vitamin C, hydrocortisone and thiamine was associated with lower mortality in severe septic and septic shock patients in this one small, single centre retrospective before-after study but causation has yet to be demonstrated.

Higher-quality studies have since been published looking at they issue. Putzu et al had a SRMA of RCTs including critically ill patients (not just sepsis). They found no statistical difference in mortality. This was covered in SGEM#268: Vitamin C Not Ready for Graduation to Routine Use.

There has been a RCT published by Fujii et al in JAMA 2020. It specifically looked at 216 patients with septic shock and found no statistical mortality benefit to vitamin C.

Has the burden of proof been met that vitamin C is a cure for sepsis? I am not convinced by the available evidence. Note that this is different than claiming vitamin C does not work. That would shift the burden of proof. I am simple accepting the null hypothesis of no mortality superiority of vitamin C compared to placebo in septic patients.

It is ok to say “I don’t know” if vitamin C works. It reminds me of a quote from Dr. Richard Feynman. I have degrees of confidence or certainty about various positions. These positions are tentative and subject to change. I am not absolutely certain about anything. To be absolutely certain could be considered a logical fallacy (nirvana fallacy). Logical fallacies will be discussed later.

2) Evidence-Based Medicine and a Five Step Process to Critical Appraisal

This was defined by Dr. David Sackett over 20 years ago (Sackett et al BMJ 1996). He defined EBM as “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.” I really like this definition, and the only tweak I would have added would be to include the word “shared“.

The definition of EBM can be visually displayed as a Venn diagram. There are three components: The literature, our clinical judgement, and the patients values/preferences.

Many people make the mistake of thinking that EBM is just about the scientific literature. This is not true. You need to know about the relevant scientific information. The literature should inform our care but not dictate our care.

Clinical judgement is very important. Sometimes you will have lots of experience and other times you may have very limited experience.

The third component of EBM is the patient. We need to ask them what they value and prefer. The easiest way to do this is to ask the patient. It should start with patients care and it ends with patient care. We all want patients to get the best care, based on the best evidence.

Levels of Evidence:

There is a hierarchy to the evidence and we want to use the best evidence to inform our patient care. The levels of evidence is usually described using a pyramid. The lowest level is expert opinion. the middle of the hierarchy is a randomized control trial and the top is considered a systematic review.

The systematic review +/- a meta-analysis is put on the top of the EBM level of evidence pyramid. However, we need to watch out for garbage in, garbage out (GIGO). This means if you take a number of crappy little studies (CLS), mash them all up into a meat grinder and spit out a point estimate down to the 5th decimal place that results is some impressive p-value is an illusion of certainty when certainty does not exist.

EBM Limitations:

Harm and the parachutes argument – Smith and Pell BMJ 2003, Hayes et al CMAJ 2018, and Yeh et al BMJ 2018
Most published research findings are false – Ioannidis PLoS 2005
Guidelines are just cookbook medicine
Good evidence is ignored
Too busy for EBM

Five Alternatives to EBM:

This was adapted from a paper by Isaacs and Fitzgerald BMJ 1999. To paraphrase Sir Winston Churchill, EBM is the worst form of medicine except for all the others that have been tried.

Eminence Based Medicine – The more senior the colleague, the less importance he or she placed on the need for anything as mundane as evidence. Experience, it seems, is worth any amount of evidence. These are the senior physicians on staff that make the “same mistakes with increasing confidence over an impressive number of years.”
Vehemence Based Bedicine – The substitution of volume for evidence as an effective technique for brow beating your colleagues and for convincing relatives of your ability. The quality of the evidence is more important than the quantity of evidence.
Eloquence Based Medicine – This is the physician with the year round suntan, Armani suit, pocket handkerchief and tongue that is as silky smooth as his silk tie. Sartorial and verbal eloquence should be no substitute for high-quality, clinically relevant evidence demonstrating a patient oriented outcome.
Nervousness Based Medicine – Fear of litigation is a powerful stimulus for over investigation and over treatment. In an atmosphere of litigation phobia (shame and blame), the only bad test is the test you didn’t think of ordering.
Confidence Based Medicine – Not to be confused with competency based medicine. Confidence based medicine is usually restricted to surgeons.

Five Steps to Critical Appraisal:

Step 1 – Frame your question using a PICO (population, intervention, control and outcome)
Step 2 – Search strategy (TRIP Database, Cochrane Collaboration, and Clinicaltrials.gov)
Step 3 – Find the least bias evidence
Step 4 – Critically appraise the literature (cebm.net and SGEM Xtra: Make it So)
Step 5 – Decide if the evidence is practice changing

3) Biases and Logical Fallacies

Bias is a systematic error in thinking that affects our decisions and judgments. We all have biases, one of mine is that star trek is better than Star Wars. Another one is that pineapple should not be on pizza. This is a hill I’m prepared to die on.

There are many different types of cognitive biases. A cognitive bias is a type of error in thinking that occurs when people are processing and interpreting information in the world around them.

One of my favourites bias is the Swedish one called the Ikea bias. We value things more if we had a hand in building something. So your research always seems more important to you than perhaps to others.

Other examples are recall bias (remember our hits and forget our misses). Hindsight bias (Monday morning quarter back) and availability bias (if you have a hammer everything looks like a nail).

Bias in research is something that systematically moves us away from the “truth” (best point estimate of the observed effect). Dr. Anthony Crocco used a sail boat as an analogy for bias. The truth lies straight ahead of the boat. The waves are like random noise in the data and the wind represents the bias pushing us off-course from our destination (the true effect size).

There are different types of bias in research. Kohn et al AEM 2013 has a great paper that can help people understand the direction of bias in studies of diagnostic test accuracy. Here are five examples of bias in diagnostic studies.

Incorporation Bias – Occurs when results of the test under study are actually used to make the final diagnosis. This makes the test appear more powerful by falsely raising the sensitivity and specificity.
Partial verification bias (Referral bias, Work-up bias) – This happens when only a certain set of patients who underwent the index test is verified by the reference standard. As an example, patients with suspected coronary artery disease whose exercise test results are positive may be more likely to undergo coronary angiography (the reference standard) than those whose exercise test results are negative. This would increases sensitivity but decreases specificity.
Differential Verification Bias (Double gold standard) – This occurs when the test results influence the choice of the reference standard. So a positive index test get an immediate/gold standard test whereas the patients with a negative index test get clinical follow-up for disease. This can raise or lower sensitivity/specificity
Spectrum Bias – Sensitivity depends on the spectrum of disease, while specificity depends on the spectrum of non-disease. So you can falsely raise sensitivity if the clinical practice has lots of very sick people (sicker than who you see in the ED). Specificity can look great if you have no sick patients in the cohort (worried well).
Imperfect Gold Standard Bias (Copper standard bias): This is what can happen if the “gold’ standard is not that good of a test. False positives and false negatives can really mess up results

Logical Fallacies:

Logical fallacies and cognitive biases are slightly different. A logical fallacies require an argument whereas cognitive biases (heuristics- mental shortcuts) refer to our default pattern of thinking. Sometimes they can crossover. Logical fallacies can be the result of a cognitive bias, but having biases does not mean that we have to commit logical fallacies.

Logical fallacies can be dichotomized into formal and informal fallacies. Formal fallacies are errors of logic: the conclusion doesn’t really “follow from” (is not supported by) the premises. Either the premises are untrue or the argument is invalid.

Informal fallacies take many forms and are widespread in everyday discourse. Very often they involve bringing irrelevant information into an argument or they are based on assumptions that, when examined, prove to be incorrect.

Formal fallacies are created when the relationship between premises and conclusion does not hold up or when premises are unsound; informal fallacies are more dependent on misuse of language and of evidence.

Check list for randomized control trials

There are a number of check lists that can be used to probe the literature for its validity. A number of BEEM critical appraisal forms are available at the SGEM Xtra Episode: Make it So. The Centre for Evidence-Based Medicine in Oxford, England also has a number of forms available at their CEBM website.

Here are the eleven questions used for the Skeptics’ Guide to Emergency Medicine critical appraisal of RCTs.

The study population included or focused on those in the emergency department.
The patients were adequately randomized.
The randomization process was concealed.
The patients were analyzed in the groups to which they were randomized.
The study patients were recruited consecutively (i.e. no selection bias).
The patients in both groups were similar with respect to prognostic factors.
All participants (patients, clinicians, outcome assessors) were unaware of group allocation.
All groups were treated equally except for the intervention.
Follow-up was complete (i.e. at least 80% for both groups).
All patient-important outcomes were considered.
The treatment effect was large enough and precise enough to be clinically significant.

5) SGEM#301: You Can’t Stop GI Bleeds with TXA

This was an SGEM Journal Club episode recorded live at McGill University Grand Rounds. This was the third time coming to McGill University Department of Emergency Medicine to give Grand Rounds. The first visit was back in 2013 for SGEM#50: Under Pressure – Vasopressin, Steroids and Epinephrine in Cardiac Arrest. The SGEM bottom line was this was interesting, but VSE protocol was not ready for routine use.

The second visit was SGEM#176: Somebody’s Watching Me – Cardiac Monitoring for Chest Pain. We were trying to answer the question: Do all patients presenting to the emergency department with chest pain need to be placed on cardiac monitoring or could some be safely removed? The SGEM bottom line was that for some patients presenting with chest pain who are chest pain free and have normal/non-specific ECG findings could potentially be safely removed from cardiac monitoring using the Ottawa CPCM Rule.

Guest skeptics for SGEM#301 were Dr. Robert Goulden and Dr. Audrey Marcotte. They are the Chief Residents from the Royal College of Emergency Medicine Program at McGill University.

The clinical question was does treatment with TXA reduce the mortality of patients with upper or lower GI bleeds? The reference was Roberts et al published in The Lancet 2020. The SGEM bottom line was that the latest evidence does not support the use of TXA in GI bleeds.

Take Home Message:

Don’t Panic – EBM, critical appraisal, logical thinking gets better with practice
EBM Rocks – Once you get a real taste of this type of practice it is hard to ever go back to other forms of medical practice.
Be Skeptical – Even of the SGEM. It does not mean you are closed minded. Rather you should be very open minded. Just not so open your brain falls out. The time to believe something is when there is sufficient evidence.

Again, the presentation is available to watch on YouTube, listen to on iTunes and all the slides can be downloaded (McGill 2020 Part 1 and McGill 2020 Part 2).

The SGEM will be back soon with another episode doing a structured critical appraisal of a recent publication. Trying to cut the knowledge translation window down from over ten years to less than one year using the power of social media. So patients get the best care, based on the best evidence.