Podcast: Play in new window | Download

Subscribe: RSS

Date: October 21st, 2020

This is an SGEM Xtra episode. I had the honour of presenting at the Department of Family Medicine’s Grand Rounds at the Schulich School of Medicine and Dentistry. The title of the talk was: How to think, not what to think. The presentation is available to watch on YouTube, listen to on iTunes and all the slides can be downloaded from this LINK.

Five Objectives:

  1. Discuss what is science
  2. Talk about who has the burden of proof
  3. Discuss Evidence-based medicine (EBM), limitations and alternatives
  4. Provide a five step approach to critical appraisal
  5. Briefly talk about COVID19 and the importance of EBM

What is Science?

It is the most reliable method for exploring the natural world. There are a number of qualities of science: Iterative, falsifiable, self-correcting and proportional.

What science isn’t is “certain”. We can have confidence around a point estimate of an observed effect size and our confidence should be in part proportional to the strength of the evidence. Science also does not make “truth” claims. Scientists do make mistakes, are flawed and susceptible to cognitive biases.

Physicians took on the image of a scientist by co-opting the white coat. Traditionally, scientists wore beige and physicians wore black to signify the somber nature of their work (like the clergy). Then came along the germ theory of disease and other scientific knowledge.

It was the Flexner Report in 1910 that fundamentally changed medical education and improved standards. You could get a medical degree in only one year before the Flexner Report. The white coat was now a symbol of scientific rigour separating physicians from “snake oil salesman”. 

Many medical schools still have white coat ceremonies. However, only 1 in 8 physicians still report wearing a white lab coat today (Globe and Mail).

Science is Usually Iterative:

Sometimes science takes giants leaps forward, but usually it takes baby steps. You probably have heard the phrase “standing on the shoulders of giants”? In Greek mythology, the blind giant Orion carried his servant Cedalion on his shoulders to act as the giant’s eyes.

The more familiar expression is attributed to Sir Isaac Newton, “If I have seen further it is by standing on the shoulders of Giants.” It has been suggested that Newton may have been throwing shade at Robert Hooke.

Hooke was the first head of the Royal Society in England. Hooke was described as being a small man and not very attractive. The rivalry between Newton and Hooke is well documented. The comments about seeing farther because of being on the shoulders of giants was thought to be a dig at Hooke’s short stature. However, this seems to be gossip and has not been proven.

Science is Falsifiable: 

If it is not falsifiable it is outside the realm/dominion of science. This philosophy of science was put forth by Karl Popper in 1934. A great example of falsifiability was the claim that all swans are white. All it takes is one black swan to falsify the claim. 

Science and Proportionality:

The evidence required to accept a claim should be in part proportional to the claim itself. The classic example was given by the famous scientist Carl Sagan (astronomer, astrophysicist and science communicator). Did the TV series Cosmos and wrote a number of popular science books (The Dragons of Eden). Sagan made the claim that there was a “fire-breathing dragon that lives in his garage”.

How much evidence would it take for you to accept the claim about the dragon? His word, pictures, videos, bones, other biological evidence, how about knowing any other dragons or dragons that breathe fire?

Compare that to if I said we just got a new puppy and it’s in the garage. You would probably take my word for it. There is nothing extraordinary about the claim. Most of you should be familiar and have had experience with a puppy at some point in your life. 

So the quality of evidence to convince you of something should be in part proportional to the claim being asserted. The summary is the famous quote by Carl Sagan that “extraordinary claims require extraordinary evidence”. 

Science is Self-Correcting:

Because science is iterative and falsifiable it is also self correcting. Science gets updated. We hopefully learn and get closer to the “truth” over time. Medical reversal is a thing and there is a great book and by Drs. Prasad and Cifu on this issue called Ending Medical Reversal: Improving Outcomes, Saving Lives.

Burden of Proof:

Those making the claim have the burden of proof. It is called a burden because it hard – not because it is easy. We start with the null hypothesis (no superiority). Evidence is presented to convince us to reject the null and accept there is superiority to their claim. If the evidence is convincing we should reject the null. If the evidence is not convincing we need to accept the null hypothesis.

It is a logical fallacy to shift the burden of proof onto those who say they do not accept the claim. They do not have to prove something wrong but rather not be convinced that the claim is valid/“true” and this is an important distinction in epistemology.

Real World Example:

Probiotics have been promoted for acute gastroenteritis (AGE) in children. Previous work in this area has been described as being “underpowered or had methodology problems related to the trial design and choice of appropriate end points.”

Schnadower et al. did a randomized control trial (RCT) of Lactobacillus rhamnosus GG vs. placebo for AGE in children (NEJM 2018). They included children 3 months to 4 years of age with gastroenteritis. The trial enrolled 971 children who to took the probiotic twice a day for five days or placebo.  The results showed no statistical difference between the two groups for their primary outcome.

We covered this RCT on SGEM#254: Probiotics for Pediatric Gastroenteritis. Can we say probiotics don’t work? No that would shift the burden of proof. However, without sufficient evidence of superiority we should accept the null hypothesis.

This study was limited to only the probiotic tested in the trial. However, Freedman et al in this same 2018 NEJM edition had similar trial looking at L. rhamnosus and L. helveticus found the same thing (no superiority of probiotics vs placebo).

It should be noted that there is some weak evidence for probiotic efficacy in antibiotic associated diarrhea (Goldenberg et al 2015). The bottom line is that probiotics cannot be routinely recommended at this time for acute gastroenteritis in children (SGEM#

Evidence-Based Medicine (EBM):

This was defined by Dr. David Sackett over 20 years ago (Sackett et al BMJ 1996). He defined EBM as “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”  I really like this definition, and the only tweak I would have added would be to include the word “shared“.  

The definition of EBM can be visually displayed as a Venn diagram. There are three components: The literature, our clinical judgement, and the patients values/preferences.

Many people make the mistake of thinking that EBM is just about the scientific literature. This is not true. You need to know about the relevant scientific information. The literature should inform our care but not dictate our care.

Clinical judgement is very important. Sometimes you will have lots of experience and other times you may have very limited experience.

The third component of EBM is the patient. We need to ask them what they value and prefer. The easiest way to do this is to ask the patient. It should start with patients care and it ends with patient care. We all want patients to get the best care, based on the best evidence. 

Levels of Evidence:

There is a hierarchy to the evidence and we want to use the best evidence to inform our patient care. The levels of evidence is usually described using a pyramid. The lowest level is expert opinion. the middle of the hierarchy is a randomized control trial and the top is considered a systematic review.

The systematic review +/- a meta-analysis is put on the top of the EBM level of evidence pyramid. However, we need to watch out for garbage in, garbage out (GIGO). This means if you take a number of crappy little studies (CLS), mash them all up into a meat grinder and spit out a point estimate down to the 5th decimal place that results is some impressive p-value is an illusion of certainty when certainty does not exist.

EBM Limitations:

  1. Harm and the parachutes
  2. Most published research findings are false
  3. Guidelines are just cookbook medicine
  4. Good evidence is ignored 
  5. Too busy for EBM

Five Alternatives to EBM:

This was adapted from a paper by Isaacs and Fitzgerald BMJ 1999. To paraphrase Sir Winston Churchill, EBM is the worst form of medicine except for all the others that have been tried.

  1. Eminence Based Medicine – The more senior the colleague, the less importance he or she placed on the need for anything as mundane as evidence. Experience, it seems, is worth any amount of evidence. These are the senior physicians on staff that make the “same mistakes with increasing confidence over an impressive number of years.”
  2. Vehemence Based Bedicine – The substitution of volume for evidence as an effective technique for brow beating your colleagues and for convincing relatives of your ability. The quality of the evidence is more important than the quantity of evidence.
  3. Eloquence Based Medicine – This is the physician with the year round suntan, Armani suit, pocket handkerchief and tongue that is as silky smooth as his silk tie. Sartorial and verbal eloquence should be no substitute for high-quality, clinically relevant evidence demonstrating a patient oriented outcome.
  4. Nervousness Based Medicine – Fear of litigation is a powerful stimulus for over investigation and over treatment. In an atmosphere of litigation phobia (shame and blame), the only bad test is the test you didn’t think of ordering.
  5. Confidence Based Medicine – Not to be confused with competency based medicine. Confidence based medicine is usually restricted to surgeons.

Five Steps to Critical Appraisal:

COVID19:

COVID19 has tested our abilities to be rational and use science to guide our care. We need evidence-based medicine more than ever. COVID19 has been bit of a dumpster fire with masking messaging, saying social rather than physical distancing, the hydroxychloroquine saga, remdesivir by press release, convalescent plasma observational study published as non-peer review pre-print, etc.

Dr. Simon Carley et al published an article in BMJ_EBM 2020 speaking about how the urgency of COVID19 represents both a threat and opportunity for EBM. I agree with the message in this article and discussed this with Dr. Carley on the SGEM Xtra: EBM and the Changingman.

“The urgency and severity of the COVID-19 pandemic contains threats and opportunities to clinicians wishing to practise EBM. We echo the call of others for critical reasoning, critical appraisal and critical thinking during these challenging times. The principles of EBM are more important now than at any other time in our careers. We must collectively do all that we can to ensure that our response to the pandemic is based on the science and not on the emotional, political or economic issues that challenge it.”

Take Home Message:

  • EBM Rocks – Once you get a real taste of this type of practice it is hard to ever go back to other forms of medical practice. 
  • EBM Answer – It all depends. Things take place in a context, often on a spectrum and rarely is there a simple dichotomized answer. 
  • Be Skeptical – Even of me. It does not mean you are closed minded. Rather you should be very open minded. Just not so open your brain falls out. The time to believe something is when there is sufficient evidence. 

Conclusion:

Again, the presentation is available to watch on YouTube, listen to on iTunes and all the slides can be downloaded from this LINK.

The SGEM will be back next episode doing a structured critical appraisal of a recent publication. Trying to cut the knowledge translation window down from over ten years to less than one year using the power of social media. So patients get the best care, based on the best evidence. 

REMEMBER TO BE SKEPTICAL OF ANYTHING YOU LEARN, EVEN IF YOU HEARD IT ON THE SKEPTICS’ GUIDE TO EMERGENCY MEDICINE.