The Scientific Method and the Pandemic
In my fifth-grade science class I remember devising a way to remember the process involved in the scientific method as my teacher, Miss Iman, articulated them. While she wrote specific steps on the chalkboard what she taught me, through those steps I copied in my notebook, was that scientific theories have to be based on rigorous observation and methodically tested. I don’t think I realized how important this methodology is and what an intellectual achievement it was when individuals like Francis Bacon and Isaac Newton discovered it.
A few months ago, I got a message from a television reporter asking about a woman with long-term COVID-19 symptoms who had tweeted that upon receipt of the 1st dose of a COVID-19 vaccine, her symptoms improved. I replied, almost reflexively, that an anecdote is not enough to determine whether the vaccine was responsible for the change in symptoms. Stray anecdotes are not the equivalent of an organized body of observations or evidence, as I will discuss below. Also, and critically, there needed to be some biological mechanism postulated to be at work and experimental study to support the idea.
Unfortunately, during the COVID-19 pandemic there have been countless examples in which this method has been abandoned or distorted. While it is true that we use anecdotes all the time in healthcare to help inform our choices, the stakes during a pandemic are too high to rely on anecdotes in a significant manner. From hydroxychloroquine to ivermectin many treatments have been touted on anecdote and hypothesis without much, or sometimes in defiance of, evidence or rigorous scientific investigation.
It all relates back to a profound question: How do we know what we know?
The confusion we’re seeing in the pandemic largely stems from an inability to distinguish between anecdotes and substantial evidence. To be able to discern the difference, one must methodically move through a process that begins with observation and ends with knowledge in the form of a theory.
Anecdotes are not worthless; in fact, the power of informal observation is crucial. And, if we start to hear a few with similar themes it may be a signal to look closer, but sometimes it is just noise. An observation alone is not enough to make causal links between two phenomena. The events could simply be correlated but not part of cause-effect relationship such as someone who develops a heart attack or neurologic condition post-vaccination, not due to the vaccine, but because they had multiple risk factors for coronary artery disease or blood clots. In this case, the association with the vaccine was merely temporal, but not causal. With informal observations we must determine how best to get representative knowledge of populations to move beyond anecdote.
Observation is a critical part of the scientific method but primarily as a basis for generating hypotheses. It is crucial to know what it actually means to actually base hypotheses or theories on observations. Systemic observation by experts who have a large context of knowledge is extremely valuable. Such a context of knowledge may include how what is observed coherently integrates with known biological principles. When thinking about anecdotes, hypothesis, or correlations it’s extremely important to hypothesize about what the cause or mechanism may be for an event to occur. Is this something that is plausible as a cause? For example, how does the claim align with what is known about immunology or similar viruses, medicines, or conditions? Is this something that makes sense based on what we already know?
These systemic observations, linked to a proposed biological mechanism consistent with the nature of the entities involved, can then form the basis for a hypothesis. A proper hypothesis is one that invokes a causal mechanism for what is being explained — how the nature of the entities involved might explain the phenomenon.
But a hypothesis as such isn’t conclusive and doesn’t give us a strong enough reason to act. We must next test hypotheses through experimentation. In clinical medicine it is largely through clinical trials we gain knowledge of the efficacy of treatments.
The gold standard for obtaining such knowledge and testing hypotheses is adequately sized double-blind, randomized, placebo-controlled trials. Such trials are ideally: large, conducted with diversity in both geography and participants, and with neither the researchers nor the volunteers knowing who is getting an active treatment (or intervention) and who is getting an innocuous placebo. At the trial’s conclusion, groups of the trial can be compared to determine whether one group fared better or worse. This study design is intended to prevent biases from confounding the data and undermining the reliability of the results.
At the end, we have a theory that is not just a guess, but a conclusion derived from the methods discussed above.
In the early studies of hydroxychloroquine as a treatment for COVID-19 which claimed to show faster symptom resolution, it was critical to use a control arm to determine if the timeline was actually accelerated or just the normal course of illness. Would people have fared the same without the medication? Was the medical treatment playing a causal or incidental role? Similarly, were other medications also given that could account for the difference? Was the study large enough to exclude chance? With the claims of alleviation of long-haul symptoms post-vaccine, it is essential to know what other medications were being taken, in how many individuals did this occur, what is the biological mechanism, and other questions that can only be answered by moving beyond anecdote to rigorous study. This type of study is not only used to determine efficacy but also safety: were the side effects that occurred linked to the treatment or just part of the disease process?
To know anything definitively about a treatment, therefore, it is necessary that the only aspect different between groups is the item being studied. This can best be achieved in appropriately designed clinical trials.
Advocates of the scientific method never wanted any of these treatments to be unsuccessful or fail. On the contrary, we wanted for these medications to be studied in a manner that had the capacity to prove, with a high level of certainty, that they were beneficial, before recommending their general use to our patients.
If anecdotes such as those associated with hydroxychloroquine were kept in the proper context and not sensationalized by the media, the President, and even some medical professionals they would not be much of a problem. However, when they are discussed as incontrovertible discoveries, splashed across headlines or tweets without context, or not followed up by sufficient study they can cause damage.
When the population believes a treatment such as ivermectin is effective because they have heard about it from neighbors, will scientists be able to recruit them to a clinical trial where they have a 50% chance of not receiving it? When hydroxychloroquine, used for rheumatological conditions like lupus was touted for its benefit versus COVID, a surge in demand caused a supply shock made it more onerous for those with lupus (which the drug has been proven to be beneficial for) to obtain the medication. While it is important to have pathways for prescribers and patients to have access to medications outside clinical trials, many people took medications like hydroxychloroquine and exposed themselves to potentially dangerous side effects before any strong signal of benefit was shown to exist (none was coming).
While some potential treatments such as vitamin C and zinc are relatively innocuous, I find that many of those that advocate for their use do not do so based on science but often have a tendency to not value the scientific method, as evidenced by the way many of them do not recognize the science that formed the basis for the use of vaccines and face coverings. Additionally, someone taking vitamin C or zinc could potentially place themselves at danger by using these substances in lieu of seeking medical attention (or delaying it). There have even been instances in which someone, misinterpreting or not fully thinking about the level of evidence, ingested a harmful substance such as a fish tank product containing hydroxychloroquine or veterinary ivermectin.
Knowledge is something that must be acquired through a specific process. The secrets of nature don’t just write themselves on our mind. Humans must work to obtain knowledge. This process is not automatic nor is it infallible. That is why a method is necessary. Such a method allows one to explicitly focus on relevant variables, isolating them from extraneous elements, and provides multiple checks against error is vital to scientific inquiry. It facilitates the dispassionate quest to discover the objective truth about reality.
In science, we use the scientific method: the process of observing reality, coming up with a hypothesis about what might be true, and testing that hypothesis as thoroughly as possible until we discover, via an objective manner, the truth. We do not need to explicitly use the scientific method in every one of our medical decisions made in the pandemic because sometimes decisions may need to be made quickly, before evidence has been fully collected and analyzed. In these instances, however, it is critical that the aim is to use this method to amass the evidence that will eventually inform decision making and the action comport with established biological facts. Also, once evidence, both positive or negative, is available it should be the guide while anecdotes should recede in importance.
What this pandemic has revealed to me is that many of us need to remember and revere this method and what it represents. Our very lives depend on it.
