Engaging With Bad Science Episode 1 – Why Sample Size Matters

Having been on the internet talking about bad science for about a year, I had until recently managed to stay away from talking about vaccines. However, I have recently broken my personal rule and talked about vaccines here. I was almost immediately reminded of why I had made such a rule, when I received quite a lot of comments listing a long series of presumably “anti-vaccination” studies (I only approved the ones that contained no foul language and seemed to come from genuine human beings). I decided that having opened the can of worms, I might as well go fishing and decided to start going through these studies and look into them. Unfortunately, I did not get very many links, which meant it was quite difficult to find the articles in question. I have, however, done the best I could. If you are interested in looking up the original references I was given, please check out the comments section under my post. In this first installment of what will undoubtedly be a thrilling ride, we are going to be looking at two of these “anti-vaccine” studies: a 2016 study published in the journal Frontiers and the 2013 report from the Journal of Toxicology. Both, I believe, outline some very teachable moments in terms of how to work with bad science.

The Frontiers study is a great example of a study that is bad science because it is published in a predatory journal. A predatory journal is a scientific journal that operates as a predatory business: they charge people to publish studies in their journal and they charge people to access these studies. They make a lot of money and they really don’t care about what kind of science they publish, They have published (and retracted) studies on chemtrails, and on predicting whether people are dead or alive based on their picture(which is kind of strange because most people who are dead used to be alive). In this case, they published an anti-vaccine article from a scientist who has raised funds on a platform that links autism to vaccines and who has made it clear they are anti-vaccine activist. What’s more, it was peer-reviewed (i.e. checked for accuracy) by a chiropractor and a scientist in public health (neither of whom had any expertise in virology, immunology or pathology, which are the disciplines relevant to this conversation). The scientific community spotted several terrible mistakes in the paper – they crucial issue being that the entire study is based on the self-reporting of mothers who have chosen not to vaccinate their children. Self-reporting means that the data in the study was not measured by a scientist, but was based on the recollections of a parent based on their impression on the mental state of their child at the time. The reason self-reported observations are not regarded as valid scientific data is that they are subjective and they are entirely dependent on the previous ideas and prejudices of whoever is making the observations. If a parent believes vaccinations are dangerous, they are more likely to report their child looking a bit unwell after they were vaccinated. For these reasons, the paper was retracted and is no longer a part of the scientific literature (even predatory journals have standards).

The second article is an interesting case because it is published in a real journal and it does have some actual scientific data in it. The problem with it is that it has a very small sample size (we are talking about four children). In fact, the person who pointed me towards the study recognized this, but seemed to think it was some type of lame excuse that the scientific community uses to get rid of data it doesn’t like. However, sample size matters! If you don’t have any background in statistics, here’s why. A small group of people is, essentially, much more likely to be accidentally biased. Imagine you want to work out how tall the average American man is. Imagine you do this by getting an average of some people you know. I am sure you can think of four people you know who are all far taller than average, and four people who are all far shorter. If you were to pick those four, you would think that the average American man was 7 foot tall! However, if you picked ten people, it would be harder and harder for you to find people who are far too tall. This means that the average height you end up measuring gets closer and closer to the real average height of an American man. If you picked 20 people, you would be even closer to reality as it would be increasingly difficult to find people who are far taller than average. Of course, as you start picking more and more people, you might well incorporate some people who are way shorter than average, which would bring you even closer to the true average size of an American man.

The people-measuring example is, of course, not related to vaccines but shows well why a small sample size is a real problem. As scientists, we make conclusions that are going to affect hundreds of thousands of people. We cannot in all conscience do this on the basis of four people. Anyone can understand that.

Do you have any anti-vaccine studies that you think might have a point? Please leave me the link in the comments below!


12 thoughts on “Engaging With Bad Science Episode 1 – Why Sample Size Matters”

  1. If you are interested in studies that show problems with vaccines, then I can suggest some good sources. An excellent option is Hilary Butler’s web-site. She has dozens of articles with links to various studies.

    Another good place to go searching for studies is The Thinking Moms Revolution. Their articles frequently link to studies which are relevant.

    Finally, James Lyon-Weiler has published a book which cites a massive number of studies and also regularly discusses studies online.

    Ya know, it really isn’t a stretch to find studies that show problems with vaccines. They are published all the time in mainstream journals and go back to the very beginning of the vaccine approach to preventing disease. Massive numbers in fact.

    They just aren’t labeled as “anti-vaccine” they are labeled as “science” and they are simply looking at problems with vaccines. And they usually say something like this: “vaccines are the best public health discovery ever.”

    Good luck on your research.

    • Hi Deborah,

      Thanks, this is really helpful. I am more interested at looking at these studies on a case by case basis, do you think you could send me a link to one that you think has a particularly good point? Thanks!

  2. Vaccine safety and protectiveness research is a huge topic. Depending on your primary interest I’d point you in different directions. One of my favorite topics is herd immunity and has vaccination helped or hurt it. I’ve listed three lectures below which walk through some of the research on this topic.

    My favorite author, on vaccine safety issues, is Dr. Suzanne Humphries. She does an excellent job citing the research she uses to form her opinions and she has credentials which are hard to fault. Some claim she is a homeopath: but she isn’t. She initially tried training as a homeopath: but as she said she couldn’t get homeopathy to work for her and ended up training as a traditional MD specializing in Nephrology. She ended up shifting to private practice when she couldn’t get her hospital’s management to look at vaccine safety research and allow her to stop vaccinating her kidney patients.

    She has many interesting lectures on her YouTube channel.

    I’d probably start with the following and see if they interest you. She includes the PubMed Id at the bottom of each slide. I typically listen to the videos at double speed. Her book provides a good overview of many of the issues and history with tons of links to the research.

    Herd Immunity Measles by Dr. Suzanne Humphries

    HERD IMMUNITY: Whooping Cough

    Wack-A-Mole vaccine game. Invasive bacterial vaccines of childhood. Tonsillectomies, antibiotics, and vaccines. This is a poor quality video which makes reading her PMID references impractical. A link to a subscription based professional version is listed in the video’s description.

    Dissolving Illusions

    Let me know if you want to go further down the rabbit hole.

    Try finding any ProVax sites which cite research, not opinion, addressing the following issues. If you would like the research the VaxSafety authors cite, let me know.

    – Why don’t all the arguments against overuse of drugs and their side effects also apply to vaccines? We are seeing vaccine strain replacement and a weaken ability to respond to strains not in the vaccine, some diseases appear to protect against other diseases so blocking them can have negative side effects, original antigenic sin, and other issues.

    – If you wanted to trigger an allergy there are various mechanisms. One method involves repeatedly injecting foreign substances, such as oils, into a body. Given the 20th century’s quality control, during vaccine production, wouldn’t you expect a large increase in allergies and new diseases? You also have issues like provocation poliomyelitis. They’ve observed increasing rates of asthma as vaccinations are done earlier.

    – One of the risks, of moving material back and forth between animals and humans, is the development of new diseases or new forms of old diseases. This was done continually, throughout the 20th century during vaccine development, production, and distribution, without the ability to identify or remove many pathogens. How much damage has been done and continues to be done?

    – Problems with claims that vaccines, like Polio and Smallpox, have been effective. Anti-vaxxers cite a laundry list of problems with the historical claims that Polio and Smallpox were vanquished by vaccination rather than better living conditions and disease management. They have also noted evidence that various diseases were spread or triggered by these and other vaccination programs.

    – Animal studies and epidemiological studies suggest various issues. Are human clinical trials and monitoring designed in such a manner that they would be likely to identify those issues, if they exist in humans and are due to vaccination?
    = Animal studies have shown that immune system stimulation is sufficient to cause developmental problems with fetuses. Mothers don’t need to catch the diseases. In studies of vaccination during pregnancy they’ve repeatedly used immune system stimulants as the control.
    = Animal studies have shown repeated problems with vaccines causing cancer. Try to find a human vaccination trial where they claim they have studied it for cancer.

    – Higher hospitalization rates for vaccinated children, when they catch a non-matching strain of Flu (3 times the rate in both a Canadian and an American study of hospital records).

    – Diseases spread by live vaccines. Significant percentages of children shed viruses for more than 4 weeks after vaccination. Look at ICU’s and their warnings against visitors with recent vaccinations.

    – Several vaccines suppress or change symptoms, without necessarily stopping spread. What are the impacts?

    – The demographics of when diseases occur have shifted to ages, earlier and later, when they are more dangerous. Booster effectiveness appears to decrease each time. Modeling has predicted upcoming outbreaks even with full vaccination as confirmed by 100% measles vaccinated populations in Africa. What is the evidence that vaccination improves longterm herd immunity?

    – Studies have shown higher hospitalization rates following killed vaccines compared to live vaccines. Why is that?

    – Infants born to vaccinated mothers tend to lose initial measles immunity, from maternal antibodies, 2 months sooner than those born to unvaccinated mothers.

    – The immune systems of children, younger than 6 months do not appear to benefit from vaccination and premature infants are being damaged. Does the initial vaccination age need to be shifted?

    – Contamination and security problems. Which vaccine contents have caused problems historically and how are such issues currently screened for? For example several diseases were spread by early vaccination efforts before quality control was improved. One pro-vaccine article cited poor quality control, in earlier decades, as evidence that the human body can handle more vaccines. The author noted that the recommended vaccine schedule used to contain an estimated 3,000 unintended antigens and they now have that down to 200 antigens.

    For example a few ingredients, cited by VaxSafety advocates are: peanut oil, squalene, SV40, RSV, retroviruses, mycoplasma, DNA fragments, … What is being done to control safety as vaccine and vaccine component production are outsourced to nations with poor quality control or security.

    – Many vaccine studies have been faulted for their short duration and for the choice of controls. How is this being addressed? Note since the 2015 Cures Act the testing is much weaker than for drugs. Studies of the VAERS database have estimated that it captures <10%, and perhaps <1% of the vaccine side effects.

    – Vaccine interactions, nutrition, and genetics. African studies have shown the order of administration, nutrition, and genetics matter.

    – Epigenetic changes, resulting from vaccines.
    vaccines. Any counter evidence?

    – Corruption within the testing and publication process. Note a list of whistle blowers and retired editors citing endemic bias and cheating.

  3. Adding onto the other comment thus far, you won’t find an anti-vaccine study that makes a valid point simply due to the authors placing their emphasis on making the biggest possible splash in the news outlets with their anti-vaccine message rather than employing a rigorous study design, methods, analysis (including statistical), and subsequent peer review. The studies you mention in your article are pretty much par for the course, although even if a horrible study is retracted from a journal, it often is just subsequently published in another journal. The anti-vaccine movement pretty much gets what it wants if the study is ever published so they have something seemingly scientific to point to, even if the study is later retracted.

    As for those long series of published anti-vaccine studies you see, I have bitten the bullet and dug into a dozen or so of the studies in a list of 30 or 40. When I read the original articles, I found that their ability to support the list poster’s anti-vaccine stance was completely lacking. One list of studies was about aluminum toxicity, and a couple of papers listed didn’t even have “aluminum” anywhere in the text. For some that did, the authors were injecting mice with 1000X the dose of aluminum that a US infant would receive his or her entire life. Plus, it was being injected into the peritoneum (into the belly but not into any organ) instead of the muscle or subcutaneous region and then they were injecting up to 6 times during the life of the animal. So, from this latter example, you can get an idea of how some groups will perform some outrageous experiments that have no relevance to vaccine safety just to get some effect that can be promulgated.

    Lastly, any well-done, peer reviewed vaccine study by conscientious authors will include the deficiencies in the vaccine’s safety, its ability to elicit an immune response, or its ability to protect against disease. This is especially true for clinical trials since they must report the adverse events that occurred during the trial. But it really takes multiple studies and digging into the strengths and weaknesses of each study on its own and then taking all of the studies into the larger context to begin to get the real picture. It’s just not black or white like the anti-vaccine groups want it to be. It’s science with its limitations and also it’s medicine with its risks and its benefits.

  4. Hi Gaia
    Interesting article. I’m not a scientist but have an interest in this topic.
    One thing I’m unsure of is what vaccines we’re actually talking about here. Are we limiting the list to vaccines currently given to children in a particular country – or are we talking about all vaccines ever administered to children ever worldwide? There have certainly been cases of vaccines being recalled because of adverse effects – so it always seems to me the line of “vaccines are safe” is fairly unscientific and vague without defining parameters.
    Surely all we can say scientifically is that a particular vaccine is “safe” after defining what we mean in terms of drug efficacy, chance of contracting the disease, likely outcome of contracting the disease, chances of adverse reactions and their likely consequences. Would you agree with this? Interested to hear your thoughts.

    • Hi Alex,

      You are of course right. Specific vaccines have had problems in the past and scientists and clinicians are always working to improve the efficacy and safety of vaccines. This is true for vaccinations as well as every other type of drug and surgical intervention we routinely rely on. The reason why we often talk about #VaccinesAreSafe as a social media campaign is because people are currently willingly choosing to not vaccinate their children because of their misunderstanding of what the risks associated with vaccination are. This is costing actual human lives, mostly of very young people who had no say in the decisions their parents made for them. So in conclusions when we talk about the fact vaccines are safe, it doesn’t mean that vaccines are completely perfect. Nothing is! What do you think?

      • I just think it’s a bit problematic to say vaccines are safe without knowing which vaccines we’re talking about or how we’re defining safe. I believe there are vaccines being administered in third world countries which were recalled in the west for having adverse reactions. Are those vaccines safe? Don’t get me wrong, I don’t hold an anti-vaccine position. I’m all for vaccines, but also for transparency and individuals being able to decide based on actual data rather than social media campaigns and obfuscated medical risk reports. That’s just my opinion though. It may well be the case that social media campaigns are the best idea for the herd / society.

        • I’m honestly not sure what is more problematic: writing about how a life-saving intervention is safe (when what I really mean is that it is broadly speaking safe while of course there are cases in which the rigorous safeguard measures we implement on medicine have failed) or
          ignoring the whole thing and doing nothing.

          • One issue is “Does the research actually backup the claims?” Gardasil research is a case study in things to watch out for. It’s been awhile, so double check the following. 90% of the controls were injected with a modified aluminum which is a known neurotoxin and immune system irritant. Itmigrates between organs, including the brain and brain stem for years. Gardasil was worse than those controls but not bad enough to be deemed significant: given the experimental design. If you knew where to look you could find the individual studies that used a saline injection for the control. In those cases Severe Adverse Events were running about 0.5% for the Gardasil group and no SAE’s for the saline.

            Putting this in personal terms: I went to a small high school of about 200 students. If they all received the vaccine you might anticipate 10 resulting in: death, were life-threatening, required inpatient hospitalization or prolongation of existing hospitalization, resulted in persistent or significant disability/incapacity, or is a congenital anomaly/birth defect …

            Digging further you found things like: cancer rates increasing if they had HPV before the vaccine. When they later studied a vaccine with additional strains they found vaccinated participants had a weaker ability to respond to strains not in the vaccine. Slate investigated the trials and concluded: “… the trials weren’t designed to properly assess safety”.

          • First of all, 0.5% of 200 is 1, not 10.
            Secondly, Gardasil has since its approval reduced the rates of infection by up to 90% in the past 10 years, thus also saving thousands of lives. It has revolutionized women’s health.
            Thirdly, let’s actually get into the Gardasil trials. I am not going to get into the validity of the Slate investigation because I don’t know enough about it. I am going to assume that they were right and that the trial was poorly designed. Since the 1990s, we have made huge improvements in the way we design and carry out clinical trials. However, nowhere in the Slate investigation does it say that Gardasil was worse than the controls. That is simply untrue. Please show me where you got this information and I will happily look into it.
            Lastly, aluminum is present in everything around us, including the food we eat and the water we drink. Not because they are polluted, but because aluminum is a very commonly occurring element on planet Earth. Yes, high doses of it are dangerous, but the doses found in vaccines are smaller than those found in a single pear.

  5. Oops, you’re correct 1 not 10. Still bad. There was a lot a variation between trials my 0.5% number came from one of the saline trials. The Gardasil 9 insert rated the overall SAE rates at 2.5% for the original versus 2.3% for Gardasil 9. My hand tremors are acting up tonight so typing is difficult. I’ll try to get you the saline study links tomorrow. Dr. Humphries has done some walk throughs on the research cited in Garadsil’s inserts. She has released newer videos as she’s discovered more. On YouTube try searching for the words: “suzanne humphries gardasil”. I usually listen to these at 2x until something jumps out at me. For example:

    I’m not aware of any good studies tracking Gardacil vaccinated to unvaccinated and cancer. There are some epidemiological studies looking at suspected cancer precursors rates. Also are they reducing just the easy to cure cases or the deadly ones. Any pointers? There was a similar issue with the measles vaccine and encephalitis rates (mild versus severe). There was an initial reduction in mild cases which later rebounded.

    Aluminum is common in bound form and has become common in pure form with industrialization. Our normal exposure is through ingestion: which is poorly absorbed and which the body is fairly good at purging. Still dangerous over time. The injected forms have been modified to be much more reactive, are 100% absorbed, and takes much longer to eject. It does tend to exit the blood quickly but often migrates to the organs. What justification is there in its use as a control?

Comments are closed.

Share This