Having been on the internet talking about bad science for about a year, I had until recently managed to stay away from talking about vaccines. However, I have recently broken my personal rule and talked about vaccines here. I was almost immediately reminded of why I had made such a rule, when I received quite a lot of comments listing a long series of presumably “anti-vaccination” studies (I only approved the ones that contained no foul language and seemed to come from genuine human beings). I decided that having opened the can of worms, I might as well go fishing and decided to start going through these studies and look into them. Unfortunately, I did not get very many links, which meant it was quite difficult to find the articles in question. I have, however, done the best I could. If you are interested in looking up the original references I was given, please check out the comments section under my post. In this first installment of what will undoubtedly be a thrilling ride, we are going to be looking at two of these “anti-vaccine” studies: a 2016 study published in the journal Frontiers and the 2013 report from the Journal of Toxicology. Both, I believe, outline some very teachable moments in terms of how to work with bad science.
The Frontiers study is a great example of a study that is bad science because it is published in a predatory journal. A predatory journal is a scientific journal that operates as a predatory business: they charge people to publish studies in their journal and they charge people to access these studies. They make a lot of money and they really don’t care about what kind of science they publish, They have published (and retracted) studies on chemtrails, and on predicting whether people are dead or alive based on their picture(which is kind of strange because most people who are dead used to be alive). In this case, they published an anti-vaccine article from a scientist who has raised funds on a platform that links autism to vaccines and who has made it clear they are anti-vaccine activist. What’s more, it was peer-reviewed (i.e. checked for accuracy) by a chiropractor and a scientist in public health (neither of whom had any expertise in virology, immunology or pathology, which are the disciplines relevant to this conversation). The scientific community spotted several terrible mistakes in the paper – they crucial issue being that the entire study is based on the self-reporting of mothers who have chosen not to vaccinate their children. Self-reporting means that the data in the study was not measured by a scientist, but was based on the recollections of a parent based on their impression on the mental state of their child at the time. The reason self-reported observations are not regarded as valid scientific data is that they are subjective and they are entirely dependent on the previous ideas and prejudices of whoever is making the observations. If a parent believes vaccinations are dangerous, they are more likely to report their child looking a bit unwell after they were vaccinated. For these reasons, the paper was retracted and is no longer a part of the scientific literature (even predatory journals have standards).
The second article is an interesting case because it is published in a real journal and it does have some actual scientific data in it. The problem with it is that it has a very small sample size (we are talking about four children). In fact, the person who pointed me towards the study recognized this, but seemed to think it was some type of lame excuse that the scientific community uses to get rid of data it doesn’t like. However, sample size matters! If you don’t have any background in statistics, here’s why. A small group of people is, essentially, much more likely to be accidentally biased. Imagine you want to work out how tall the average American man is. Imagine you do this by getting an average of some people you know. I am sure you can think of four people you know who are all far taller than average, and four people who are all far shorter. If you were to pick those four, you would think that the average American man was 7 foot tall! However, if you picked ten people, it would be harder and harder for you to find people who are far too tall. This means that the average height you end up measuring gets closer and closer to the real average height of an American man. If you picked 20 people, you would be even closer to reality as it would be increasingly difficult to find people who are far taller than average. Of course, as you start picking more and more people, you might well incorporate some people who are way shorter than average, which would bring you even closer to the true average size of an American man.
The people-measuring example is, of course, not related to vaccines but shows well why a small sample size is a real problem. As scientists, we make conclusions that are going to affect hundreds of thousands of people. We cannot in all conscience do this on the basis of four people. Anyone can understand that.
Do you have any anti-vaccine studies that you think might have a point? Please leave me the link in the comments below!