Right. That’s the theory done and we’ve covered the risks from covid. What about the vaccines?
I've picked a nice example, purely because the numbers are higher, which might mean random variability isn't so noticeable. But I definitely picked it on purpose -- anyone wanting to see more will have to go to the paper.
Anyway, my condition of choice is Guillain-Barré syndrome after the AZ vaccine:
Hmm. What have we here? I suppose it looks similar to the previous chart I showed (their data for Bell’s Palsy after covid) -- although note that we've now got vaccination at day zero instead of PCR positive test.
Nevertheless, it looks about right -- they have the vaccine, cases go up a bit, then you go back to baseline.
But there are a few puzzling aspects in their data:
Does it actually return to baseline? I said that this was crucial, but there's not actually any evidence that it does. The final data point in the green shaded area hasn't returned to baseline, and, although I suppose it might, as far as the data is concerned it doesn't. This is a massive red flag — it has to return to baseline to use this approach; if it doesn't then you have to lengthen the study period (the green shaded area), not simply ignore it because '28 days is what you chose and you're sticking with it'.
And what is going on in the pink shaded area? The authors will claim that people who get diagnosed with Guillain-Barré syndrome will delay getting vaccinated — but is this true? I think the opposite was happening — people were advised to get vaccinated because all sorts of conditions would make them vulnerable to serious covid disease...
Related to this is the side effect data for the first week after vaccination — again, this is lower than baseline but this time the author's couldn't use the excuse they used for the data in the pink shaded area. Why is the first week risk lower than baseline?
Are their assumptions about the data true, or have they used a flawed analysis? Well, I suppose I should start with what they want us to think.
I'm going to invent some actual daily numbers that could explain the data. Note that these are completely made up and chosen so that they fit the averages (that we do know, from the paper); we’re trying to guess what the underlying data might have looked like. We have to guess in this way because they’ve not made the underlying data available to us — only the authors have access to these data.
Just to make it absolutely clear -- the red horizontal lines are the averages for each time period from the paper, the circles have been made up by me to fit the averages.
So, in my made up data you've got a baseline (remember, the average for both yellow shaded areas), a lower incidence before vaccination (supposedly because those diagnosed wait for a bit before vaccination; there's no supporting evidence for this), a rapid rise in diagnosis of Guillain-Barré syndrome, which then even more rapidly declines back to the baseline.
Well, those little circles do match what the paper says, but I did have to try hard to make them fit... I'm not sure that it is the most likely explanation... What about a different interpretation of the data?
I’ve had a think about it and have come up with a different set of made up data that also fits the averages given in the paper...
Here I've included the red horizontal lines from prior graphs to show the averages from the paper. Again, the circles are data made up by me to fit the averages. I'll state this again -- these are made up and not real data. I'm just trying to show that there are other potential underlying data that fit the averages in the paper.
So, what have we got with this alternative interpretation?
The background rate in the leftmost yellow shaded area is lower before the vaccination, but now I've got rid of that pesky reduction in incidence of Guillain-Barré syndrome before the vaccination (the pink shaded area).
Guillain-Barré syndrome diagnosis numbers rise after vaccination, as before (the green shaded area).
But now case numbers decline much more slowly afterwards, maintaining a higher level for the duration of the post-vaccination study period.
The baseline is the average of the lower numbers in the yellow shaded section before vaccination, and the higher numbers in the yellow shaded section after vaccination. The average that they calculated for baseline is the same, but it now doesn't actually match the actual case numbers before or after vaccination.
Hmm. Now we've got a completely different interpretation of the data, but the actual averages, as found in the paper, are as before. What are the implications of this?
Well, for a start the calculation of the relative rate of Guillain-Barré syndrome post vaccination will be very different. Our pre-vaccine level is much lower, so every estimate of risk after vaccination will be much higher. This has a massive effect on the estimated risks of the vaccines.
And, before there were only cases of Guillain-Barré syndrome in the 28 days after vaccination. Using the new assumptions cases continue to be high for some time afterwards — so, not only are the risk levels higher than stated in the paper (bullet point above), they also stay high for longer — and this increases the relative risk of the vaccine for this condition even further.
As this alternative analysis effectively alters the baseline, I’ve redrawn the graph with the new estimate of the baseline given this alternative set of potential underlying data (the new potential baseline is based only on the pre-vaccination data and is shown in blue).
Is this alternative interpretation correct? I just don't know. But the paper definitely is suspect — without proof that the risk returns to zero in the 28 day period their analysis is deeply flawed. I'd say that it is more likely that my alternative 'guess' in this post is correct, than the authors' interpretation — but then I would say that, wouldn't I.
Ah, but actually I do have a bit of supporting evidence. In the supplementary material to the paper they actually give a little more data — they give data for a fifth week after vaccination (Supplementary Table 7b). This shows that the data doesn't return to baseline (for Guillain-Barré syndrome and AZ vaccine their IRR for the fifth week was 1.55 — definitely not recovered even to their baseline). So, the author's analysis is definitely not correct. Sure, it doesn't show how incorrect, but it definitely shows that their baseline has been contaminated with cases that appear to be coming from the vaccine. Nevertheless, the authors have definitely been beyond negligent in this regard — if they'd just had 'forgotten to check' for this effect then that would be one thing, but they've got the data that shows that their choice of 28 days as their ‘vaccine effect’ period is too short, and they decided to ignore it anyway.
If they were postgraduates I'd point this out to them and tell them to go back and do it again. As it stands I can't see how it was accepted for publication -- it really is a desperately naughty error.
How far does this go? They show data for all of the conditions and for both Pfizer and AZ vaccines in table 7b — at best I'd say there's little evidence of return to baseline for most of them, and some even suggest an increase in incidence rate.
I'd also note that the authors' have a previous paper on thrombocytopenia (and other related) risk — this shows exactly the same mistake. I've no idea why that one was accepted for publication either.