Is the Yellow Card database completely useless or not?

Answer: Mostly useless.

Summary: The Yellow Card system is used to collate information on side effects and complications for medicines and treatments. In this post I compare a small set of Yellow Card side effects to data obtained from hospital admissions. I find that for most side effects the Yellow Card system significantly under-reports the side effects. I also discuss what else might have been done to improve our understanding of the side effects of the covid vaccines.

A few days ago I posted an analysis of a paper by Hippisley-Cox et al that investigated the rate of neurological side effects of the covid vaccines — in that series of posts I concluded that they broke a fundamental rule of the self-control methodology and thus their data significantly under-estimated the risks of the covid vaccines relative to natural infection with covid.

However, despite their misuse of the self-control methodology they did do a reasonable job at identifying the incidence rate of those side effect / complications in the NHS hospitalisation database and in their cross-correlation with the vaccination database. It must be remembered that their analysis only looked for side effects within 28 days of vaccination. Moreover, the data only included side effects that resulted in hospitalisation — more minor side effects and those where the individual decided not to seek medical assistance will not be counted in the data. Thus the Hippisley-Cox et al data will be an under-estimate of the full number of side-effects — but it is at least a start.

The authors also undertook a similar study into clot-related side effects in an earlier paper, with similar identification of the incidence of hospital admissions following vaccination using the NHS hospital admissions database.

We also have a database of side effects from the vaccines in the Yellow Card reporting system. This is a system that has existed for many years to allow the reporting of side effects resulting from medicines/treatments, and has naturally been used to record the covid vaccine side effects.

The question then is: how many of the side effects identified by Hippisley-Cox et al were also present in the Yellow Card database?

The side effect data from the Hippisley Cox et al papers is readily found in the papers (Table 2 in both papers), but the identification of matching side-effects in the Yellow Card database is a little more complex to work through.

• Firstly, we need to find historical summaries of the Yellow Card data relating to the covid vaccines for the periods in question to match the date range in the Hippisley-Cox et al papers — to the 24th April for the paper on blood-clot related side effects (thrombocytopenia and thromboembolism) and to the 31st May for the paper on neurological problems. Thankfully, historical Yellow Card data can be found in internet archives.

• I selected the Yellow Card reports with the earliest ‘database lock’ date after the Hippisley-Cox periods (28th April and 2nd June). Thus the Yellow Card data should be slightly higher than the Hippisley-Cox et al data, but only by the smallest of margins.

• Then we need to make sure that we are identifying the right side effects. For some, like Bell’s Palsy, there is only a single line in the database and matching the two sources is relatively easy. For others, like clotting related disorders, there are multiple possible side effects in the Yellow-Card database that match.

• Thus it is necessary to go through the Yellow-Card in detail to ensure that all matching side effects are identified. I have erred on the side of caution in this analysis and will probably have identified some side effects reported in the Yellow-Card system that might not have been similarly identified by Hippisley-Cox et al — but feel free to go through the reports and see what you make of them.

The main output of the analysis is the proportion of the cases that were identified in the comprehensive Hippisley-Cox et al analysis that were present in the Yellow-Card database

Without any further ado, here’s the data, first for the Astra Zeneca vaccine:

And second for the Pfizer vaccine:

It is clear from those tables that the Yellow-Card system doesn’t reliably record side effect incidence — reporting rates range from low percentages all the way up to over-reporting of some.

There is perhaps a little more to say on this matter:

• The Yellow Card reports for the Astra Zeneca vaccine generally has higher reporting rates than for the Pfizer vaccine; on average, its reporting rate is around three times higher. I would note that during the time in question there were concerns about the Astra Zeneca vaccine — does this gross difference in reporting rate suggest that the Yellow Card system is highly sensitive to medical and public opinion?

• The higher incidence rate side effects tend to have consistently lower reporting rates, compared with the rare side effects that are more inconsistent. Are more common side effects more likely to be ignored by the medical profession and considered as unrelated, thus minimising reporting into the Yellow Card system?

• Several rare side effects have very high reporting rates, while others are very low. I note that the high-reporting rate side effects are named after people (e.g. Guillain-Barré syndrome) while side effects with only a medical description have low reporting rates (e.g. subarachnoid hemorrhage). Is the reporting rate of ‘medically interesting’ side effects higher than rare but boring side effects?

It is important to note that these data will include instances where the ‘side effect’ was merely coincidental with the taking of the vaccine — ie, that there was no causal relationship between the vaccine and the side effect. I accept that — this is not a comprehensive investigation into vaccine risk. What I have done here is look into whether the (potential) side effect was recorded in the Yellow-Card system. Note that it is no good suggesting that ‘unrelated conditions shouldn’t be reported’ — the whole point of these databases is to record everything and let the issues be identified by statistical analysis of the data. If medical professionals are required to make a decision about whether a medical condition is actually a side effect then that merely introduces bias into the data which makes a full analysis more difficult.

---

That’s the analysis of the Hippisley-Cox et al papers done, but I have another interesting example that shows how complex it can be to interpret data in the Yellow Card database properly; the incidence rate of side effects related to menstruation. Reporting is always complex for any side-effects related to sexual and reproductive health, but there are hidden complexities beyond that. Here’s a chart of the reporting into the Yellow Card database of all side effects related to menstruation disorders occurring after the Astra Zeneca vaccine (data for Pfizer not shown, but it is similar):

(apologies for the variable width of the bars; the Yellow Card summary reports were not updated at consistent intervals)

I’ve colour coded the periods of interest.

• The early data points are coloured in light blue. As the general roll-out of the vaccine in this period was to those aged over 60 most menstruation related side effects were reported by the small number of healthcare workers and others who were pre-menopausal.

• From late April the vaccine was offered to those aged under 50. The roll-out into the pre-menopausal age group resulted in the substantial increase in reporting seen in the green bars.

• But what about the red bars? Why is there a substantial increase in reporting of menstruation related side effects for this period? A little digging identified the culprit — In late March BBC Radio 4’s Women’s Hour (a daily radio programme in the UK dedicated to reporting on issues that relate to women) had a short article on menstruation related side effects of the vaccine, and their interviewee asked the audience to report any changes in their periods into the Yellow Card system.

I think these data illustrate how the Yellow-Card system does not give a robust data set of side-effects that is simple to interpret.

---

The data that I have presented here adds to a body of evidence that suggests the Yellow Card system is not fit for purpose. It was brought into existence many years ago, based on difficulties in exploring data deeply embedded in medical records; in the days of paper based medical records collation of data from different sources was extremely difficult and a separate system for recording side effects was essential. However, with the advent of large electronic databases of hospital admission data and vaccination status the need for a separate system to record side effects is reduced substantially.

The coronavirus pandemic was a medical emergency that justified the roll-out of vaccines under emergency regulation to the most vulnerable. However, this did not remove the need for robust pharmacovigilance. We should have used an automated side effect monitoring system based on cross-referencing the hospital admissions data to the vaccination database, rather than relying on the barely adequate Yellow Card system. This should have been reported weekly/fortnightly, so that up to date information on side effects could be collated and the public informed.

I’m sure there would be a response from the healthcare profession that this approach would merely mix up all sorts of normal medical conditions into the data for potential medicine/treatment related side effects, making the data to broad to be useful. I’d argue the opposite — as it stands the Yellow Card system is only provided with data when a healthcare professional decides that the condition that they’re seeing is probably related to the medicine/treatment, yet different professional have different thresholds for when they believe a genuine side effect. They’re also biased towards reporting side effects that are already considered to exist (conditions not on the side effect list are less likely to be reported) and also biased to only report conditions that emerge in a short period of time after the medicine or treatment is given. An independent and comprehensive database would remove these biases and allow for statisticians to identify side effects, not matter how much the medicine provider or healthcare professional thinks that they can’t exist.

But that’s not enough. A system based on hospital admissions would only identify the more serious side effects. Other side effects might be too minor to warrant the seeking of healthcare but might be indicative of serious problems to come, in which case early warning of their existence might offer improved outcomes. Other side effects, such as those related to menstruation discussed above, might be associated with embarrassment, discomfort or worry, and the highlighting of these side effects could make an affected individual be more likely to seek appropriate symptom relieving medication or perhaps make a personal and informed decision to forego vaccination given the presence of risks with minimal benefits.

The problem of identifying minor side effects with important medical, psychological or long-term health consequences could have been mitigated through the use of cohort based pharmacovigilance in a sub-set of those vaccinated. While cohort based studies are expensive and complex to undertake, the uncertainty of the safety of the coronavirus vaccines (given the short timescales available for the initial clinical trials) and the sheer cost to the economy of error would have made such studies worthwhile. It could be argued that there was insufficient time to set up such studies, but this would only be true for the initial roll-out of the vaccines — cohort studies could have been set up for the second dose of the vaccine, younger age groups, specific risk groups (pregnancy) and, currently, the booster vaccinations. Oh, and cohort based studies could also have given an indication as to the real-world effectiveness of the vaccines at preventing infection, hospitalisation and death and how they change with time — surely this information would also have been very useful.

But even that wouldn’t be enough. There is also the problem of side-effects with very minor symptoms. For example, if there is an issue with side effects related to blood-clotting which has been identified by hospital admission data, perhaps there is also a problem with micro-clots that induce problems with minimal symptoms in the short term but that have longer term health issues. Similarly, is there sub-clinical threshold to the heart that will have consequences many years down the line? Such problems might have been identified via targeted studies into the reaction to the vaccines using tests such as measurement of D-dimer or troponin levels.

Of course, many will say that we are already doing these things — and I would accept that cohort based studies are being undertaken and there are specific investigations into sub-clinical threshold problems. The issue is timescales — these are all operating over a medium to long timescale, but we require these information in the short term — there’s no point in finding out that the vaccines has a serious side effect that changes the risk:benefit ratio for those not vulnerable to covid after they’ve all been vaccinated.

Back in November 2020 the eminent scientist Roy Andersen said in a paper:

What the duration of immunity is for a given COVID-19 vaccine will only be resolved once community-wide vaccination programmes progress. Phase 3 trials will tell us about efficacy and safety, but well designed phase 4 trials are essential based on representative and large numbers of those vaccinated and follow up over time. These studies will record any serious adverse events and identify whether repeatedly exposed individuals acquire coronavirus infections, particularly SARS-CoV-2, and if they do, what is the severity of disease. These cohort based longitudinal studies will need careful planning and sustained funding, probably from governments with industry contributing.

We seem to have done very little in the way of Phase 4 trials — we could barely have done any worse pharmacovigilance if we’d tried.