Left Brain Right Brain

A recent discussion here on LeftBrainRightBrain involved the peer review process and, in specific, how the 1998 Lancet paper by Wakefield and coworkers was reviewed. As I prepared a response I saw that (a) the response was getting long and (b) this gives a discussion of the peer review process in general, which could be of interest to some readers. So I decided to blog yet again about Mr. Wakefield. So, with apologies to those who have tired of Andrew Wakefied:

Here is Richard Horton being questioned about the general process of peer review of papers for The Lancet (from Day 16 of the GMC hearing). Note that it was typical for 3 referees to be engaged. It has been reported that the Wakefield Lancet paper used 6 reviewers. If so, it is very interesting. Why would the Lancet have gone after extra reviewers?

Lower down, you will find a discussion of the process involved in the Wakefield et al. Lancet paper.

Q Once that process is complete, they have seen the paper and written reports on it, what happens next?
A Those reports back in 1997 would have been sent in my mail, collated by the editor and, when all the reports were available, then he or she would present the paper plus the peer review comments at the Thursday afternoon editorial meeting and the debate would ensue.

Q Is it customary for people who have reviewed the paper to say critical things as well as positive things?
A Extremely, yes, very much so.

Q How do you handle those as far as the authors of the paper are concerned?
A We have two separate components. They say things that they are happy to be transmitted to the authors and they also provide confidential comments that they say they report to the editors, which often presents us with a tricky situation because they are frequently very polite to authors and somewhat less polite when they are discussing a paper in front of us privately.

Q On those parts that they are happy to have relayed to the authors, do you discuss any criticisms that have been made?
A On the Thursday afternoon meetings?

Q Or at all?
A We certainly debate all the pros and cons of the paper at the manuscript meeting and make a decision then whether we are going to reject the paper, seek further opinion or open negotiations with the authors.

Q Will there be, at that state, any general discussion about other matters which might be relevant to publication?
A Certainly we will be discussing whether the paper has aspects of it that might cause controversy and should be considered in judging a paper, whether there are any conflicts of interest that should be taken into account in considering the paper or any other aspects of the work that might have cause for concern.

Q At the end of that meeting, do you then make a final decision as to whether you are going to reject the paper or admit it for publication?
A Three options: either to reject, to open negotiations or to seek further opinion.

Q Can you tell us what open negotiations means?
A That means that if there are questions raised by one or more of the peer reviewers that we think require explanation or further elucidation in a revised manuscript by the authors, then we will go back to the authors and say please can you take a look at your paper, in the light of these reviewers’ comments, and make the necessary adjustments. They hopefully will take account of those questions, submit a revised version of the paper and then we go back into a Thursday afternoon discussion to decide whether that is acceptable or not.

Q The third option you said was to seek a further opinion. Is that over and above the peer reviews you have already had done?
A Yes, that is right. Particularly I would say now that is quite a common option but back in 1997 it was less common.

And here is a discussion of the Wakefield et al. 1998 Lancet paper:

Q Do you have any recollection or are you assuming that it would have been the norm as far as the number of others who were concerned?
A Yes, it would have been the norm to have sent it to three external advisers and a statistical reviewer.

Q Do you have any recollection of the nature of the reviews that were received?
A From what I can recall, there were two aspects that were most important. The first was that all of the reviewers remarked on the original nature of the description of the syndrome and felt that this was something that merited consideration for a general medical journal but there was concern about the reporting of the parents’ testimony relating to a possible temporal link with MMR vaccine.

Q We have the log book for when the paper was first discussed. I think it is right, I should say, so the Panel is clear, the actual reviews are no longer available?
A I am afraid that is so.

Q We do have the log book. If you look in the same volume, page 637, we see, in the middle of the page, the left-hand number is a manuscript number, is that correct?
A Correct. That would have meant it was submitted in November, so the first two digits reflect the month, then the next set of digits the simple sequence with which it was entered up by our office.

Q We are about ten figures down at 11096 “new GI syndrome in children”, is that correct?
A Correct, and JB refers to John Bignall under “Ed”.

Q What is the last column?
A The last column is about the decision about the paper. You can see we received two papers: one describing a new G I syndrome in children, and then a second attempting to define the cause of that syndrome. Both papers were taken through peer review by Dr Bignall. The first one, PP means put points, so a set of manuscript reviewer points were put to the authors, and the second paper was rejected after peer review.

Q As far as the title “A new G I syndrome in children” was that of any legal relevance, the title you have given it there?
A This has been a source of much discussion in the past eight or nine years. Many people have focused on the fact or asked the question was The Lancet in some way supporting the linking of MMR vaccination with this syndrome. From our point of view, when we first received the paper, the parental testimony was actually incidental. The central thrust of the paper was this new syndrome. This is not an uncommon kind of report. If you read any text book of epidemiology, the very first description of any new syndrome often comes with either a case report or a case series. If you go back and look in history at, for example, the first reports of HIV and Aids, the first reports of variant CJD, they all began with a simple case series very much like this one. Then it is quite typical for the investigators to ask, and it is their obligation to ask, the families or the patients “Do you have any thoughts yourself about any behaviours or activities you might have done that might have precipitated this syndrome?” For example, if you go back and look at the first reported series of Aids or variant CJD you see questions like that raised and the answers are very speculative. In this case, again, the answers were very speculative. Eight of the 12 families put potential temporal link with MMR vaccination was made. That was very much supplemental to the major theme of the paper which was this new syndrome.

Q Before I turn you to another page, I just wanted to complete things. As far as the second paper, which you rejected, was concerned, did that come from the same research group?
A It did come from the same research group although I cannot recall, and I do not have a record, of exactly what the authorship of that paper was. I do not recollect.

Q Can you help us as to its nature at all?
A From what I can remember, this was a laboratory study trying to identify what the possible cause of the new syndrome was. From what I can remember, this was an attempt to try to isolate a component of the MMR vaccine with this syndrome.

Q Without that paper, the paper with which we are concerned, the 11096 paper made only the temporal link, is that correct, with the MMR vaccine?
A That is exactly right. Not only did it only make the temporal link but it was made very clear in that paper that such a temporal link was not a proof of association, moreover that there was no published evidence to support any association between the vaccine and the new syndrome.

Q If you turn back to the paper, page 783 and go on to 787, the discussion session, we see, in the left-hand column:

“We did not prove an association between measles, mumps and rubella vaccine and the syndrome described. Virological studies are underway that may help to resolve this issue.

If there is a causal link between measles, mumps and rubella vaccine and this syndrome, a rising incidence might be anticipated after the introduction of this vaccine in the UK in 1988. Published evidence is inadequate to show whether there is a change in incidence or a link with measles, mumps and rubella vaccine.”

Then it goes on to the possibilities of a pre-disposition. Do you recall if that paper was, so far as you can remember, in the original submission, the paper originally submitted?
A Those sentences? I certainly remember that “we did not prove an association”, that sentence, was in the final accepted manuscript, yes.

Q When the paper was submitted to you and considered at the manuscript meeting and logged as we have seen with PP beside it, were various amendments then made to the paper in agreement with the authors?
A Yes, that is absolutely routine practice in peer review. Often quite substantial change to papers are made.

Q Then ultimately a decision has to be taken as to whether to publish with those amendments in place, is that correct?
A Correct.

Q If we go back to page 645 we see, the fourth entry down on that page, 11096, now called “new syndrome in children”, is that correct?
A Correct.

Q We see the editor initials JB and then “accepted as ER”, although it is cut off, is that right?
A Yes. ER stands for early report.

Q Rather than asking you to explain that, I would like to go back to look at a description which is given in a editorial which deals with writing for The Lancet, page 615, on the left-hand side, half-way down the page:

“Early reports may simply be preliminary, the first results from a study, whereas subsequent analysis is planned, for example, of an incidental interesting observation from a study set up with another purpose or they may be early in the sense of being well short of changing clinical practice. These papers will tend to be shorter than articles.”

Does that broadly sum up what is meant by early report?
A Yes. What we were trying to do then – and I should say we have dropped the section Early Reports in more recent years – was to offer the opportunity for researchers to identify something at a very early stage before we were absolutely cast solid certain that what was being reported was totally factual. It gave the opportunity for new ideas, for innovation, to be included in the medical literature which we thought was an extremely important function of a medical journal.

Q This report was categorised as an early report and indeed it appears at the top of the actual paper. Can you help us as to which category it fell into? We see here two categories: the first results from a study where a subsequent analysis is planned, or an incidental interesting observation from a study set up with another purpose. Did you have any need to analyse?
A You are very generous in crediting our categorisation in precise terms. The way we felt about early reports was: is this a preliminary observation; is this raising something that is completely original that requires more in-depth investigation to confirm whether it is absolutely true or not. What we were trying to do in instructions to authors, because you can never cover every eventuality in written guidelines, the closest it would come to would be may be early in the sense of being well short of changing clinical practice.

Q I think it is right that when it was actually published it was published with a commentary.
A Yes. There were several mechanisms that we tried to focus on for making sure that when this paper was published it would not cause an adverse public health impact: one was obviously the statement that was obvious already in the paper about no proof of causation or association; a second was making sure that this paper was identified as an early report; but third, and possibly most importantly, we wanted to have external respected experts in measles eradication and control to offer their view. For us the comments that we published was vital in trying to set the context, which was essentially look at this paper with an open mind but please remember that measles vaccination has saved many hundreds of thousands of children’s lives and in considering this first report do not let it have an adverse effect on measles vaccination.

Q Can I ask you, first of all, how common was it at that the time for you to commission this commentary? You commissioned this commentary, is that correct?
A Yes.

Q How common was it for you to commission a commentary to go with a paper that was being published?
A Not uncommon if we were concerned about the interpretation of a paper but much less common than it is today. I would say today in almost every case a research paper will have a comment running with it.

Q Is the comment sought from somebody wholly independent from The Lancet and the researchers?
A Independent of The Lancet certainly. It is very hard sometimes to have people who are completely independent of the investigators. They are often experts in the field and fields, even globally, are often quite small. They will almost certainly be aware of the research or be aware of the investigators. They may even know the investigators very well professionally but we hope they will give an independent judgment about the quality and meaning of the paper, yes.

Q Your commissioning of it in the context of this paper. You have told us that as far as you were concerned, the reference to the link with MMR was relatively tentatively expressed. Did you nonetheless have concerns about the impact it might have?
A We did have concerns. These concerns were raised by the reviewers of the paper and they were also raised by my colleagues and myself in discussion of the paper on a Thursday afternoon. It was clear that if we were going to move ahead and publish this paper, we had two options: either we erased or asked the authors to erase the parental testimony about the possible temporal association with MMR vaccine or, if we were going to publish, we keep that in, but we give as much context as we possibly can.

Q If we could look at the commentary at page 788, it was commissioned from Robert Chen and Frank DeStefano, who work at the Vaccine Safety and Development Activity National Immunisation Program in Atlanta, Georgia. Is that correct?
A Correct. I think just to clarify, the Centre for Disease Control and Prevention is an internationally recognised centre for public health, based, as it is, in the United States, but with very strong global recognition.

Q If you do not mind bearing with me for a moment, just so the Panel can see what this is about – I am not going to read the whole of this, but if we can just run through it – we can see it says:

“Although immunisations rank among the most important public health measures, no vaccine is perfectly safe. Because vaccines are given to millions of healthy people, usually infants, extremely high standards for vaccine safety are demanded. It is therefore important to examine, critically and with an open mind, the report by Andrew Wakefield and colleagues of several children whose chronic bowel and behavioural abnormalities were linked by their parents and physicians to measles, mumps and rubella (MMR) vaccination.”

Then it sets out the various ways that adverse events of a vaccine can be said to be caused:

“ .. if it is associated with a specific laboratory finding and a specific clinical syndrome or both. Alternatively, a clinical or epidemiological study is needed to find out whether the rate of a given syndrome in vaccinated individuals exceeds that expected among unvaccinated controls. Such studies require acquisition of data in an unbiased way. Because of the inherent methodological limitations of epidemiological studies, biological plausibility, consistency, strength and specificity of association must also be considered in inferring causation. How well then do the features of the association reported by Wakefield and colleagues fit with causality?”

Then they point out:

“First, hundreds of millions of people worldwide … have received measles-containing vaccine without developing either chronic bowel or behavioural problems sine the mid-1960s. This finding provides important negative evidence as well as an appropriate framework for the assessment of [the paper].”

It goes on:

“Is the syndrome reported today clinically unique? Ileal lymphoid hyperplasia is non-specific. Autism was known well before MMR vaccine became available. Are there unique laboratory features, including detection of vaccine viruses in clinical specimens where they would not be expected? Although Wakefield has reported the detection of these viruses in patients with inflammatory bowel disease (IBD), other investigators, using more sensitive and specific assays, have not been able to reproduce these findings.”

Then it refers to a negative report which was actually in the same copy of The Lancet. It goes on:

“There is no report of detection of vaccine viruses in the bowel, brain or other tissue of the patients … ”

Then they look at the epidemiological questions:

“Is there selection bias? The Wakefield report is based on cases referred to a group known to be specially interested in studying the relation of MMR vaccine with IBD, rather than a population-based study. A first dose of MMR is given to about 600,000 children every year in the UK, most during the second year of life, the time when autism first becomes manifest. Not surprisingly, therefore, some cases will follow MMR vaccination. Biased case-ascertainment, as in this study, will exaggerate the association.”

Then it says:

“Was there recall bias. It is usually difficult to date precisely the onset of a syndrome such as autism. Parents and others may attempt to relate its onset to an unusual event such as coincidental postvaccinal reaction. The clearest example of such an association was the link between infantile spasms and pertussis vaccine;”

That is, the whooping cough vaccine –

“ … the vaccine tends to unmask rather than cause the syndrome.

There are other reasons for doubt about the association reported by Wakefield and colleagues. They suggest that MMR immunisation may lead to IBD, which results in malabsorption, consequent neurological damage, and ‘autism’. However, behavioural changes preceded bowel symptoms in almost all their reported cases.”

They go on to say:

“Vaccine-safety concerns gain prominence whenever the incidence of vaccine-preventable diseases falls to negligible levels and when the number of vaccine adverse events, whether true reactions or those coincidental to the vaccination but falsely attributed to it … rises as a consequence of high vaccine coverage. False attribution usually occurs because many developmental abnormalities first manifest in the early years of life, which is also when several vaccines – which can cause crying, fever, and, occasionally, febrile seizures – are given.”

Then it underlines the need for effective and credible systems for the detection of vaccine associated adverse events and it ends, you may think, rather prophetically, by saying:

“Without such a system, vaccine-safety concerns such as that reported by Wakefield and colleagues may snowball into societal tragedies when the media and the public confuse association with causality and shun immunisation. This painful history was shared by the UK (among others) over pertussis in the 1970s after another similar case series was widely publicised, and it is likely to be repeated all too easily over MMR. This would be tragic because passion would then conquer reason and the facts again in the UK.”

You published that commentary in those terms, Dr Horton. Did you feel that that was a responsible way forward, given the concerns which you have expressed?
A At the time, most certainly we did.

Q As far as you were concerned, did it highlight the criticisms which could be made in relation to the paper which you were publishing?
A It highlighted the criticisms that, as I recall, were made at the peer review stage, the concerns about possible bias. It highlighted what we were most anxious about, which was any adverse effect which might follow on MMR vaccination, but it also, fairly, we thought at the time, said, “Treat this study with an open mind.”

Q Did the paper in fact result in a very significant amount of correspondence to The Lancet?
A I think you might say that!

Q You say it in that tone of voice. Tones of voice do not always come over in the transcript. Are you suggesting that it was exceptional?
A Well, remember the context. The context was that when the paper was published, it was not published in a medical journal; it was launched, I think would be an appropriate word, at a press conference where other statements were made which were radically different from the statements made in the paper.

Q As far as the press conference is concerned, I think it is right that you did not attend. Is that correct?
A That is correct.

Q But Dr Bignall, the editor directly involved, did attend.
A That is correct.

Q You obviously cannot tell us anything about the press conference, because you were not present, but what I would like you to deal with is this. Did The Lancet have anything to do with the arranging of that?
A No, it did not, sadly.

Q How usual is it as an occurrence for there to be a press briefing or conference prior to the publication of a scientific paper?
A It is not common, but it is increasingly so, because often institutions, funding bodies and authors themselves want to make a splash of their paper to get more publicity for it, especially if it has something important to say. That can be for wholly good reasons. If there is a concern about the efficacy of a treatment or the adverse effect of a treatment, then it is very important that that gets wide publicity.

Q We have in the bundle an example of some of the correspondence which was generated. If you go to page 818, please – I am not going to take you through all that correspondence, Dr Horton, because I am going to invite the Panel to take some time at the end of your evidence to read the documents which we are producing – but just dealing with it very briefly, from page 818, this is the March edition of The Lancet, the paper having been published in February, we see the first letter, for instance, from the Programme on Immunisation of the World Health Organisation, the second one is from the Department of Public Health at Barnsley Health Authority. Then we have a letter which seems to come from a personal address and then at page 819 one from the Scottish Centre for Infection and Environmental Health. Going over to page 820, one from the Institute of Child Health. Would it be fair for me to summarise it by saying that those letters were mainly concerned with concerns as to the public health implications of the paper?
A Oh, absolutely.

Q We see underneath that on page 821 a reply by Dr Wakefield. Again, would it be the norm for you to give publication room to the author of a paper, if that paper has been the subject of significant criticism?
A Yes, indeed. We would consider it a fair way to conduct the debate that there would be responses, but in the same issue allow the original author to respond or in this case, as you can see, the authors divided in their responses.

Q We see, as you say, one from Dr Wakefield and then one from Professor Murch,
Dr Thomson and Professor Walker-Smith. At the end, we see also your own reply:

“The Lancet has been quick to criticise scientific and journalistic exuberance about the release of data that might unduly aggravate public concern. By contrast with these past episodes and with the implied criticism in the letters we publish this week, the paper by Andrew Wakefield and colleagues is an example of how researchers, editors and those concerned with the public’s health can work together to present new evidence in a scientifically balanced and careful way. Wakefield et al informed the UK Department of Health of their findings in 1997 and supplied them with a final copy of their Lancet paper in advance of publication.”

You then say, “(Wakefield A.J, personal communication)” So this arose as a result of information which Dr Wakefield gave to you. Is that right?
A That is right, yes.

Q You say, “There are at least four parts to this story.”
A I should just say, I do not have the second page of that. It goes on.

Q I think that is the end of your reply in relation to the correspondence.
A I think I probably would have gone on to explain what the four parts of the story were. Otherwise it would have been an extremely negligent reply on my part.