A look at the analysis plan for DeStefano’s MMR study: no evidence of fraud

16 Oct

Andrew Wakefield and Brian Hooker have been making claims that the CDC are involved in misconduct in autism research. In case you haven’t followed the story, it basically goes like this:

1) the CDC planned on a study of MMR and autism using the MADDSP data.

2) That the CDC created a research plan.

3) That the CDC found results they didn’t want to report: an calculated odds ratio for African American boys. So the CDC team allegedly deviated from that plan and didn’t report that result.

4) That the CDC introduced a new analysis after the plan: that they would include birth certificate data.  While the CDC rationale for this new analysis was to provide more data (confounding variables) for the analysis, the allegedly real reason was to dilute the sample set and make statistically significant results disappear.

Here’s a paragraph from one of the press releases about the Hooker study:

According to Dr. Thompson’s statement, “Decisions were made regarding which findings to report after the data was collected.” Thompson’s conversations with Hooker confirmed that it was only after the CDC study coauthors observed results indicating a statistical association between MMR timing and autism among African-Americans boys, that they introduced the Georgia birth certificate criterion as a requirement for participation in the study. This had the effect of reducing the sample size by 41% and eliminating the statistical significance of the finding, which Hooker calls “a direct deviation from the agreed upon final study protocol – a serious violation.”

Or so goes the story. But as is often the case with Andrew Wakefield and Brian Hooker, the facts don’t match the claims.

In a recent video, Mr. Wakefield shows us the research plan the CDC had drafted.  One red flag with Mr. Wakefield’s approach so far has been how he tries to tightly manage the flow of information.  He has not shared the analysis plan in total and only now has he provided us with a couple screenshots.  Begs the question: what are they hiding?

Here’s one screenshot from that video. This one is where he gets the idea that the plan was to report race for the entire sample.

draft analysis plan screenshot 2

Here’s the full text, in case that’s difficult to read:

Statistical Analysis

We will use conditional logistic regression stratified by matched sets to estimate the odds ratios for association between age at MMR vaccination and autism. In the main analyses, we will include all autism cases.

Potential confounding variables will be evaluated individually for their association with the autism case definition. Those with an odds ratio p-value < 0.20 will be included as covariates in a conditional logistic regression model to estimate adjusted odds ratios for the association between age at vaccination and autism. The only variable available to be assessed as a potential confounder using the entire sample is child’s race. For the children born in Georgia for whom we have birth certificate data, several sub-analyses will be carried out similar to the main analyses to assess the effect of several other potential confounding variables. A recent case control study (CDC, 2001) carried out with a subset of the autism cases from this study found that age matched cases and controls differed on several important background factors including maternal age, maternal education, birth type, and parity. The variables that will be assessed as potential confounders in this study will be birth weight, APGAR scores, gestational age, birth type, parity, maternal age, maternal race/ethnicity, and maternal education. (See Table 2 for how variables will be categorized.)

There are two interesting points in the above.  First, the sentence Mr. Wakefield highlights doesn’t say what he claims.  The only variable available to be assessed as a potential confounder using the entire sample is child’s race. The plan doesn’t say that they will test and report race.  Consider the context: this is a section of the plan called “statistical analysis”. Put in context with the entire paragraph, this sentence is clear: the full dataset is limited because it only has one variable available.

The CDC didn’t deviate from the plan when they didn’t report on race for the total sample because that was never in the plan.  If you want more evidence of this, the end of the paragraph says “See Table 2 for how variables will be categorized”.  Table 2 is titled “Descriptive Statistics for Children Born in Georgia with Birth Certificate Records”.  The variables will be categorized in the birth certificate sample.

The second interesting point from the paragraph Mr. Wakefield has shown us is this: the CDC plan included a birth certificate sample.

Here’s a screenshot of the analysis plan from that new video, showing the front page of the analysis plan:

draft analysis plan screenshot

Shown with this voice over by Mr. Wakefield (while the screenshot above is shown going up in flames…very dramatic)

“Over the ensuing months, after the data after the data had been collected and analyzed, and strictly forbidden in the proper conduct of science, the group abandoned the approved analysis plan, introducing a revised analysis plan to help them deal with their problem.”

So, in case you were thinking, “that’s an analysis plan, how do we know it’s the analysis plan”, well, you have Mr. Wakefield’s word on it.  This is the “approved analysis plan” that the CDC allegedly had to revise.

What interests me about this as that’s the same plan that I have and was preparing to write about.  It’s nice now to be able to be able to say that this is, indeed, the same document that Mr. Wakefield and Mr. Hooker are working with.

We’ve already seen two big mistakes by the Wakefield/Hooker team: first that the analysis plan doesn’t include a call to report on race separately in the total sample (the group without the birth certificates), second that the CDC “approved analysis plan” included analysis of a subset with birth certificate data.

So, what were the objectives of the study as in the plan?

Objectives:
We did not have information regarding onset of symptoms for most cases in this study and this limited our ability to do certain types of analyses such as case series analyses. In addition, a totally unexposed group (i.e., never received the MMR vaccine or other measles containing vaccine) was not available since measles, mumps, and rubella vaccination are required for school attendance in Georgia. The following objectives are considered the primary objectives for this study.
1) To determine if case children were more likely than their matched controls to have been vaccinated with MMR before 36 months of age. DSM-IV criteria for autism require that onset of symptoms occur before 36 months of age. Therefore, the 36-month cut-off is one that by definition can be used to classify a definitely “unexposed” group.
2) To determine whether there was a difference between cases and controls in the proportion of children exposed to their first dose of MMR vaccine before 18 months of age. This objective is based on the research that suggests the timing of first parental concern for the development of autism appears around 18 months of age (Taylor et al, 1999). In addition, Cathy Lord has reported that the range of first parental concern for regression was between 12 and 23 months of age with a mode of 19-21 months.
3) To determine whether the age distribution for receipt of the MMR vaccine differs between cases and controls.

They showed the data for the 36 and 18 month cutoffs.  Age distribution was covered in Table 2.

Analysis of Autism subgroups

The IOM (2001) specifically recommended additional research regarding autism subgroups and MMR. We will examine several subtypes of autism in this study. Data from the Metropolitan Atlanta Congenital Defects Program will be included in the sub-analyses to identify particular sub-groups. The following sub-group analyses will be conducted:

1) Analyses excluding cases with an established cause for autism or a co-occurring condition suggesting an early prenatal etiology (e.g., tuberous sclerosis, fragile X, or other congenital/chromosomal anomalies.)

We propose to conduct a case-control sub-analysis looking at cases without an established or presumptive cause for autism, such as tuberous sclerosis, fragile X, and other congenital/chromosomal anomalies. The purpose of doing this analysis is to create a more homogeneous case group that may be more likely to be impacted by the timing of the MMR vaccine. The objectives from the primary analyses will be replicated in this sub-analysis.

2) Analyses of Isolated versus Non-isolated Autism.

Isolated autism cases are cases with no other co-morbid developmental disability while non-isolated cases do have a co-morbid developmental disability. Previous research suggests that the majority of non-isolated cases have a co-existing developmental disability of mental retardation (CDC, 2001). Both isolated and non-isolated cases will be compared separately to controls. The objectives from the primary analyses will be replicated in this sub-analysis.

3) Analyses examining Gender Effects

Males are at substantially higher risk for autism and may be more vulnerable to the exposure associated with the MMR vaccine. We will analyze males and females separately and replicate the main objectives of the primary analyses as well as examine the potential confounders available from Georgia birth certificates.

4) Analyses excluding autism cases with known onset prior to 1 year of age.

For a subset of autism cases, we were able to identify the timing of parental concern. This sub-analysis will exclude all cases excluded with an established or presumptive cause for autism (e.g., tuberous sclerosis, fragile X, and other congenital/chromosomal anomalies.) and children for whom we have been able to identify first parental concern prior to 12 months of age.

Just in case anyone reading this is one of the few that has been following Mr. Wakefield’s video releases: in a new video Mr. Wakefield is trying to claim that the isolated autism subanalysis was not done.  Except that it was.  They made a minor change to autism without MR, which gave essentially the same result that Mr. Wakefield claims was hidden.

Destefano_table_4 highlighted

Autism without MR has an odds ratio of 2.45 with a 95% confidence interval of 1.20 to 5.00.  I’ll write about this new video soon as there’s much sleight of hand going on, but Mr. Wakefield is claiming that a result of odd ratio = 2.48 with confidence interval of 1.16 to 5.31 was not reported.  Besides ignoring the fact that the data were reported by the CDC, Mr. Wakefield ignores the fact that these are raw-data results.  Total sample, unadjusted analysis.  In the adjusted analysis the result does not suggest an association.

But, getting back to the main point: the claims of fraud are just not founded on fact.  The two main claims of “fraud” are just wrong.  The analysis plan did not state that they would do a subanalysis by race for the total sample.  The addition of the birth certificate data is in the plan, not in some sort of revision.  And Mr. Wakefield and Mr. Hooker knew this.

I am reminded of a quote from an ABC News article recently

“There are always going to be those people at the edges of science who want to shout because they don’t want to believe what the data are showing,” said Dr. Margaret Moon, a pediatrician and bioethicist at Johns Hopkins Berman Institute of Bioethics. She said she thought the study author “manipulated the data and manipulated the media in a very savvy and sophisticated way.”

“It’s not good. It’s not fair. It’s not honest. But it’s savvy,” Moon said.

By Matt Carey

Advertisements

24 Responses to “A look at the analysis plan for DeStefano’s MMR study: no evidence of fraud”

  1. Jake Crosby October 17, 2014 at 02:56 #

    It only says “Draft of Analysis Plan.”

    From Thompson: “…I believe that the final study protocol was not followed.”

    • Sullivan (Matt Carey) October 17, 2014 at 05:28 #

      Yep, it says draft. Which is why I was happy that Andrew Wakefield confirmed that this is the document from which they got their idea that race was supposed to be reported for the total sample.

      Thompson: “I believe…”

      He also believed he could trust Andrew Wakefield and Brian Hooker. I’ll take what is written over belief.

      Ands, even if a draft (filename includes “final” as received by the way) it shows that the CDC planned to do a birth certificate analysis at the start. In fact, I’f you think this is just a draft that means the birth certificate analysis was thought of well upstream.

      So,junk analysis by Hooker, resulting in bad results that were not worthy of publication, CDC planned in advance to do a birth certificate analysis, no intent to use race as a confounding variable for the total sample…what exactly is left other than betrayed trust and amateur videos?

      • reissd October 17, 2014 at 14:51 #

        Parents that may be misled into a world of undeserved guilt and anger and children that may be subjected to untested, dangerous treatments or left unprotected against diseases because of a fake conspiracy.

    • reissd October 17, 2014 at 18:06 #

      Thompson has not told us in what way the protocol was not followed. If he makes a concrete claim, it can be looked at. General “beliefs” really aren’t good enough to overturn a paper.

      If the claims are the ones made by Hooker and Wakefield, as the post above demonstrates, they are wrong to dishonest.

      • Sullivan (Matt Carey) October 17, 2014 at 18:26 #

        Sure, there could be something else. Information that hasn’t been made public yet. But if whatever Mr. Thompson thinks is a breach of protocol is the same thing that Wakefield is claiming in this video, Thompson is wrong.

        And it wouldn’t change the fact that Mr. Wakefield is currently misleading people.

      • reissd October 17, 2014 at 18:28 #

        agreed.

    • lilady October 17, 2014 at 20:52 #

      Jake, you keep posting about extraneous issues on your own blog. IMO, you are blowing smoke about the publisher of Hooker’s reanalysis study’s removal and the later decision to fully retract Hooker’s paper, citing Hooker’s poor study design/execution, massaging of the statistics and the undeclared COIs on the part of Hooker and the (unnamed) peer reviewers. You then posted a long harangue about Retraction Watch, which, IMO, is another deliberate distraction.

      You are unwilling (or unable) to provide any plausible explanations for why Hooker did his “reanalysis” of the DeStefano, et al case control study, using a cohort study format.

      You’re making excuses on your blog, for why Hooker hasn’t found the time to reply to your email(s) and thus you find yourself in the untenable position of supporting Hooker’s fraudulent study.

    • Sullivan (Matt Carey) October 22, 2014 at 17:19 #

      Mr. Crosby,

      I take it you’ve read the complaint that came out today. Did you notice where Mr. Wakefield and Mr. Hooker changed key wording?

      The plan states:

      “The only variable available to be assessed as a potential confounder using the entire sample is child’s race. ”

      The complaint quotes this as:

      ” “The only variable that will be assessed as a potential confounder using the entire sample will be the child’s race.” ”

      Perhaps you have an explanation for why that was changed? And why Mr. Hooker signed on to it.

      • Lawrence October 22, 2014 at 17:34 #

        Because it makes better copy for the folks over at AoA & allows Wakefield to continue to solicit funds for his “documentaries.”

      • Sullivan (Matt Carey) October 22, 2014 at 17:50 #

        Yep. And if Wakefield can troll any one on his cc list into making a comment, he has fodder for his “documentary”. Too bad if they figure out after the fact that they were lied to.

  2. Narad October 17, 2014 at 06:43 #

    Perhaps the most impressive part of the Y—be presence of the “Autism Media Channel” is how little interest its uploads manage to draw. As a random point of comparison this Finnish cover of “Delta Dawn” beats even the most notorious of them.

    • Sullivan (Matt Carey) October 17, 2014 at 17:24 #

      Well, The Wakefield/Hooker smear video–where they claim the CDC are conducting a new Tuskegee experiment–got a lot of views from the CNN iReport website. I think they are hoping for another such hit.

  3. Chris Preston October 18, 2014 at 08:26 #

    Why am I not surprised that what Andrew Wakefield states in the video is not accurate?

    I had previously had an idea about what Thompson thought had been covered up. It clearly has to do with African-American boys vaccinated before 36 months of age, because that is what he stated in the press release and that is what Hooker has focused on. However, I can’t see anything in the Analysis Plan that covered a slice by race and gender, unless it is under section 3 on page 9.

    So perhaps it was just a case that Thompson mistook an unadjusted analysis for an adjusted analysis?

Trackbacks/Pingbacks

  1. Brian Hooker and Andrew Wakefield send a complaint to the CDC about its vaccine research. Everyone yawns. – Respectful Insolence - October 23, 2014

    […] #CDCwhistleblower thing at the time. So I let this cup pass for a change. Fortunately, Matt Carey took it on about a week ago, and that makes my job easier. It’s worth reading in its entirety, but the Cliffs Notes version […]

  2. Brian Hooker and Andrew Wakefield send a complaint to the CDC about its vaccine research. Everyone yawns. [Respectful Insolence] | Gaia Gazette - October 23, 2014

    […] #CDCwhistleblower thing at the time. So I let this cup pass for a change. Fortunately, Matt Carey took it on about a week ago, and that makes my job easier. It’s worth reading in its entirety, but the Cliffs Notes […]

  3. Andrew Wakefield and Brian Hooker complain. Not honestly, but they complain | Left Brain Right Brain - October 23, 2014

    […] like the Tuskegee experiment!” video. The thing is that the plan didn’t call for that. As I recently discussed, the sentence Mr. Wakefield misquotes was a statement of the limitations of the dataset they had […]

  4. Brian Hooker and Andrew Wakefield accuse the CDC of scientific fraud. Irony meters everywhere explode. | Healthy Lifestyle 101 - October 28, 2014

    […] #CDCwhistleblower thing. So I let this antivaccine cup pass for a change. Fortunately, Matt Carey took it on more than a week ago, and that makes my job easier. Matt’s post is worth reading in its entirety, but the Cliffs Notes […]

  5. Did a high ranking whistleblower really reveal that the CDC covered up proof that vaccines cause autism in African-American boys? « Science-Based Medicine - April 3, 2015

    […] #CDCwhistleblower thing. So I let this antivaccine cup pass for a change. Fortunately, Matt Carey took it on more than a week ago, and that makes my job easier. Matt’s post is worth reading in its entirety, but the Cliffs Notes […]

  6. Brian Hooker and Andrew Wakefield accuse the CDC of scientific fraud. Irony meters everywhere explode. « Science-Based Medicine - April 3, 2015

    […] #CDCwhistleblower thing. So I let this antivaccine cup pass for a change. Fortunately, Matt Carey took it on more than a week ago, and that makes my job easier. Matt’s post is worth reading in its entirety, but the Cliffs Notes […]

  7. A provaxer watched Vaxxed | vaccinesworkblog - July 10, 2016

    […] already been done BEFORE the film was made, Bigtree does not bother to interview other scientists, including Dr Matt Carey, for their opinion of the data analysis. He does not say anything about trying to contact Dr Carey […]

  8. In which Andrew Wakefield and Del Bigtree’s antivaccine documentary VAXXED is reviewed…with Insolence! – Respectful Insolence - July 18, 2016

    […] and autism in African American children. I’ve discussed this issue before, as has Matt Carey long before VAXXED and in his review of VAXXED. In the VAXXED narrative, CDC scientists didn’t like what they […]

  9. Vaxxed fumisterie – Initiative Rationnelle - January 27, 2017

    […] (27/07/2015): Matt Carey de Left Brain Right Brain au sujet du plan d’analyse de l’étude. Le sous-groupe des certificats de naissance faisait parti du plan approuvé pour […]

  10. A look back at the so called “CDC Whistleblower” story and how Vaxxed is misleading | Left Brain Right Brain - February 10, 2017

    […] A look at the analysis plan for DeStefano’s MMR study: no evidence of fraud Harpocrates Speaks on: MMR, the CDC and Brian Hooker: A Guide for Parents and the Media […]

  11. In which Andrew Wakefield and Del Bigtree's antivaccine documentary VAXXED is reviewed...with Insolence! - RESPECTFUL INSOLENCE - November 29, 2017

    […] and autism in African American children. I’ve discussed this issue before, as has Matt Carey long before VAXXED and in his review of VAXXED. In the VAXXED narrative, CDC scientists didn’t like what they […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: