Huffington Post uses erroneous data to promote autism epidemic

26 Aug

correction:

As noted in the comments below, Mr. Kirby appears to be basing argument suggesting that the Hepatitis B vaccine could have caused autism on ADDM data, not on the NSCH dataset, as I assumed.

A recent blog post on the Huffington Post contains serious errors and should be edited or pulled.  At the very least a public acknowledgment of the error must be made.

Using data from the recently published 2007 National Survey of Children’s Health to estimate autism prevalence, a Huffington Post blogger (David Kirby) attempted to draw a connection between the Hepatitis B vaccine and an “explosion” of autism . Here is what he wrote:

If there is an environmental component to autism, hopefully scientists will want to know which exposures might have increased between, say, 1992 and 1996.

One possible answer is the Hepatitis B vaccine, (which also contained 25 micrograms of mercury containing thimerosal).

Introduced in 1991, it was the first vaccine ever given on a population basis to newborn babies (within the first three hours after delivery) in human history.

But according to the CDC’s National Immunization Survey (which also includes parental telephone interviews), only 8% of infant children received the Hep B vaccine in 1992, when that birth cohort showed an ASD rate of 60-per-10,000.

By 1994, the number of children receiving Hep B vaccine had reached just 27% — and the cohort showed an ASD rate of 66-per-10,000.

But the Hep B coverage rate had risen to 82% by 1996, when that cohort’s ASD rate exploded to around 100-per-10,000.

Correlation, obviously, does not equal causation. And no one is suggesting that Hepatitis B vaccine is the singular “cause” of autism. But the uptake rate of that particular immunization is at least one environmental factor that did demonstrably change during the period in question.

Emphasis is mine. I emphasized the data which  are the data that are incorrect.

The analysis is simple. Here are the actual results compared to what Mr. Kirby misreported:

1992 “birth cohort*”:
102 per 10,000 (not 60 per 10,000 as on HuffPo)

1994 “birth cohort*”:
113 per 10,000 (not 66 per 10,000 as on HuffPo)

1996 “birth cohort*”:
111 per 10,000 (close to the “around” 100 per 10,000 quoted).

Or, to put it very simply: Mr. Kirby’s statement that there was an “explosion” in the autism rates is incorrect. The evidence that the introduction of the Hepatitis B vaccine is somehow related to the increase in autism rates is false.

That entire statement isn’t even a misinterpretation–it is just simply, demonstrably, false.

Unfortunately, this isn’t Mr. Kirby’s first clear and serious error. He has a history of mistakes. Unfortunately, he doesn’t have a history of correcting his mistakes. Consider these examples:

In June 2008, epiwonk publish a blog post “David Kirby: HuffPost Report on CDC’s Vaccine Safety Datalink Uninformative and Completely Misleading“, demonstrating clear errors in Mr. Kirby’s post “CDC: Vaccine Study Design “Uninformative and Potentially Misleading“”.

The errors were serious enough that Mr. Kirby rewrote his blog post as CDC: Vaccine Study Used Flawed Methods. This included the following introduction:

NOTE: My original post on this topic mischaracterized the 2003 CDC vaccine investigation as an “Ecological Study,” which it was not. I am reposting this piece to reflect that information accurately, but also to point out that many of the weaknesses identified in the CDC’s data and methods apply to the published 2003 “retrospective cohort” study, as much as they do to any future “ecological” ones. I regret and apologize for the error.

Mr. Kirby “regrets” and “apologizes” for the error. Yet his original, erroneous blog post is still on the Huffington Post website. He never took it down. He didn’t even add an apology or correction note to the piece. Anyone following a link to it would have no idea that even the author acknowledges the serious flaws in that piece.

It is also worth noting that the “corrected” version of Mr. Kirby’s blog post was also in error. Again, as noted by epiwonk, this time in his piece “David Kirby HuffPost, Take 2: My Original Story was Flawed, So Here’s A Second (”Corrected”) Story That’s Still Flawed, But I Hope I Can Snow You Under Again This Time…

Mr. Kirby compounded this error when he recreated it in his first “congressional briefing”, September 2008. Mr. Kirby misquoted a report by the National Institute of Enviornmental Health Sciences, and he was caught by a knowledgeable staffer.

Again, Mr. Kirbty has failed to correct his error.  He posted his power point presentation to his website, but without any acknowledgment of the error on page 6.  In the transcript for this talk, he only states, “NOTE: This statement omits important details of the CDC response” and sends you to other sites “For a more detailed explanation”. The “transcript” makes no reference to the exchange between Mr. Kirby and the congressional staffer, nor does it acknowledge that the omission was critical to the point being made. The transcript is noted as being a “Rush transcription by Nancy Hokkanen”. Being in a rush is not an excuse to leave important flaws unexplained.

Math errors are also not new to Mr. Kirby. In May 2008, Mr. Kirby wrote a piece analyzing data from Scotland. In doing so, Mr. Kirby misread a graph resulting in a factor of 10 error in a key piece of information (he misread a bar graph . After his error was blogged, Mr. Kirby corrected his Huffington Post piece. What he didn’t do, and he should have, was to note in the blog piece that he made the error and corrected it.

Mr. Kirby placed his Scotland data post in two sites: Huffington Post and the Age of Autism blog. In yet another odd move by Mr. Kirby, he left the original version of his post, complete with the factor of 10 error, on the Age of Autism blog (it still has 34,000 instead of 3,400). As noted above, Mr. Kirby obviously knows about the error, since he corrected it on the Huffington Post.

Since he clearly knew that his post on the Age of Autism blog had a big error, why didn’t he make a correction (with acknowledgment of the error) there?

Mr. Kirby had a bit of a problem with understanding the difference between Change.Org Change.Gov (the Obama transition team’s website) and Change.Org (a website that hosts blogs on important topics, including autism) (also noted here and here) He made a clear correction on the Huffington Post. However, his post on the Age of Autism blog just disappeared without a comment.

But let’s get back to the present. Mr. Kirby has blogged erroneous data and used this to show a false correlation between the Hepatitis B vaccine introduction and the rise in autism rates.

In case anyone is thinking, “are you sure you checked your own numbers, Sullivan?” The answer is yes. I double checked. I asked a frequent commenter on this blog, Dawn, to check my numbers. Another commenter independently collected and graphed the NCSH data as well. No evidence for an “explosion” of autism rates. Take a look at the graph. Mr. Kirby claimed that the 2007 survey data showed autism rates of about 60/10,000 for kids aged 13 and 15. There are no rates below 80 per 10,000 for the kids in those age ranges in that dataset.

So here we have a man with a history of errors, and with a history of failing to adequately correct his errors. He now has a new, big, obvious error. This error is likely the most serious of those listed here, in my opinion. Mr. Kirby has convinced people that the Hepatitis B vaccine could be causing autism. That was a serious accusation, and it was wrong. The question before us now is this: what will Mr. Kirby do now that he knows he made a mistake?

I’m very curious about that, so I’ve emailed Mr. Kirby and one of the editors at the Huffington Post with this information. I’ll let you all know what I hear back.

*note: the NSCH data are not given as “birth cohorts”. Instead, they are given by age. The survey was performed in 2007 and 2008. So, the 15 year old age group is roughly the “1992 birth cohort”. Likewise, 13 year olds are the 1994 “cohort” and 11 year olds are the 1996 “cohort”.

EDIT: Note that I too have a problem with keeping Change.Org and Change.Gov separate. This correction was made after the post was published.

26 Responses to “Huffington Post uses erroneous data to promote autism epidemic”

  1. Liz Ditz August 26, 2009 at 17:55 #

    Oops, Sullivan

    “Mr. Kirby had a bit of a problem with understanding the difference between Change.Org Change.gov (the Obama transition team’s website) and Change.Org (a website that hosts blogs on important topics, including autism) (also noted here and here) He made a clear correction on the Huffington Post. However, his post on the Age of Autism blog just disappeared without a comment.”

    • Sullivan August 26, 2009 at 18:09 #

      Thanks for catching that one Liz.

      I thought when I published this that there would likely be a few mistakes–and given the nature of this post, people would notice and comment on the mistakes.

      I’ll correct the mistake and make note of it in the post.

  2. kev August 26, 2009 at 19:05 #

    in my opinion one of David’s biggest errors is stating again and again that US govt has conceeded that Hannah Polings autism was caused by vaccines when no such statement has ever been made public. And lets mot forget the serious blow to the thiomersal idea in 2005 2007 and i think this year David promised but now fails to see or discuss.

    • Sullivan August 26, 2009 at 21:12 #

      in my opinion one of David’s biggest errors is stating again and again that US govt has conceeded that Hannah Polings autism was caused by vaccines when no such statement has ever been made public. And lets mot forget the serious blow to the thiomersal idea in 2005 2007 and i think this year David promised but now fails to see or discuss.

      Kev,

      One thing I tried to do here was stick to errors that are so clear that Mr. Kirby couldn’t argue his way out with an “interpretation”. His handling of the entire mitochondria/autism story is riddled with mistakes.

  3. Ringside Seat August 26, 2009 at 19:35 #

    I shouldn’t think David Kirby could care less whether his stats are right, or whether he ought to correct his mistakes.

    • Sullivan August 26, 2009 at 21:21 #

      I shouldn’t think David Kirby could care less whether his stats are right, or whether he ought to correct his mistakes.

      Ringside Seat:

      Normally would suggest caution before making such statements–it is really hard to understand intent. But in this case we have multiple examples where Mr. Kirby has failed to correct obviously erroneous statements.

  4. Dawn August 26, 2009 at 22:41 #

    Hi, Sullivan. I thought the graph sounded interesting when I read your post, but couldn’t pull it up at work. Did it when I got home. It is interesting. Think I’ll try to pull the 2003 datasets like I did the 2007 and run them through my program and see what kind of numbers I get for 2003. One nice thing about SAS…once you’ve learned to use it, it’s easy to look at multiple data sets. I also think I’ll see what the gender numbers look like; I didn’t take the time to break the data down that way. I’ll let you know what I find. 🙂

  5. Dawn August 26, 2009 at 22:43 #

    Also…debating about registering to put a comment on Huffpo but since the comments looked a bit old I don’t know if anyone would actually read that the numbers are wrong.

  6. Dan Hollenbeck August 26, 2009 at 22:52 #

    Sullivan,

    What makes you assume David Kirby is using NSCH data in the quoted paragraphs?

    When I read his original blog article I found the following text where he is using the identical autism prevalence numbers and cohorts but in a context of the ADDM network. The 1992 and 1994 cohort prevalence numbers appear to be from ADDM and 1996 cohort is hearsay from the Adventures in Autism blog.

    So far, ADDM has published data from just two birth cohorts: children born in 1992 (eight-year-olds in 2000) and those born in 1994 (eight-year-olds in 2002). The 1992 cohort revealed an estimated ASD rate of one in 166, or 60-per-10,000.

    For the 1994 cohort, the estimate was revised upward to one in 150, or 10% higher, at 66-per-10,000.

    CDC officials have been analyzing the 1996 birth cohort (2004 data on 8-year-olds) for years. I asked the agency a few months ago about the slow progress in releasing the numbers and was told that the data were currently “under review.” No response was given to written questions about data collected from the 1998 or 2000 cohorts (in 2006 and 2008, respectively).

    I also submitted a Freedom of Information Act request to the CDC for the raw data it had collected to date. That request is still pending.

    But just the other day, the Adventures in Autism blog reported that CDC was about to release its 1996 birth cohort data, and that those data would also show ASD prevalence rates along the lines of 100-per-10,000, or a whopping one percent of US children.

    Are you saying David Kirby is using ADDM data, but he is incorrectly quoting the ADDM data?

    Also, your blog article could be improved if there was a way to link directly to the NSCH data web page you are using other than just linking to the home page of NSCH. It is clear the NSCH site was developed by the lowest bidder so directly linking might not be possible.

    Thanks, Dan
    Dan Hollenbeck

  7. Sullivan August 26, 2009 at 23:07 #

    Dan Hollenbeck,

    It seems plausible that he is using the ADDM data. That would be a great thing for David Kirby to communicate–both here and in his blog piece.

    If so, he could then describe why the NSCH dataset–which is the major focus of his blog piece–totally contradicts the point he is trying to make.

    I’ll be happy to admit my mistake if it there and especially if David Kirby will address the contradiction.

    • Sullivan August 27, 2009 at 00:02 #

      I’ve edited the post to note the error.

      Dan, boy I wish I had talked to you before I posted this. I had assumed that David Kirby had merely made a mistake in his analysis. I didn’t realize that he had ignored the dataset which directly contradicted his argument in favor of data that isn’t even published or known yet.

      That would have been such a better blog piece. Making errors in SAS, especially when you try to import into Excel, is so easy and explainable. That’s what I figured happened. But, choosing to ignore the NSCH dataset…dang. I’ll give him the benefit of the doubt that he just neglected to look at the data rather than suggest he purposely ignored it.

  8. MJ August 27, 2009 at 00:34 #

    “No evidence for an “explosion” of autism rates.”

    Uhm, if you lined up the data from the NSCH by birth cohort instead of by ages it does show an increasing number of children with autism. Take a look at the following chart – the 2003 data is lined up to the age the child would have been in 2007 (ie age 2 is age 6)

    I think the trend for the chart is clear. Whether you call that “explosive” or not really depends on what the word means to you. I would call 40 per 10,000 (age 21 – 2003) to 120 in 10,000 (age 8 – 2007) a definite increase.

    You can’t really look at a survey that that represents a single point in time and draw the conclusion the rates aren’t rising. It would be better if there were more than two data sets but we are going to have to wait for 2013 for that.

    Also, I think you had best fess up to your own errors before you criticize others for their mistakes. For example, you made numerous errors in your last post using NSCH data ( https://leftbrainrightbrain.co.uk/?p=2777 ), see http://autismjabberwocky.blogspot.com/2009/08/how-not-to-compare-data.html .

    • Sullivan August 27, 2009 at 01:03 #

      Uhm, if you lined up the data from the NSCH by birth cohort instead of by ages it does show an increasing number of children with autism. Take a look at the following chart – the 2003 data is lined up to the age the child would have been in 2007 (ie age 2 is age 6)

      http://lh5.ggpht.com/_WoSi1HFM…..s_2003.gif

      I think the trend for the chart is clear. Whether you call that “explosive” or not really depends on what the word means to you. I would call 40 per 10,000 (age 21 – 2003) to 120 in 10,000 (age 8 – 2007) a definite increase.

      Sorry, the graph is too small to see. I’ll take your word on it.

      I’ve compared the two datasets (2003 and 2007) as well. I wouldn’t blog it. Why? Because you can’t compare the 2003 and 2007 datasets. The questions asked are not comparable. In 2003, they asked about “autism” in 2007 they asked about “autism or ASD”.

      If you want to compare 2003 to 2007 data, go ahead. If you can’t see the huge error you are making, that’s fine. We’ve already discussed that on this site, in a different blog post.

      You can’t really look at a survey that that represents a single point in time and draw the conclusion the rates aren’t rising. It would be better if there were more than two data sets but we are going to have to wait for 2013 for that.

      Actually, you can. If you look at various ages and use consistent criteria, you can. Absolutely. Without question.

      Also, I think you had best fess up to your own errors before you criticize others for their mistakes. For example, you made numerous errors in your last post using NSCH data ( https://leftbrainrightbrain.co.uk/?p=2777 ), see http://autismjabberwocky.blogs…..-data.html .

      I’ll take your word for it. I make mistakes. I note them (see the post above).

  9. MJ August 27, 2009 at 01:07 #

    “Sorry, the graph is too small to see. I’ll take your word on it.”

    My bad, I guess I choose the “thumbnail” option by mistake. This one is actually visible.

    • Sullivan August 27, 2009 at 01:36 #

      Here’s your graph:

      [note–I tried to insert the graph above and failed)

      So, your 16 year olds. I assume that is 16 years old in the 2007 survey. That makes them 12 in the 2003 survey.

      That very same age group increased from a rate of 40 (in 2003) to 90 (in 2007).

      If you want to compare these datasets, you have to address the question: where were the 50 kids per 10,000 who weren’t counted in 2003 for that age group?

      Seriously, for the same approximate birth year, the “rate” doubled in 4 years. This for kids who were 12 years old at the start–old enough to supposedly be accurately counted.

      That doesn’t make sense if the rate is defined the same way. Which it wasn’t.

  10. MJ August 27, 2009 at 01:53 #

    Sullivan you said –

    “I’ve compared the two datasets (2003 and 2007) as well. I wouldn’t blog it. Why? Because you can’t compare the 2003 and 2007 datasets. The questions asked are not comparable. In 2003, they asked about “autism” in 2007 they asked about “autism or ASD”.”

    But you will directly compare a measure incidence against one showing prevalence and that is somehow more appropriate? Plus you neglect the fact that the question from 2007 is also narrower than the one from 2003 as well – in 2003 the word used is “ever” had autism but in 2007 it is “currently” have autism (or asd). Which do do think is going to have a larger impact?

    “Actually, you can. If you look at various ages and use consistent criteria, you can. Absolutely. Without question.”

    Didn’t you say here – https://leftbrainrightbrain.co.uk/?p=2740 – the first time that you mentioned the NSCH data set that –

    “The real question here is whether the prevalence of autism is really increasing. It is a very good question. It just isn’t one that we can answer with the data we have.”

    So what you are now saying is that the data in this same set, while it cannot be used to answer the question of whether the prevalence of autism is rising can in fact be used to prove that it isn’t?

  11. Sullivan August 27, 2009 at 02:05 #

    “But you will directly compare a measure incidence against one showing prevalence and that is somehow more appropriate?”

    First, I noted the error. Second the “incidence” data you refer to isn’t “incidence” at all. The author of that graph hasn’t added a comment to his website, nor has he pulled the graph last I checked.

    You do realize that it isn’t incidence, right?

    ““The real question here is whether the prevalence of autism is really increasing. It is a very good question. It just isn’t one that we can answer with the data we have.”

    Yep. Accurate then and accurate now. We can look for trends and generate hypotheses from these data, but I wouldn’t really want to consider them scientific.

    Totally consistent with what I have said in this thread.

    There are some really interesting questions that can be raised from the NSCH dataset. Very interesting. But I wouldn’t

    I notice you are avoiding the question of how can you compare the 2003 and 2007 datasets. The answer is clear: you can’t.

    Too bad. It would be valuable to see what the data suggest. (note, suggest, not prove).

    Or, to answer your question directly:

    “So what you are now saying is that the data in this same set, while it cannot be used to answer the question of whether the prevalence of autism is rising can in fact be used to prove that it isn’t?”

    Nope. I can’t use these data to prove that it isn’t going up. I don’t purport to do so. If I have made a confusing statement here, point it out and let me correct it.

    What we can do is point out that the data do not support the idea that the prevalence went up. They are very suggestive that the Hepatitis B proposition is false. That, as I have said, directly contradicts Mr. Kirby’s argument. The NSCH dataset are not the highest quality, but neither are the ADDM data, I am sorry to say (and have said repeatedly on this blog). So, the comparison can be made. Now, if I were to attempt to pit the NSCH dataset against a true prevalence study where population screens and diagnostic evaluations were being used, you would have a valid point.

  12. MJ August 27, 2009 at 02:17 #

    “If you want to compare these datasets, you have to address the question: where were the 50 kids per 10,000 who weren’t counted in 2003 for that age group?”

    Really? I think the simple answers are all of the following –

    A) There were diagnosed between 2003 and 2007. There is no reason to think that an older child would not be diagnosed with autism, especially the “higher” functioning ones.

    B) The populations differed between the years, I don’t think they called back the same people from 2003, did you?

    C) Additional children added because of the addition of “ASD” to the question.

    You are implying that C dominates the data and makes it completely bogus. I think there is an effect there but it doesn’t come close to dominating it.

    If you stop to consider what the typical ratios of autism to pdd-nos to aspergers normally are you would realize that if the second two groups were completely excluded from 2003 then the values from 2003 would be much lower. If “classic” autism alone is running at 60 per 10,000 then we really are in trouble. And if the “classic” group was at that level in 2003, then adding in the other two groups (which are normally double the classic group) would have made the 2007 double or triple the 2003 numbers.

    Yet we don’t see that in the older ages in the comparison, just the younger ones where the numbers from 2003 would not be as reliable. Hence the addition of the ASD part is likely not dominating the comparision.

    “That very same age group increased from a rate of 40 (in 2003) to 90 (in 2007).”

    Not to quibble the actual numbers are 87.7 to 47.6 which is 85 percent increase, not a 125 percent increase like you indicate. Now if you look at age 16, that is about a 120 percent increase but you didn’t pick that point even though the different was higher.

    I am not saying that the chart is free from artifact or flaws but the direction of the rates as you go down in age is clear AND the data lines up decently (but clearly not perfectly) well between the two survey years.

    • Sullivan August 27, 2009 at 03:42 #

      I am not saying that the chart is free from artifact or flaws but the direction of the rates as you go down in age is clear AND the data lines up decently (but clearly not perfectly) well between the two survey years.”

      And if that is good enough for you, you have lower standards of quality data than I. Feel free to draw whatever conclusions you want.

  13. MJ August 27, 2009 at 02:29 #

    “If I have made a confusing statement here, point it out and let me correct it.”

    Ok, tell me what I am misunderstanding then. First I wrote –

    “You can’t really look at a survey that that represents a single point in time and draw the conclusion the rates aren’t rising. It would be better if there were more than two data sets but we are going to have to wait for 2013 for that.”

    To which you replied –

    “Actually, you can. If you look at various ages and use consistent criteria, you can. Absolutely. Without question.”

    Which leads me to think they you are saying that you can look at just the NSCH data set from 2007 and draw the conclusion that autism rates aren’t rising.

    To me this contradicts your statement on an earlier post that –

    “The real question here is whether the prevalence of autism is really increasing. It is a very good question. It just isn’t one that we can answer with the data we have.”

    Which says to me that you can’t look at the data and draw the conclusion that the data is rising.

    So where I am misunderstanding what you are saying?

    • Sullivan August 27, 2009 at 03:41 #

      This statement:

      “Actually, you can. If you look at various ages and use consistent criteria, you can. Absolutely. Without question.”

      You are correct–that was not a correct response to your question. I misread “survey” as “study” On certainly can do a study in a single point in time and get quality data to draw conclusions.

      The follow up statement would be this: One can not do two surveys at different times and draw hard conclusions. Sorry, surveys just aren’t that high quality. It’s a survey. No one checked that the kids are actually autistic. No one checked if parents of autistic kids are more likely to respond to the calls, as far as I know.

      Glad to clear that up for you.

  14. MJ August 27, 2009 at 04:29 #

    “And if that is good enough for you, you have lower standards of quality data than I”

    Well, you know me, the lower quality the data, the more I like it. And I must saw, what a well thought out, logical response.

    • Sullivan August 27, 2009 at 04:56 #

      Actually, MJ, it is.

      The survey data are good for some things. They are not good for others. If you think that comparing two different datasets which asked significantly different questions is a valuable thing to do, then you have different ideas

      Sorry if the statement was too terse–but I’ve gone over the reasoning for that many times. Many times. If the last time appears terse after many long discussions on the very same topic, I apologize.

  15. Dawn August 27, 2009 at 14:04 #

    Morning, Sullivan. Just reviewed the data and emailed you. Some interesting numbers, I thought. Actually, given the more restrictive question, the numbers aren’t too far off what I expected to see, since I think more children are diagnosed with ASD or PDD instead of Autism.

  16. Stephanie August 28, 2009 at 03:12 #

    I’m sorry. Maybe this question is beside the point, but why would a survey be an appropriate method to measure prevalence or incidence of autism? In the U.S., there are projects that mine medical files and school records for this data. Wouldn’t these be more reliable means to get the information? My understanding is that as long as the patient’s identity is kept confidential there are no (legal) privacy issues involved. Between the diagnosticians and the school system, they should be able to get a fairly accurate picture of pravelence and incidence over time for a specific location.

Trackbacks/Pingbacks

  1. Twitter Trackbacks for Autism Blog - Huffington Post uses erroneous data to promote autism epidemic « Left Brain/Right Brain [leftbrainrightbrain.co.uk] on Topsy.com - August 28, 2009

    […] link is being shared on Twitter right now. @lizditz, an influential author, said HuffPo Blogger uses […]

Leave a Reply to Stephanie Cancel reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: