Comment on “Timing of Increased Autistic Disorder Cumulative Incidence”

19 Jan

In 2010 a paper was published called “Timing of increased autistic disorder cumulative incidence.” The paper has made very little, if any, impact on the scientific community. But it has become part of the stable of poor quality papers used by those claiming that vaccines caused an autism epidemic.

The paper took data from other published papers and applied a “hockey-stick” analysis to try to identify change points in the administrative prevalence of autism in California, Japan and Denmark. Here’s the main figure from that paper (click to enlarge)

Figure-1-from-Timing-paper

The idea of a hockey-stick analysis is to fit the data to two lines of different slopes which meet at a change point. Those two lines look like a hockey stick, hence the name. For multiple reasons, I believe this analysis was not appropriate for these data.

The paper is being discussed in the literature. Once by Helen Ratajczak in her review paper Theoretical aspects of autism: Causes – A review. Another citation comes from a published response to that review: Coincidental associations do not provide proof for the etiology of autism. Also “Hypothesis: Conjugate vaccines may predispose children to autism spectrum disorders“.

Two additional papers citing the Timing paper include “Mast cell activation and autism” (funded by the National Autism Association, an organization which promotes the vaccine-autism epidemic idea) and “Oxytocin and autism: a hypothesis to research. Can perinatal oxitocinergic manipulation facilitate autism?” (in Spanish).

In my view, much like Ms. Ratajczak’s review, the major impact of “Timing of Increased Autistic Disorder Cumulative Incidence” has not been in the scientific literature. An internet search quickly shows that both papers have been quite well received by those promoting vaccines as a cause of autism, both within part of the autism/parent community and from the anti-abortion community. The “Timing” paper was immediately promoted by David Kirby in an article at the Huffington Post (Mr. Kirby was a major promoter of the idea that mercury caused an autism epidemic). The paper has since been picked up by many, including Andrew Wakefield who attempts to give his interpretation of a “hockey stick” analysis in his talks (click to enlarge):

Wakefield-Jamaica

The “Timing” paper is, quite frankly, weak at best. Weak enough that I am unsure why the authors’ superiors at the EPA chose to approve it even with the disclaimer, “Approval does not signify that the contents reflect the views of the Agency” (a disclaimer which Mr. Kirby ignored as he made comments like “according to the EPA” in his piece). With much better analyses of the California Data by Peter Bearman’s group at Columbia and Irva Hertz-Picciotto‘s group at U.C. Davis, the time for such simple analyses as in the MacDonald and Paul paper is past. Especially in a highly charged area such as autism.

If I had room given the word count restrictions on a reply I would have included some of these points. Instead in “Comment on Timing of Increased Autistic Disorder Cumulative Incidence” I focused on three points. First, the source that MacDonald and Paul used for their California data has a very clear and explicit disclaimer about the fact that those data are not high enough quality for scientific research. Second, the data are exponential. One can fit a “hockey-stick” to exponential data but the results are meaningless. There is no change point in an exponential curve. Third, plotting the data shows that there are change points, but at 1960 and 1974, not 1988 as MacDonald and Paul claimed from fitting one of the exponential regions of data.

In their original paper, MacDonald and Paul point out: “All data were taken from the publications with no attempt to access the original data.” This, as I pointed out in my comment, was unfortunate because the CDDS makes their data available to the public. This would allow one to double check hypotheses, such as whether a “hockey-stick” analysis is appropriate. For many reasons, it is an inappropriate analysis.

First, the California Department of Developmental Services (CDDS) make it clear that these data are not to be used to draw scientific conclusions. From the report where the EPA authors gathered their data:

The information presented in this report is purely descriptive in nature and standing alone, should not be used to draw scientifically valid conclusions about the incidence or prevalence of ASD in California. The numbers of persons with ASD described in this report reflect point-in-time counts and do not constitute formal epidemiological measures of incidence or prevalence. The information contained in this report is limited by factors such as case finding, accuracy of diagnosis and the recording, on an individual basis, of a large array of information contained in the records of persons comprising California’s Developmental Services System. Finally, it is important to note that entry into the California Developmental Services System is voluntary. This may further alter the data presented herein relative to the actual population of persons with autism in California.

If one ignores this major point (as the EPA authors did), there are still other reasons why their analysis method is inappropriate. One big reason is that trying to look for a single “change point” year in California isn’t supported by the data. The fact that autism rates vary dramatically by geography within California (as shown by both Prof. Hertz-Picciotto’s group and Prof Bearman’s group) points away from any universal exposure (such as vaccines). The data I have from the CDDS which breaks down the counts by region only go back to the early 1990’s, so with this and space considerations I did not included these data. These geographic data make it clear that not only do the autism rates vary by region, the time trends of those rates vary a great deal from one region to another. In other words, what is a change point for one region may not be one for another. Applying a single change point to all of California is not warranted using these data.

Another reason why the hockey-stick analysis is inappropriate is the fact that it forces a functional form to the data which is plainly a bad fit. A hockey-stick analysis fits the time trend to two lines with a “change point” where the lines intersect. Unfortunately, the data are exponential. The result is quite remarkable, really, given the geographic variability and the changing social influences on autism rates.

If one takes one of the CDDS datasets (I used one from 2007) and combines it with census type data, one can produce this figure (Figure 1 from the published comment):

Comment-figure

Graphing the data on a log-normal graph such as this shows that the data are exponential. Going all the way back to birth year 1930. It isn’t a simple exponential, though. There is a region around 1960 to 1974 where the growth stalled. It is remarkable that the same time constant fits the data all the way back to 1930, with the exception of this 1960-1974 region.

Fitting exponential data to two lines just doesn’t make sense. There is no “change point” in an exponential. One can force a fit onto exponential data, but it isn’t meaningful.

Using the log-normal plot I supplied one can see that there are change points in the trend. Obvious to any observer. But they are in 1960 and 1974, not in about 1987/88 as MacDonald and Paul calculated.

As is customary, MacDonald and Paul supplied a reply to my comment. In this they make only a very brief reference to the fact that the very document from which they pulled the California data states it is inappropriate to use it the way they did: “We agree with Carey (3) that analysis of long term epidemiological studies would be desirable and that there are a number of potential confounding issues associated with analysis of administrative databases.”

One mistake I made was in not clearly spelling out that fitting a hockey-stick to exponential data is inappropriate. It is obvious, but rather than address this problem MacDonald and Paul state:

Changepoints were determined by fitting a hockey-stick model (10) to the data for each dataset. This approach uses ordered data and piecewise linear regression to split the response variable into two groups. A linear regression line is generated for each group, and the point of intersection for these regression lines and the residual sum of squares for each line are determined. The intersection point that minimizes the residual sum of squares is the changepoint.

Carey (3) used a log transformation of the cumulative incidence to produce a log-linear relationship for the CDDS data of the form: Log (Cumulative Incidence) = B0 + B1 (Year). Subsequently, he states that he could not observe changes in the log-linear relationship of CDDS cumulative incidence at or around our changepoint year of 1989, but no other analysis was performed. Examining original CDDS data in the inset of Carey’s (3) Figure 1, it certainly seems likely that there is a changepoint in the 1985-1990 range, and being unable to observe such a change in the log-linear plot may be purely an artifact of the scaling of the plot. We conducted a changepoint analysis on transformed CDDS data from 1970 to 1997 (from (7)) and found a changepoint in 1984. The shift to an earlier changepoint using the log transformed data may result from stabilization of the variance associated with the transformation, and the resulting shift in the minimization point for the residual sum of squares for the regression line for the larger cumulative incidence values in later years.

It’s an odd response. The authors are focused on defending their original result of a change point in the 1980’s rather than considering the entire new dataset. They ignore the problems inherent in claiming a change point in exponential data, but I should have stressed that more in my comment. Even if MacDonald and Paul claim it is appropriate to make this fit, they ignore the obvious change points in the log-normal graph. Consider the change point at about 1960. It is abundantly clear in the log-normal graph. In the inset of my figure, the linear graph, that change point is still obvious to the eye.

If the real goal of their work was to identify change points there is no reason to ignore those which were (a) outside of their original time span and (b) obvious in a different presentation of the data. This is not just flawed, it is irresponsible. They are ignoring their own stated goal:

As we point out in the paper, while artifacts associated with observed increases in various studies cannot be ruled out, from a precautionary standpoint, it seems prudent to assume that at least some portion of the observed increases in incidence is real and results from the interaction of environmental factors with genetically susceptible populations. Since exposure to environmental factors is potentially preventable, identification of relevant candidate factors should be a research priority.

Why, I would ask, are potential environmental candidates which might involve change points in 1960 and 1974 not important, but one in the late 1980’s is?

Advertisements

13 Responses to “Comment on “Timing of Increased Autistic Disorder Cumulative Incidence””

  1. Neuroskeptic January 21, 2012 at 10:27 #

    Interesting stuff. But I don’t see why this hockey-stick theory, even in the original form you criticize, has been so eagerly adopted by anti-vaccine types.

    One could equally well say that the late 1980s represents the start of the modern era of high awareness of autism and increased diagnosis.

    Notably, the original Autism Diagnostic Interview was developed in 1989… and the late 1980s was the time of the publication of many of the classic research papers that set the groundwork for the modern understanding of autism.

  2. Jen in TX January 21, 2012 at 16:35 #

    It was also around the time that acetaminophen replaced aspirin for treating childhood fevers, combined with J&J’s aggressive marketing of Tylenol after the cyanide tamperings in 1982 & 1986.

    http://on.fb.me/x3tmac

    OOPS!

  3. Science Mom January 21, 2012 at 17:40 #

    Jen, why don’t you produce some real research into your pet hypothesis before making such vapid and embarrassing claims okay? That silly little graph can be generated using any number of factors such as increase in number of televisions/household, CD players and bad hairstyles.

  4. Jen in TX January 21, 2012 at 18:23 #

    I’m guessing you missed Dr. Shaw’s webinar on this very subject last night? Too bad. It was very informative, and touched on many of the things I’ve been hollering about for the past 3 years.

    But methinks you are more interested in sniping at me than you are in the science behind this, because this theory, if proven true, validates the claims of those who said their kids regressed after shots, and those who were involved with making those recommendations for pre and post vax Tylenol use, as well as those who recklessly foisted this drug on the public, will have to face responsibility for this.

    But…just in case I’m wrong about you and you are interested in evidence, you can find numerous articles that show the many, many problems with acetaminophen (not just autism!) on my FB page. Once you’ve read everything there, you can come back and tell me why we shouldn’t be aggressively pursuing this. Or, you can easily find Dr. Shaw on FB, too, and I’m sure he would be happy to answer any questions you have about this. His FB page could use some skeptical points of view, so have at it!

    Have a lovely day, Science Mom. 🙂

    • Sullivan January 21, 2012 at 18:30 #

      Jen,

      I see you skipped over much discussion of why it is inappropriate to do what you are doing and just grabbed the info that supports your preconceived position.

      I’ve seen Shaw and his kind come and go. I’ve seen the nearly fraudulent way the Tylenol hypothesis was promoted in the literature by seeking input from a very biased group of parents and withholding that information from reviewers and referees.

      The idea of a single cause for the rise in autism diagnoses is as close to definitively wrong as any science question can be.

  5. Neuroskeptic January 21, 2012 at 18:41 #

    “Autism is caused by MMR”
    (years later)”…we’ve looked into it, and it’s not.”
    “I meant mercury.”
    (years later)”…we’ve looked into it, and it’s not.”
    “I meant acetaminophen”.

    In a few years you’ll be onto something else, so why not give up now and move on early? It would save time in the long run.

    I reckon you should forget acetaminophen and skip ahead to the next culprit. Which is (looks into his crystal ball) Prozac or something like that.

    Mark my words, in 5 years we will be having the same discussion about another drug.

    • Sullivan January 22, 2012 at 03:18 #

      “In a few years you’ll be onto something else, so why not give up now and move on early? It would save time in the long run.”

      Experience tells us she won’t change her mind. Wakefield is still chasing MMR. Blaxill is still chasing mercury.

      Others will come along with other ideas. The movement is more of a snowball, gathering disparate theories and never abandoning any.

  6. Science Mom January 21, 2012 at 21:08 #

    I’m guessing you missed Dr. Shaw’s webinar on this very subject last night? Too bad. It was very informative, and touched on many of the things I’ve been hollering about for the past 3 years.

    I’m not interested in mewlings of self-proclaimed expurts; show me the data, simple. If he has any research on the topic then post it. You found a guy (a dodgy one at that) who says what you want to hear; colour me unimpressed.

    But methinks you are more interested in sniping at me than you are in the science behind this, because this theory, if proven true, validates the claims of those who said their kids regressed after shots, and those who were involved with making those recommendations for pre and post vax Tylenol use, as well as those who recklessly foisted this drug on the public, will have to face responsibility for this.

    I’m interested in both but you aren’t as you don’t have the data to support your claims. You don’t seem to know what a theory is Jen so please stop using the word to lend credibility to your rantings about acetaminophen. I’ve seen what you have to present and thus far, nothing you have shown me remotely supports your claim. You can’t even decipher the Waring papers which used Tylenol as a probe and nothing to do with autism.

    But…just in case I’m wrong about you and you are interested in evidence, you can find numerous articles that show the many, many problems with acetaminophen (not just autism!) on my FB page. Once you’ve read everything there, you can come back and tell me why we shouldn’t be aggressively pursuing this. Or, you can easily find Dr. Shaw on FB, too, and I’m sure he would be happy to answer any questions you have about this. His FB page could use some skeptical points of view, so have at it!

    That’s not how it works; I don’t do FB and you are the one making fantastical claims and hiding behind a self-proclaimed not-authority so you show the citations for the studies here. You have them handy so do so and I also don’t seek clarifications from quacks who don’t even have any research; I talk to real scientists with actual peer-reviewed studies. Stop being lazy and dramatic, post the studies or shut it.

  7. Moderation January 22, 2012 at 05:53 #

    I think everyone here is missing Jen’s point. If you re-read her post you will see that she has found the the “one true cause” of all illness: Tylenol! And while she does not have the research to back it up. She does have her unfailing intuition – evidence be damned.

  8. daedalus2u January 26, 2012 at 17:13 #

    Sully, that graph is very interesting. I can’t get your comment where I am now, could you email it to me? Thanks.

    The timing of the transition from one exponential to the next sort-of coincides with atmospheric nuclear testing.

    http://en.wikipedia.org/wiki/File:Radiocarbon_bomb_spike.svg

    It is probably not 14C. But there were other radioisotopes.

    It also sort-of coincides with the widespread use of synthetic detergents and then conditioning shampoo.

Trackbacks/Pingbacks

  1. Autism Blog – Comment on “Timing of Increased Autistic Disorder … | My Autism Site | All About Autism - January 20, 2012

    […] Go here to read the rest: Autism Blog – Comment on “Timing of Increased Autistic Disorder … […]

  2. A fishing expedition at Vaccine Court | Left Brain Right Brain - July 14, 2013

    […] the method set forth by two people at the FDA who presented such a changepoint analysis previously. I found that analysis lacking and submitted a comment to the journal on it. I find Ms. Deisher’s analysis lacking as […]

  3. Autism, Denmark and again no link with vaccines. | Left Brain Right Brain - August 25, 2013

    […] recurrence risk doesn’t change with time, so that’s good evidence against such an idea. There is no changepoint in the California data in the 1980′s, as it is exponential and fitting it to two straight lines is just a mistake. What about the […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: