It Doesn’t Look Like It’s Declining

22 Sep

David Kirby wrote a post titled Is Autism Declining? back in July (that I was reminded of after reading an entry by Sullivan.) I’ll address the primary claim of that post shortly.

For pattern and context, I would like to recap Kirby’s forays into the world of autism epidemiology. They seem to have started in November, 2005, when Citizen Cain explained to him that “new cases” or “intakes” were not a meaningful epidemiological metric, but that instead, he should be looking at the 3-5 caseload to get an indication of a possible drop in autism incidence.

When 2007 came and Kirby’s prediction of a drop in the California DDS 3-5 caseload famously failed, he apparently started to furiously scan various databases for any sign of a drop in “the numbers.” First,  he suggested we might be seeing a drop in speech-language impairment in IDEA. But the drops he was seeing were easily shown to be nothing out of the ordinary. Additionally, I found there were discrepancies between the IDEA data publicly available and the data he presented. Subsequently, David Kirby emailed a message to the EOHarm mailing list announcing he had found 4 states in IDEA with 3-5 caseload drops in autism between 2004 and 2005. Once again it was demonstrated that these drops were not unusual.

David Kirby never addressed these rebuttals. In fact, links to said rebuttals were not even allowed by whoever moderates messages in his blog at Huffington Post. But that is a different matter.

In his latest attempt at finding drops in the numbers, David Kirby tells us that while the California DDS 3-5 caseload is still rising as of Q2 2007, he obtained birth-year-cohort data that allowed him to look at 3 year olds in isolation.

But among the very youngest kids counted, the story was the opposite. At the end of June 2006, there were 688 children born in 2003 with autism diagnoses. This June, the number of kids born in 2004 with autism was 632, a statistically significant drop of 56 children, or 8.1 percent less than last year at this time.

I had sent a request to California DDS for this data, but I did not have to wait for it since a fellow blogger already had a copy. (Thanks.) I should note that I could not verify the 632 caseload number for June 2006, as the copy of the data I was given only ran to March 2006, but I’ll take David Kirby’s word for it.

I naturally decided to put the data in a graph. The following represents the caseload of autistic children in California DDS who should have been 3 years of age by June of each year shown.

cddsbyc22.jpg

You see that drop between 2006 and 2007? That is what David Kirby found which he believes is significant. Let’s be honest here. Does that look like a significant drop between 2006 and 2007, or does the spike between 2005 and 2006 look more significant than that? Caseload growth between 2005 and 2007 is actually pretty steep and in line with the previous trend, isn’t it?

Clearly, that’s a pretty weak finding and it can’t really tell us anything. If caseload drops to 1995 levels or something of the sort, then we’d be discussing something of interest. As things stand, 2007 has seen less autistic 3 year-olds than 2006, but a lot more than 2005. So? 

David Kirby, however, asserts it’s a “statistically significant” change. But I’m really not sure if this assertion is correct. Readers with a statistical background might be able to help with this one. If we assume a sample size of 540,000 (the population of 3 year-olds in California) and we use a simple confidence interval calculator (e.g. this one) the results I get are:

  • 2006: 688 (95% CI 638 – 741)
  • 2007: 632 (95% CI 584 – 683)

The confidence intervals (or error bars) overlap, so it does not appear that the change is statistically significant. But it’s possible David Kirby is using a different method to calculate the confidence intervals.

Conclusion

Data available is not sufficient to assert that the caseload of 3 year-olds served by California DDS might have begun a downward trend or even a leveling-off trend. Furthermore, the claim that the caseload change observed between 2006 and 2007 is statistically significant requires further support and clarification.

Advertisements

10 Responses to “It Doesn’t Look Like It’s Declining”

  1. Sullivan September 22, 2007 at 01:08 #

    Kirby has been following the Geier lead in “statistically significant”.

    They assume gaussian statistics or something and ignore the historical fluctuations in the data.

    Let’s see if/what he comes up with after the next quarterly report.

  2. Joseph September 22, 2007 at 01:23 #

    And another thing is that I’m sure he looked at many different angles. Did the caseload for 4 year olds drop? How about 5 year olds? Could I possibly make a claim about 2 year olds? How about data for March? (Too small a drop there.) October? No good. June – that’s the one.

    It’s the same thing he did when he found 4 states with drops in IDEA. As I noted, when he finds 25 states, we can start to talk about leveling off of the IDEA autistic population.
    If you look hard enough, you’ll find some fluctuation that on the surface is similar to what you want to show.

  3. Schwartz September 22, 2007 at 14:53 #

    Joseph,

    I thought there were so many general problems with the CDDS data that it precludes any meaningful analysis anyways?

    Is that still the general consensus?

  4. Joseph September 22, 2007 at 15:58 #

    Schwartz: I’ve been very clear about that in the past. The California DDS provides a wealth of information that is by no means useless. However, it’s important to understand the caveats of the data. For one, the children with an autism classification in CDDS is not the same as the total number of autistic children in California. This is an obvious point. And there is no way to know how accurate the count is, unless you were to do a whole population screening and compare.

    So when the data is used as evidence that the total number of ASD children in California has risen, this is clearly an invalid use of the data.

    Most children with an autism classification in CDDS do fall in some ASD category. I believe only 1% have been found to be completely outside the spectrum. In that sense, we can say that CDDS counts are a good approximation of a lower bound of the autistic population.

    There is other data of interest in CDDS, such as the proportion of autistic children evaluated as having mental retardation. By looking at drops in this proportion, there are indications of what one might call ‘broadening criteria’ or ‘more awareness’.

    Now, if David Kirby makes a claim about California DDS data, it stands to reason that a rebuttal would involve analyzing California DDS data.

  5. isles September 22, 2007 at 16:00 #

    “But it’s possible David Kirby is using a different method to calculate the confidence intervals.”

    Oh, Joseph, you kill me. Kirby hasn’t shown the mathematical acumen to figure the tip in a restaurant, much less calculate a confidence interval. He just likes to sound all sciencey by tossing out phrases like “statistically significant.”

    I’m not offering an opinion on the numbers, just snorting at the idea of Kirby having the wherewithal to do it.

  6. Sullivan September 22, 2007 at 23:11 #

    My guess on reading the “statistically significant” comment from Mr. Kirby was that it was probably from an email.

    “Dear David. Take a look, the numbers went down. I checkd and it is statistically significant. Sincerely, EOHarm member 1023”

  7. A Non September 23, 2007 at 00:26 #

    The real decline is among 2.5 year old Aquarians with blue eyes. Check it out David.

  8. Schwartz September 23, 2007 at 00:28 #

    “The California DDS provides a wealth of information that is by no means useless. However, it’s important to understand the caveats of the data. ”

    OK, I’m glad someone thinks so, and I agree it can at least provide a lower bound.

    I’m just so used to people explaining six different ways that the CDDS data is useless for any analytics regarding the rates of Autism — usually in response to Kirby.

  9. Joseph September 23, 2007 at 13:58 #

    To elaborate on that point, suppose the 3-5 caseload of autistics in CDDS grows to a level where it includes 2% of the 3-5 population of children in the state. Obviously, that wouldn’t be something that can be brushed off by saying “nah, that’s not epidemiological data”, unless one could show, for example, that most kids classified with ASD in the system are actually ADHD or something.

    But the numbers in CDDS have traditionally been low, so any gradual and smooth caseload increase can easily be explained as recognition catching up with the epidemiology.

    The current 3-5 CDDS prevalence of autism is roughly 40 in 10,000.

Trackbacks/Pingbacks

  1. In 1931… : Left Brain/Right Brain - September 27, 2007

    […] It Doesn’t Look Like It’s Declining […]

Comments are closed.

%d bloggers like this: