Recently a paper has made some headlines, and
the message getting out seems to be that living near a farm field where there
has been pesticide applications has been found to increase the risk of Autism
spectrum disorder. A few things about the paper. First, one of the things I
admire about econometric work is the attempt to make use of some data set, some
variable, or some measurement to estimate the effect of some intervention or
policy, in a world where we can’t always get our hands on the thing we are really
trying to measure. The book Freakonomics comes to mind, or quasi-experimental
designs and the use of instrumental
variables.
Second, I’m not an epidemiologist, entemologist, or have a background in toxicology. I don’t have
the subject matter expertise to critique this article, but I can express my
appreciation for the statistical methods that they used. While the authors
could not (or simply did not)
actually measure pesticide exposure in any medical or biological sense, they
attempted to infer that distance from
an agricultural field might correlate well enough to proxy for exposure. That is a large assumption and perhaps one of
the greatest challenges of the study. It
is not a study on actual exposure. So I’ll try to only refer to exposure from this point in quotes. But the authors did make clever use of
some interesting data sources. They matched up required reported pesticide applications
and report dates with zipcodes of the study respondents and reported pregnancy
stages to determine distance from application and at what point of their
pregnancy they were exposed. They
reported distance in three bands or buffer zones of 1.25, 1.5, & 1.75 km This was actually nice work, if distance could be equated to
some known level of exposure. Unfortunately, while they cited some other work
attempting to tie exposure to ASD, I did not see a citation in the body of the
text where any work had been done justifying the use of distance as a proxy, or
those particular bands. More on this later. They also attempted to control for
a number of confounders, applied survey weighting to ‘weight up’ the effects to reflect the parent population, and in
addition, at least based on my reading, may have even tried to control for some
level of selection bias by using IPTW
regression with SAS.
Discussion of Results
There were at least four major findings in the
paper:
(1) Proximity to organophosphates at some point
during gestation was associated with a 60% increased risk for ASD
(2) higher for 3rd trimester exposures [OR = 2.0, 95% confidence
interval (CI) = (1.1, 3.6)],
(3) and 2nd trimester chlorpyrifos applications: OR = 3.3
[95% CI = (1.5, 7.4)].
(4)Children of mothers residing near pyrethroid
insecticide applications just prior to conception or during 3rd trimester were
at greater risk for both ASD and DD, with OR's ranging from 1.7 to 2.3.
So where do we go with these results? First off
all of these findings are based on odds ratios. The reported odds ratio in the
first finding above was 1.60 which implies a [1.6-1.0]*100 = 60% increase in
odds of ASD for ‘exposed’ vs ‘non-exposed’ children. This is an
increase in odds, and does not have
the exact same interpretation as an increase in probability. (see
more about logistic regression and odds ratios here). Some might read the
headline and walk away with the wrong idea that living within proximity of farm
fields with organophosphate applications constitutes ‘exposure’ to organophosphates and is associated with a 60% increased probability of ASD,
but that is stacking one large assumption on top of another misinterpretation.
However, these findings are but a slice of the
full results reported in the paper. Table 3 reports a number of findings across
the distance bands, types of pesticide, and pregnancy stage. One thing about
odds ratios, an odds ratio of ‘1’ implies no effect. The vast majority of these
findings were associated with odds ratios with 95% confidence intervals
containing 1, or very very close to 1. For those that like to interpret
p-values, a 95% CI for an odds ratio that contains 1 implies that the estimated
regression coefficient in the model has a p-value > .05, i.e.
non-significant results.
Another interesting thing about the table, is
that there doesn’t seem to be any pattern of distance/pregnancy stage/chemistry
associated with the estimated effects or odds ratios. A point made well in a
recent blog post regarding this study at scienceblogs.com here.
Sensitivity
From the paper: “In additional analyses, we evaluated the sensitivity of the estimates
to the choice of buffer size, using 4 additional sizes between 1 and 2km:
results and interpretation remained stable (data not shown).”
That’s unfortunate too. Given the previous
discussion of odds ratios, lack of empirical support or literature related to
using distance as a proxy for exposure, you would think more sensitivity
analysis would be merited to show robustness to all of these assumptions even if
and especially if there is no previous precedent in the literature related to
distance. This in combination with
the previous discussion regarding the large number of insignificant odds ratios
and select reporting of the marginally significant results is probably what
fueled accusations of data
drudging.
Omitted Controls
From the Paper: “Primarily, our exposure estimation approach does not encompass all
potential sources of exposure to each of these compounds: among them external
non-agricultural sources (e.g. institutional use, such as around schools);
residential indoor use; professional pesticide application in or around the
home for gardening, landscaping or other pest control; as well as dietary
sources (Morgan 2012).”
So, there are a number of important routes of
exposure that were not controlled for, or perhaps a good deal of omitted
variable bias and unobserved heterogeneity. The point of my post is not to pick apart a study linking
pesticides to ASD. There are no perfect data sets and no perfect experimental
designs. All studies have weaknesses, and my interpretation of this study
certainly has flaws. The point is, while this study has made some headlines
with some media outlets, and seems scary; it is not one that should be used to
draw sharp conclusions or to run to your legislator for new regulations.
This reminds me of a quote
I have shared here recently:
"Social scientists and policymakers alike seem driven to draw
sharp conclusions, even when these can be generated only by imposing much
stronger assumptions than can be defended. We need to develop a greater
tolerance for ambiguity. We must face up to the fact that we cannot answer all
of the questions that we ask." (Manski, 1995)
References:
Manski, C.F. 1995.
Identification Problems in the Social Sciences. Cambridge: Harvard University
Press.
Neurodevelopmental Disorders and Prenatal Residential
Proximity to Agricultural Pesticides: The CHARGE Study
Janie F. Shelton, Estella M. Geraghty, Daniel J. Tancredi,
Lora D. Delwiche, Rebecca J. Schmidt, Beate Ritz, Robin L. Hansen, and Irva
Hertz-Picciotto
Environmental Health Perspectives. June 23, 2014
No comments:
Post a Comment