Copyright © 1995 by Steven J. Milloy. All rights reserved. First edition. Published by the Cato Institute, 1000 Massachusetts Avenue, N.W., Washington, D.C. 20002. Library of Congress Catalog Number: 95-72177. International Standard Book Number: 0-9647463-2-8.
Chapter 4
"Data" Collection
Back in the very first chapter, we said a good risk is ubiquitous and has exposures that are difficult to impossible to measure. Remember dioxin? How can you measure exposures that are found everywhere all the time? The advantage of exposures that are so difficult to measure is you don't have to "measure" them at all. You can just make 'em up.
But few people will ever notice or learn that your exposures aren't real. And it's almost impossible to validate your methodology or verify your results. So not only do you get to make up the data, but no one can check on you. Although this clearly violates one of the basic tenets of the scientific method (i.e., results should be capable of replication), it's a distinct advantage in the risk business.
Unfortunately, there are no general guidelines about how to do this. As in biological plausibility, you need to be creative. Maybe some of the examples we talk about here will inspire you.
Making the most of irrelevant documentation
Let's consider those epidemiologic studies that attempted to associate dioxin with cancer. Dioxin is ubiquitous in the environment; everybody is exposed to it and everybody carries some of it around in their fatty tissues. How do you study dioxin when everyone is exposed to it as part of everyday life? Isn't it impossible to find people who have not been exposed? Yes, absolutely. Has that stopped the risk assessors? Absolutely not!
Because dioxin is a by-product of certain industrial chemical processes, some epidemiologists theorized that chemical plant workers — by virtue of their employment status — should have relatively higher exposures to dioxin than the general public. So if dioxin is associated with cancer, there should be a higher incidence of cancer among these workers than among the general population. This theory almost makes sense.
So the epidemiologists decided to do a case-control study (the quick-n-dirty type, remember?) of the cancer rate of former chemical plant workers. Through employment records, our valiant epidemiologists chose subjects who worked in chemical processes in which it was assumed that some occupational exposure could have occurred. They also assumed the longer a worker worked in a process where dioxin exposure was possible, the greater the worker's exposure to dioxin.
The beauty of this scheme is that it is highly intuitive. It makes sense that chemical workers probably come into contact with chemicals. The longer they work with chemicals, the greater their exposure to these chemicals. This line of reasoning, however, says nothing about which specific chemicals the workers used, whether the chemicals included dioxin, whether dioxin was the only chemical the workers came into contact with or how much exposure to any chemical actually occurred.
Studies relying on this technique (which, by the way, have generally reported weak to very weak statistical associations) have been highly regarded by the public health community. Even though this ignores the metaphysical certainty that the chemical plant workers were exposed to many chemicals, not just dioxin. Or that it's impossible to tell from employment records if workers were even exposed to dioxin on the job.
From a purely scientific point of view, this data as well as these results mean absolutely nothing. But that's the beauty of this particular technique. It's so intuitive that it can overcome its own fatal shortcomings.
Thanks for the memories
Another useful data collection technique is the survey method, a technique that could be compared to political polling. It's not nearly as reliable, though. But it is very simple and straightforward. All you need is a series of "relevant" and preferably "loaded" questions that can be posed to a population of interest.
Typically, this involves asking your subjects whether they remember being exposed to the risk you're investigating. You may ask them to recall something from 50 years before. It's truly an amazing technique, considering people often can't remember what they had for dinner last week.
Sure, you must rely on the interviewees' memories of decades earlier. But typically, when you tell them what you're doing (trying to find someone to blame for their illness), they will be more than happy to remember whatever you want. Consider the diesel exhaust epidemiology studies.
Epidemiologists looking at diesel exhaust selected study subjects based solely on their status as workers who, at some time, either operated diesel equipment or worked in the vicinity of operating diesel equipment. For dioxin studies, researchers relied on employer-maintained employment records as the primary data source. But the diesel exhaust epidemiologists conducted personal interviews with former workers, most of whom were elderly and many of whom had lung cancer.
This is masterful... let's see, I'm going to ask people dying of lung cancer if they think diesel exhaust caused their illness. Do you think they'll try harder to remember a past exposure to explain the illness? It's called recall bias, it's pure genius, and it's guaranteed to give the "right" result.
The survey technique can even be taken one step further, if necessary. Maybe the person you want to interview is no longer alive. What should you do? The answer is simple: just interview the next of kin, like a son or daughter. Don't worry if they were not even around at the time of interest. Don't worry about using hearsay evidence. That's a trivial detail no one will ever get around to noticing anyway. You'd just be wasting good recall bias.
Unfortunately for the diesel exhaust epidemiologists, many independent and unbiased experts have since concluded the lack of definitive exposure data for diesel study populations precludes the use of available epidemiologic data to develop quantitative estimates of cancer risks. You mean we can't estimate cancer risks for things just because we don't like them?
I guess the dioxin epidemiologists got lucky.
Next Week: Chapter 5 Mining for Statistical Associations
Click here to return to the home page
Copyright © 1996 Steven J. Milloy. All rights reserved. Site developed and hosted by WestLake Consulting.