Copyright © 1995 by Steven J. Milloy. All rights reserved. First edition. Published by the Cato Institute, 1000 Massachusetts Avenue, N.W., Washington, D.C. 20002. Library of Congress Catalog Number: 95-72177. International Standard Book Number: 0-9647463-2-8.
Chapter 6
The Mixmaster Technique
What if you don't have the time or the money or the inclination to do your own epidemiologic study? What if others have already published epidemiologic studies on your risk but they didn't find anything convincing. Or some found something while others haven't? Well, just be very creative.
You could take the existing studies, assume that they are similar enough to be combined and, voila!, you have an entirely new study. This technique is called meta-analysis. The best way to demonstrate the power of meta-analysis is to show you the greatest masterpiece, the Mona Lisa, of all meta-analyses: the Environmental Protection Agency's risk assessment on environmental tobacco smoke (ETS). There simply is no better example of this technique at work.
At the time the ETS risk assessment was conducted, there were 30 published (and who knows how many unpublished) epidemiologic studies on ETS conducted in a number of countries. Of the 30 published studies, eight reported statistically significant associations between exposure to ETS and lung cancer; 22 other studies reported either no association or no statistically significant association. Of the 11 studies that examined U.S. populations, only one reported a statistically significant association.
Realizing the difficulty of credibly associating ETS with lung cancer based on conflicting studies, the ever-resourceful EPA chose meta-analysis. Using this technique, EPA combined the 11 U.S. ETS epidemiologic studies and came up with a relative risk of 1.19 that was statistically significant at a 90 percent confidence level. (Note: Even though their results weren't statistically significant at a 95 percent level they were resourceful enough to claim statistical significance at a lower level. Another clutch decision!) With this "statistically significant" relative risk, EPA went on to estimate 3,000 lung cancer deaths can be attributed to ETS every year.
What's so amazing about all this? Well, EPA did such a good job picking a target for its risk assessment and meta-analysis that the intrinsic characteristics of the target itself were strong enough to overcome the scientific deficiency of the meta-analysis.
ETS was a classic target. The risk was unprovable (any risk would be too small to find, a fact borne out when 10 out of 11 U.S. studies turned up nothing). ETS is a common exposure. The cause-and-effect relationship in question is intuitive. The tobacco industry is easy to pick on. ETS is an involuntary risk. And, for non-smokers, there's no personal sacrifice involved in forcing others to quit. The technical deficiencies, while numerous and significant, were no match for these intrinsic characteristics.
Now remember, meta-analysis depends on the assumption that the studies are similar enough to be combined. Yet mixing the different ETS studies is like mixing apples and oranges. You see, none of the ETS studies contain real exposure information. All the "exposure" data was derived from elderly women being prodded to remember their husbands' smoking habits of decades earlier (like the diesel exhaust studies). Or they came from the memories of other relatives.
None of this exposure data was ever validated or verified for accuracy. The clincher, however, is that each ETS study asked different types of study populations different questions about different time frames. To combine these studies together is truly the epidemiologic personification of the data processing acronym GIGO (garbage in, garbage out).
But, in the end, you've got to give credit where credit is due. EPA picked the right target — and hit the bull's eye. The rest is risk assessment history. Maybe this is really a lesson in picking a good target.
Click here for Chapter 7: Instant Risk
Click here to return to the home page
Copyright © 1996 Steven J. Milloy. All rights reserved. Site developed and hosted by WestLake Solutions, Inc.