Hunt, Morton M. How Science Takes Stock: The Story of Meta-Analysis. (1997). Reviewed by Glen Y. Wilson

Hunt, Morton M. How Science Takes Stock: The Story of Meta-Analysis. (1997). NY: Russell Sage Foundation Pp.xii+210 $29.95         ISBN 0-87154-389-3 Reviewed by Glen Y. Wilson Arizona State University         What does research tell us about the effects that class sizes have on academic achievement? Does increased educational spending lead to increased student achievement? Is increased homework the key to higher student achievement? These are questions to which there seem to be no clear or consistent answers. Research studies on these issues are seen to report positive effects, negative effects and no effects at all. The hodge-podge of conflicting research results confuses educational practitioners, policymakers, and lay persons and provides strong reason for people to reject scientific research and instead rely on "common sense." The ever-present recommendations for further research may increasingly lead to views of educational research as unproductive and wasteful.         The scientific community's answer to the chaos in research findings is a relatively new methodology—since the mid- 1970s—known as meta-analysis. Meta-Analysis has been characterized as the application of the same scientific rigor to research literature reviews as is employed in the conduct of empirical research studies. According to Glass, a pioneer of the method, "the findings of multiple studies should be regarded as a complex data set, no more comprehensible without statistical analysis than would hundreds of data points in one study" (Glass, 1981). In How Science Takes Stock: The Story of Meta-Analysis, Morton Hunt offers numerous examples from education, psychology and other social science fields to show how meta-analysis has been used to synthesize disparate research findings and identify patterns to arrive at conclusions unavailable to the researchers of the primary studies. Through his use of examples, Hunt illuminates the problem to be solved, the strategies the researchers followed, the application of the meta-analytic techniques, the results gained, and the public policy implications of the meta-analytic findings. Hunt's telling of the story of meta-analysis humanizes the method in a lively, interesting fashion while imparting enough information to allow the reader to become an informed consumer of meta-analytic studies. It should be noted that Hunt has not written a methods text, but something that might be more accurately described as "history of science." Indeed, Hunt has a well-established reputation as a historian and chronicler of science, having published such works as The Universe Within (1982), Profiles of Social Research (1985), The Compassionate Beast (1990), and The Story of Psychology (1994). It is no exaggeration to say that he is perhaps the most prolific and respected writer on social sciences for popular audiences, his work being prominent in such magazines as The Atlantic and the New Yorker, as well as in the trade book section of bookstores around the world.         This book was supported by a grant from, and published by, the Russell Sage Foundation. The intended audiences of the book are people with a general intellectual interest in science and those in the public policy arena: members of Congress, state legislatures, agency officers and staff. The Foundation hopes that through this book, policymakers and others in the policy sphere might gain a greater appreciation of meta-analysis as a means of effectively utilizing research in the solving of social problems.         Chapter One discusses the explosive growth in scientific research, the shortcomings of typical narrative literature reviews and review articles, the history of developments that led to the meta-analytic method, brief summaries of several noteworthy meta-analyses and a concise listing of the major benefits of meta-analysis. According to Hunt, some of the widely accepted benefits include: Physicians can now make decisions as to the use of therapies or diagnostic procedures on the basis of a single article that synthesizes the findings of tens, scores, or hundreds of clinical studies. Scientists in every field can similarly gain a coherent view of the central reality behind the multifarious and often discordant findings of research in their areas. Meta-analysis of a series of small clinical trials of a new therapy often yields a finding on the basis of which physicians can confidently begin using it without waiting long years for a massive trial to be conducted In every science, meta-analysis can generally synthesize differing results, but when it cannot, it can often identify the moderator and mediator variables that account for the irreconcilable differences. By doing so, the meta-analysis identifies the precise areas in which future research is needed, a function of considerable value to science. On a pragmatic level, meta-analysis of a wide range of social problems have profound implications for social policy; their findings...offer policymakers easily assimilated syntheses of bodies of research they have neither the time nor the training to evaluate on their own (pp. 18-19).         Chapter Two is primarily devoted to the story of how meta-analysis was developed. Interests in determining whether or not psychotherapy produced beneficial outcomes drove Gene Glass and Mary Lee Smith to develop and conduct the first meta-analytical study. Glass and Smith's findings revealed that "the combined effect of psychotherapy in their 375 studies, comprising about 40,000 treated and untreated subjects, had an effect size of .68—over two-thirds of a standard deviation" (p. 34). This finding meant that "while the median treated client (at the middle of the curve) was as mentally ill before therapy as the median control individual...after therapy, the treated client was healthier than three-quarters of the untreated group. In the social sciences, so large an effect of any intervention...is almost unheard of" (p. 34). Prior to this meta-analysis, there was a 25-year argument between the supporters and critics of psychotherapy as to whether it worked or not. Although some diehard critics remained, the Smith and Glass study as well as subsequent studies have resolved this question to the satisfaction of most experts. It was in connection with the integration of the psychotherapy outcome literature that Glass devised many of the techniques that remain a part of the meta-analysis armamentarium. Indeed, the term "meta-analysis" was first used in 1976 in Glass's presidential address to the American Educational Research Association (Glass, 1976).         Chapter Three is of particular interest to those in education. It centers on the use of meta-analysis to resolve conflicting issues in education. The major example used in this chapter was the controversy over whether school spending has positive effects on student achievement. During the 1980s, Eric A. Hanushek, a professor of economics and political science at the University of Rochester, wrote several articles claiming that empirical research shows that increased school spending is not related to increased student achievement. This claim was embraced by the Reagan-era U.S. Department of Education and other conservatives. Distress and anger at the effects Hanushek's findings were having on schools prompted Richard Laine, Rob Greenwald, Bill McKersie and Rochelle Gutierrez, graduate students at the University of Chicago, to conduct a limited meta-analysis of the research Hanushek used as the basis for his findings. The results from the preliminary meta-analysis indicated that Hanushek was wrong and that his data supported a conclusion opposite to the one he arrived at. This result led Professor Larry Hedges, also of the University of Chicago, Laine, and Greenwald to take on a more ambitious project: the full- scale meta-analysis of all seven cost variables investigated by Hanushek for their relationship to student achievement. The final outcome of this complete meta-analysis was published in the Educational Researcher under the title "Does Money Matter? A Meta-Analysis of Studies of the Effects of Differential School Inputs on Student Outcomes." As before, the findings strongly contradicted Hanushek's conclusions. Hedges, Laine, and Greenwald found that an increase in per pupil expenditure of five hundred dollars was associated with a nearly 24 percent (.7 standard deviation units) increase in student achievement compared with similar students in a school that did not receive an increase in per pupil expenditure. Again, meta-analysis was able to provide a more definitive answer to resolve a long-running conflict. The details of this story contextualize the use of meta-analysis in an engaging and interesting manner, while providing insights into the use of this method. Hunt also provides brief accounts of meta-analysis employed to study questions about homework and about the relationship between gender and science achievement. The chapter ends with a discussion of the use of meta-analysis to help generate causal explanations.         Chapter Four offers several cases of meta-analysis applied to questions in medicine and health-care. The major case is that of using streptokinase, a clot-busting enzyme, to treat acute myocardial infarction (AMI) due to blood clots. Cumulative meta-analyses of AMI therapies led to the discovery that a meta-analysis of a relatively small number of studies could show results similar to the results of a massive study. This suggests the possibility that physicians could begin to use new therapies perhaps years before the results of massive clinical trials are completed.         Chapter Five reports several cases of meta-analysis applied to the social sciences. Hunt discusses issues related to the validity of meta-analyses. Of particular interest was the technique used to address the so-called "file drawer" problem. The file drawer problem "refers to the possibility that there may exist, buried in file drawers, a number of unpublished studies that could nullify the findings of the meta-analysis" (p.118). This is a real problem due to the publication bias that journal editors have for studies that report statistically significant results (cited in Light, Singer & Willett, 1990). The file drawer test determines how many studies with nonsignificant findings would be necessary to negate the meta-analytic findings. In the example cited in Chapter Five of Ambady and Rosenthal's meta-analysis of the ability to make judgments of relative strangers from "thin slices" of nonverbal behavior, 7,111 unretrieved studies with non- significant findings would be necessary to make the argument that the statistically significant studies in hand were only Type I errors (p. 118).         Chapter Six illustrates the use of meta-analysis to guide social policy. A detailed example of a meta-analytic evaluation for Congress of the special supplemental food program for women, infants, and children (WIC) is provided. This chapter gives a rich accounting of the political concerns and realities of producing research for public policy purposes. Brief meta-analytic cases recounted in the chapter include the question of lumpectomy vs. mastectomy treatments for breast cancer and whether secondhand tobacco smoke is indeed, carcinogenic.         Chapter Seven is a discussion of the future of meta-analysis. Considering the past growth and acceptance of meta-analysis as a method of inquiry, Hunt suggests that this will likely continue over the next several years. It is suggested that the future of meta-analysis is the extension of the method beyond mere aggregated description to more variegated explanations of what works well and where.         How Science Takes Stock also contains a thirteen-page appendix by meta-analyst Harris Cooper that offers a more detailed discussion of three of the major statistical issues concerning the method: obtaining effect sizes in a combinable form, combining effect sizes across studies, and interpretation of effect sizes.         Hunt takes a potentially boring topic like statistical methods and presents it in such a way that it comes alive with human drama and the excitement of scientific discovery. Although centered around the experiences of key researchers, How Science Takes Stock still manages to convey enough information about the mechanics of the technique to allow the reader to develop an appreciation of the meta-analytic method and its place in resolving scientific disputes and social policy controversies. References Glass, G. V (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8. Glass, G. V, McGaw, B., & Smith, M. L. (1981). Meta-Analysis in Social Research. Beverly Hills, CA: SAGE. Hedges, L. V., Laine, R. D., & Greenwald, R. (1994). Does Money Matter? A Meta-Analysis of Studies of the Effects of Differential School Inputs on Student Outcomes. Educational Researcher, 23(3): 5-14. Hunt, M. M. (1982). The Universe Within: A New Science Explores the Human Mind. New York: Simon and Schuster. Hunt, M. M. (1985). Profiles of Social Research: The Scientific Study of Human Interactions. New York: Russell Sage Foundation. Hunt, M. M. (1990). The Compassionate Beast: What Science Is Discovering About the Humane Side of Humankind. New York: Morrow. Hunt, M. M. (1994). The Story of Psychology. New York: Doubleday. Light, R. J., Singer, J. D., & Willett, J. B. (1990). By Design: Planning Research on Higher Education. Cambridge, MA: Harvard University Press. About the Reviewer Glen Y. Wilson Glen Y. Wilson is a Ph.D. student in the Division of Educational Leadership and Policy Studies in the College of Education at Arizona State University. He is interested in the school choice movement as a school reform strategy and in issues of fairness and equity related to high stakes testing.