Phelps, Richard P. (Ed.) (December, 2008). Correcting fallacies about educational and psychological testing. Reviewed by Caven S. Mcloughlin and Heather Doyle, Kent State University

Phelps, Richard P. (Ed.) (December, 2008). Correcting fallacies about educational and psychological testing. Washington, D.C.: American Psychological Association Pp. 287         ISBN 9781433803925 Reviewed by Caven S. Mcloughlin and Heather Doyle Kent State University June 19, 2009 "Standardized testing is used for diagnosis, selection, and measuring of achievement in many social fields, including psychology, education, employment, and professional credentialing. Its benefits are numerous and substantial but are not well articulated or well publicized. Measurement specialists are mainly positive about the worth of standardized testing. Meanwhile, sources that engage public debate tend to be less scientifically informed and more negative about the value of testing." Correcting Fallacies about Educational and Psychological Testing edited by Richard Phelps addresses many of the misconceptions associated with the various types of tests. While assessment tests—such as intelligence, measures or proficiency exams—are thought of as unbiased measures of skills or ability, many people both in the public and in the fields of education and psychology raise questions regarding tests’ usefulness and objectivity. In their technical communications, measurement specialists are generally enamored of standardized testing. Meanwhile, those who engage public debate tend to be less scientifically informed and generally more negative about the value of testing. The contributors to this volume contend that most criticisms ignore readily accessible scientific evidence and have the unfortunate effect of discrediting the entire testing enterprise. This edited text responds to the controversies surrounding the application of testing devices for the diagnosis, measurement and measuring of achievement and concludes that much of the public ‘wisdom’ about tests should better be described as myths. Richard P. Phelps, PhD grew up in St. Louis, Missouri. He received degrees from Washington University, St. Louis, Missouri; University of Indiana, Bloomington; and Harvard University, Cambridge, Massachusetts; as well as a PhD from the Wharton School at the University of Pennsylvania, Philadelphia. He taught secondary school mathematics in Burkina Faso, West Africa; worked at the Organization for Economic Co-operation and Development in Paris, France; the U.S. General Accounting Office, Washington, DC; WESTAT, Rockville, Maryland; and the Indiana Department of Education, Indianapolis. Dr. Phelps is editor and co-author of Defending Standardized Testing (pub: Lawrence Erlbaum, 2005); author of Standardized Testing Primer (pub: Peter Lang, 2007) and Kill the Messenger (pub: Transaction, 2003, 2005, 2008), and lead author for several statistical compendia. He is the book review editor for Educational Horizons, the quarterly journal of Pi Lambda Theta, the International Honor Society and Professional Association in Education. His Awards derive from Educational Testing Service, American Education Finance Association, and the National Center for Education Statistics. He is a Morganthau Fellow (Wharton), Kennedy Fellow, and Gustav Harris Scholarship recipient (Harvard). Dr. Phelps currently serves as Director of Policy Research at ACT, Inc. The Editor’s prior writings are as an unabashed defender of psychological testing. For example, Kill the Messenger (2003, 2005, 2008; with a subtitle of ‘The War on Standardized Testing’) "…reviews the debate over standardized testing requirements and argues in favor of testing.” That text was selected as a Profiler's Pick by YBP, Inc. and Academia Magazine from among their CORE 1,000 recommended books for academic libraries in 2003-4. Defending Standardized Testing (2005) had as its mission to “present a defense of standardized testing and a practical vision for its promise and future.” The Table of Contents illustrates the journey cast by the editor in formulating the text’s analysis and illustrates the eminence of the book’s contributors through their academic affiliations. Foreword (Thomas Oakland; Professor of Educational Psychology at the University of Florida) Introduction and Overview (Richard P. Phelps) Logical Fallacies Used to Dismiss the Evidence on Intelligence Testing (Linda S. Gottfredson; Professor of Education & Affiliated Faculty in the University Honors Program at the University of Delaware) Psychological Diagnostic Testing: Addressing Challenges in Clinical Applications of Testing (Janet F. Carlson & Kurt F. Geisinger; Research Professor of Educational Psychology at the University of Nebraska-Lincoln and Director, Buros Center for Testing and W. C. Meierhenry Distinguished University Professor of Educational Psychology at the University of Nebraska-Lincoln, respectively) Educational Achievement Testing: Critiques and Rebuttals (Richard P. Phelps) College Admission Testing: Myths and Realities in an Age of Admissions Hype (Wayne J. Camara; Vice President for Research and Analysis at the College Board) Criticisms of Employment Testing: A Commentary (Ernest H. O'Boyle & Michael A. McDaniel; doctoral candidate in organizational behavior at Virginia Commonwealth University and Professor of Human Resources and Organizational Behavior at Virginia Commonwealth University, respectively) Mission—Protect the Public: Licensure and Certification Testing in the 21st Century (Stephen G. Sireci & Ron K. Hambleton; Director of the Center for Educational Assessment and Professor in the Research and Evaluation Methods Program in the School of Education at the University of Massachusetts, Amherst and Distinguished University Professor and Chairperson of the Research and Evaluation Methods Program and Executive Director of the Center for Educational Assessment at the University of Massachusetts Amherst, respectively) Mistaken Impressions of Large-Scale Cognitive Diagnostic Testing (Jacqueline P. Leighton; Professor of Educational Psychology in the Centre for Research in Applied Measurement and Evaluation, University of Alberta) Summary & Discussion (Richard P. Phelps & Linda S. Gottfredson) The Foreword by Thomas Oakland illustrates how tests and evaluative procedures are used more now than ever before in the fields of education and psychology and opines that because of their widespread use, questions regarding the use of tests need to be addressed in order to validate their use and the consequences of their results. In the Introduction, Phelps addresses the troubled reputation that has plagued standardized tests in addition to providing a definition of the parameters of ‘testing.’ Phelps’ definition for the concept of standardized tests encompasses much more than the multiple-choice questions and bubble-sheets typically associated with the disdain much of the public feels toward this ‘benchmarking’ brand of individual assessment/evaluation. The Table of Contents, this initial conversation regarding standardized tests, and the introduction to the way in which the book is organized, provides the reader with a preview of what to expect in the remainder of the book. A brief summary of each chapter is provided by Phelps in the Introduction; additionally, the logic behind the organization of the book is supplied in the Introduction. Topics are generally covered in chronological order, with test-types developed earliest addressed in early chapters and more recently developed testing systems covered in subsequent chapters. The chapters of the book, each authored by experts in the field of measurement, are divided according to test type and use. Chapters follow a format wherein fallacies are typically addressed individually with headings and subheadings used to organize the chapter. In the first chapter (Logical Fallacies Used to Dismiss the Evidence on Intelligence Testing; Linda S. Gottfredson), the reader is introduced to the topic of intelligence testing and the misconceptions with which they are commonly associated. Gottfredson also addresses the impact of fallacies on the perception of intelligence testing. While the discussion of fallacies is primarily framed around their influence on intelligence testing, the topic provides a rationale for this book. The following description of fallacies is provided: They not only present falsehoods as truths but also reason falsely about the facts, thus making those persons they persuade largely insensible to correction. Effectively rebutting a fallacy’s false conclusion therefore requires exposing how its reasoning turns the truth on its head. The chapter does a thorough job of addressing the widespread myths regarding intelligence testing including those associated with test-design, test-score differences, test-validation, causation, and the politics of test use. Chapter Two (Psychological Diagnostic Testing: Addressing Challenges in Clinical Applications of Testing, Janet F. Carlson & Kurt F. Geisinger) provides some insight into the fallacies associated with diagnostic or clinical testing. One of the diagnostic controversies discussed in this chapter relates to the motives undergirding test selection. Carlson and Geisinger state that in the past decade test selection has become less a force of habit and influenced to a greater degree by specific referral questions. There is also debate regarding the how third party payers affect decisions regarding test choice. Educational achievement testing is addressed in the third chapter of the book by Editor Phelps (Educational Achievement Testing: Critiques and Rebuttals). Because of the numerous attacks on achievement testing, Phelps focuses solely on the fallacies associated with the dissemination of information on testing and directs the reader to his books Defending Standardized Testing and Kill the Messenger: The War on Standardized Testing for more details on other criticisms of achievement testing. A table is provided early in the chapter listing 22 of the most common fallacies and rebuttal sources. Phelps presents case studies featuring research and policy implications to address the four major fallacies discussed within the chapter. For example, in one of the case studies, Phelps addresses the claim that students in the United States are tested more than students in other countries by listing quotes and their sources providing support to each claim; he unequivocally states that empirical evidence for these claims is non-existent. Phelps also discusses how unfounded claims themselves are themselves originated, influenced, and even spawn policy debates. He also discusses research completed by the Organisation for Economic Co-operation and Development (OECD) on the number and duration of tests administered in the US and 13 other countries and finds that, in fact, US students face less time being subjected to testing than in all other countries reporting. In the conclusion of this chapter, the longest in the book, Phelps states that in education unpopular research is easily censored and suppressed. Because the current popular viewpoint is that too much time and energy is devoted to standardized testing, research and debate indicating a contrary view becomes nearly impossible to find. The censorship of research results in the rampant misunderstanding of the topic by public because findings related to both sides of the debate are not represented fairly. College admissions’ testing is becoming more of a significant factor as the number of high school students applying to postsecondary institutions continues to increase as college-completion increasingly becomes a rite-of-passage for those seeking adult economic independence. Accordingly, the fourth chapter of this book (College Admission Testing: Myths and Realities in an Age of Admissions Hype) takes on the topic of college entrance testing and associated myths, in addition to explaining the appropriate role of the tests. The reasons for the increase in the importance of admissions testing—such as the decline in high-wage manufacturing jobs and the increase in gap between wages of high school graduates and college graduates—are discussed in the first section of the chapter. The components of notable importance to colleges, according to the Admissions Trends Survey (see www.nacacnet.org/trends), are identified at the beginning of the chapter and include entrance exams such as the ACT and SAT. While the benefits of admissions testing are covered, the majority of the chapter is devoted to addressing the criticisms related to the ACT and SAT. Criticisms addressed in the chapter include misuse of results, bias, invalidity, SAT optional schools, and coaching. Separate sections address each criticism. In each section the criticism is stated; then, research is presented to refute the claim. The chapter conclusion discusses how the misinterpretation of scores and other admissions criteria lead educators, students, and parents to overemphasize their importance, which results in unreasonable courses of action to bolster scores. Admissions tests are identified as the easy target because they provide objective data that may be troubling. Concern is raised regarding the acceptance of negative opinions regarding entrance exams from individuals who are unqualified to evaluate the value of such exams. Chapter five is a commentary on the criticisms of employment testing (Criticisms of Employment Testing: A Commentary). While personality and intellectual ability tests initially come to mind, integrity tests, resumes, reference checks, and drug testing all fall under the realm of employment testing. Criticisms related to fairness for all racial and ethnic groups and the validity of employment tests across a variety of fields and contexts are first discussed. The foundation of each criticism is discussed and evidence suggesting a contrary perspective is presented. Questions regarding the accuracy of testing are also addressed. The authors refute claims that employment testing results in bad hiring decisions, assess the wrong content, and that tests do not provide any information that could not otherwise be gathered by talking to the applicant; they present research literature indicating a contrary position to each such assertion. Administrative concerns such as the cost of testing, training, and the time it takes to administer the testing are also addressed. The information in this chapter is presented in a way that would be helpful to any individual who has questions regarding employment testing including professionals in the Human Resources sector. The chapter on licensure and certification testing (Mission-Protect the Public: Licensure and Certification Testing in the 21st Century) provides an overview of topics pertinent to test development and use. Sections covering the standards for developing and evaluating licensure tests, the item formats and tests designs, testing across languages and cultures, and the means for calculating passing-scores are each determined are presented. This is done in an effort to address the fallacies of the ‘myths’ and provide evidence that criticisms associated with these topics are based on falsehoods. The description of the steps in the development of licensure tests, and how items are selected for inclusion, would be of interest to all involved in this process including test-constructors and test-takers. The importance and purpose of licensure and certification testing is emphasized throughout the chapter. Large-scale cognitive diagnostic testing in education is included in the penultimate chapter (Mistaken Impressions of Large-Scale Cognitive Diagnostic Testing) which provides a description of diagnostic testing as well as ways in which diagnostic testing is used in education. This background knowledge ensures that all readers have an adequate understanding of this testing format prior to learning about the associated fallacies. The chapter authors describe the fallacies as ‘mistaken impressions’ and list the eight that are addressed in the text in the initial pages of the chapter. Each of these impressions is addressed in a single section of text containing convincing evidence from references repudiating each impression. The final chapter of the book (Summary & Discussion) is devoted to summarizing and discussing the information presented in the text. An exhibit listing of all the ‘mistaken impressions’ addressed in the preceding chapters is provided and all of the myths are sorted into categories based on the attribute called into question (e.g., Validity, Efficiency, Fairness, etc.). Where there is some rationale for the existence of fallacies it is provided; and the role of education and science in the criticisms of testing is outlined. This chapter also articulates the difference between information provided for the purpose of advocacy and information derived from scientific studies and journals. The authors assert that many of the criticisms of testing originate as advocacy and, in turn, are themselves not supported by empirically-derived science. The power of repetition in multiple formats and from multiple dissemination-sources is explained as the primary reason why the oft-repeated criticisms of testing are eventually accepted widespread as ‘fact.’ The chapter ends with the following statement: “The easiest way to win a debate is by not inviting the opponent. The critics rightly fear an open, fair scientific contest.” Phelps’ book covers all of the major types of testing typically confronted by professionals in the fields of education and psychology. Addressing topics ranging from proficiency testing to certification/licensure testing, individuals of many professional stripes will have a vested interest in considering this discussion. An additional incentive derives from the greater contemporary emphasis on accountability. Thus, even in lay conversation the topic of testing is commonplace and most usually reflective of commentary that is critical. This book addresses virtually all the condemnations that typically are cited in a manner that would be understandable to a layperson while not compromising the quality of the content. Consequently, students and practitioners in the fields of the education and psychology are presented with an informed view of how testing is generally negatively perceived and shown where evidence exists to disprove the misperceptions. While it is commonplace for professionals in education and psychology to hear criticisms of testing, we often do not take the time to contemplate the origin of the criticism. Correcting Fallacies provides a persuasive, accessible and altogether unbiased view of testing, founded on empirical evidence rather than languishing in opinion. Correcting Fallacies covers a hot topic in a manner that is both thought-provoking and informative. Reference< Phelps, R.P. (Ed.) (2008). Correcting fallacies about educational and psychological testing. Washington, D.C.: American Psychological Association. Review commentary cited by the American Psychological Association, the book’s publisher, in publicity materials has included the following (to May 2009): "The mythology surrounding testing is varied and vast. Too often, opinion trumps scholarship and advocacy distorts science. This timely book is required reading for all who seek straight talk about testing." ~A. Jackson Stenner, PhD, CEO, MetaMetrics, Inc., Durham, NC "This is an important and readable book that addresses many of the erroneous beliefs regarding educational and psychological assessment. It will be of interest to teachers, administrators, and policymakers to help them better understand the complexities involved in educational testing and how tests can be used to improve student learning." ~Craig S. Wells, PhD, Assistant Professor, Research and Evaluation Methods Program, University of Massachusetts, Amherst "This refreshing book confronts current misperceptions in testing head on. Written by some of the most authoritative scholars in the field, each essay in Correcting Fallacies about Educational and Psychological Testing persuades with research data, logical reasoning, and lengthy reference lists, providing compelling evidence against the "untruths about testing" that have gained traction in the public discourse over the past few decades. I enthusiastically recommend it to policymakers and anyone else who has the power to spread the gospel of 'truth in testing.'" ~Mary Lyn Bourque, EdD, Mid Atlantic Psychometric Services, Leesburg, VA "Correcting Fallacies about Educational and Psychological Testing is a long overdue analysis of what is true and not true in the field of testing. Done correctly, for the right reasons, testing can be a most valuable tool. Richard P. Phelps and his team have greatly advanced the rigor of reporting real evidence in a field that should not be based on opinion." ~The Honorable David P. Driscoll, former Commissioner of Education, Commonwealth of Massachusetts "Standardized testing has been respected by academics yet criticized by the public for many years, and this book responds to the controversies surrounding test applications for the diagnosis, measurement and measuring of achievement. Editor Phelps has written and edited several books and scholarly journals, and he has enlisted the help of several measurement specialists to explore both sides of the public debate about standardized testing while offering suggestions for the improvement of testing practices. Written for policymakers, social scientists and scholars in education and psychology, this book also includes a glossary of testing terms." ~Annotation ©2009 Book News, Inc., Portland, OR, www.booknews.com) About the Reviewers Caven S. Mcloughlin, PhD, is Professor of School Psychology at Kent State University, Ohio. For over 20 years he has engaged in training preschool school psychologists supported by funding from the US Office of Special Education Programs. His publication interests have four themes: Exploring the importance of ‘wellness’ on children’s health and happiness; describing two of the fastest-developing ‘industries’ within education (home-schooling & charter schools); exploring the power of NCLB regulations in changing the shape and feel of public education; and providing parents and teachers with accessible information about the contemporary forces in social policy affecting public education Heather Doyle, MEd, NCSP, is a school psychology doctoral student at KSU completing her doctoral internship employed by Medina City Schools, OH. As a scientist-practitioner, Heather’s research interests include positive behavior supports, relational aggression, and behavior interventions. She is also interested in the implementation of Response-to-Intervention, childhood obesity, social emotional learning, group counseling, and consultation within the school setting.