Russell, Michael. (2006). Technology and Assessment: A Tale of Two Interpretations. Reviewed by Christine Greenhow, Harvard University

Russell, Michael. (2006). Technology and Assessment: A Tale of Two Interpretations. Charlotte, NC: Information Age Publishing, Inc. Pp. vii + 243 $69.95 (Hardcover)   1-59311-039-1 Reviewed by Christine Greenhow Harvard University October 7, 2006 In Technology and Assessment: A Tale of Two Interpretations, Michael Russell raises critical questions that have long concerned those of us in the field of educational technology. How do instructional uses of technology relate to or influence students’ learning? How do we measure technology’s influence in ways that accurately capture the complexities as well as speak to a broad audience of policy makers, administrators, and teachers? And, what should we do with the results of such assessments as researchers and practitioners? However, Russell’s book is also for test-makers, measurement experts, and psychometricians concerned with how to apply ubiquitous, networked technologies toward improving the design, administration and implementation of large scale testing and assessment in U.S. education. Interest in quality testing practices has become widespread in the wake of No Child Left Behind legislation that promotes reliance on tests as a means of ensuring educational accountability. Two Co-occurring and Uncoordinated Educational Reform Movements In Technology and Assessment, Russell presents two interpretations of technology and assessment and persuasively lays out the gaps between them. He argues that one interpretation represents the field of student assessment and the other, the field of educational technology. He argues in the opening chapter that each field espouses technology and assessment as “reform tools” but does so in an uncoordinated and often contradictory manner: Separately, instructional uses of computer-based technology and assessment represent two of the most pervasive and persistent educational reform movements introduced over the past two decades. Today, there is hardly a public K-12 school that is not equipped with computers and nearly all schools…are connected to the Internet. Similarly, every state in the nation has a formal student assessment program…Despite the popularity of technology and assessment, these two reform tools for improving teaching and learning have been developed and implemented in an uncoordinated manner (p. 2). Russell builds the case that the two interpretations of technology and assessment held by testing professionals and educational technology researchers work against each other in stimulating educational improvement. For example, in many schools today students learn to become better writers by using word processing software on a school-based or home computer but many are required to perform writing tests on paper. The result is that the test scores for students who have become accustomed to writing with computers “severely underestimated” the students’ ability when they were forced to write with paper and pencil (Russell & Haney, 2000). In this example the testing conditions did not reflect the actual conditions under which tech-savvy students do their writing and, as a result, some students were not able to demonstrate their competency on the test as it was administered. This is just one of several examples Russell provides, many from his own research, of the lack of coordination between the assessment and instructional technology movements. Recently, I encountered evidence of the gap Russell describes in reading an article in the education section of my local newspaper. The article described how some teachers were deliberately cutting back on all curricula they deemed non-essential for assessment purposes; that is, content they did not expect their students to be tested on in the state’s high stakes exams. Fearing that under-prepared students would fail these tests and that failure rates would have consequences for the school system, teachers were reportedly dropping whole subjects such as art, music, and social studies as well as methods such as the use of multimedia projects because these content area or practices were not represented on the exams. Similarly, Russell and Abrams (2004) found that a desire to improve students’ test scores prevented teachers from using computers to teach writing skills even though word processors have been shown to have a positive impact on students’ attitudes and writing competency. These examples of contradictory practices effectively hook the reader’s interest in wanting to know more about how they emerged and what we ought to do about them. Introduction: The Rise of Technology and Assessment in K-12 Education Russell’s book constructs a tale of how two “uncoordinated” reform movements arose: the themes or philosophies that underlie them, their central practices, major developments and the factors that prevent them from converging. Through clear writing and sequencing of chapters punctuated by helpful examples, the reader is drawn into the tensions between these two fields to anticipate the final chapter, where Russell offers “strategies” to help “bring these two communities together in order to more fully capitalize on the power of computer-based technologies” and “more fully assess these impacts” (p. 9). Most importantly, Russell suggests that the contradictory practices emerging in classrooms today are the result of a “gap in intentions” and values between the fields of testing and instructional technology. On the one hand, educational technology researchers and reformers view technology as a tool to facilitate problem-solving, creative thinking, collaboration, and higher-order thinking. By comparison, the authors of NCLB, he argues, view technology as a tool to assist students in mastering “a predefined body of knowledge and skills” (p.5). Not surprisingly given NCLB legislation, funding for large scale test developers generates tests that provide “reliable measures of skills and knowledge that comprise specific domains.” Test-developers and measurement experts, he argues, primarily view technology in terms of how it can promote greater efficiencies in test-making, delivery, and scoring (p. 7). Citing Becker and Lovitts (2000) and Means, Penuel and Quellmalz (2000), Russell asserts that tests designed to be administered on a large scale are not “sensitive to the full range of learning that occurs as students work with computers in schools” and yet, “scores on these tests are often the measure used to examine the effect of computer use on student learning” (p. 6). This mismatch does not serve either field well. Those concerned with educational technology are frequently called upon to prove that teaching with technology improves students’ learning but lack the resources to design and implement assessments that appropriately align with and measure learning objectives on a large scale. Meanwhile, those concerned with testing and assessment seek to accurately and efficiently measure students’ content area competencies and use results to inform the improvement of teaching and learning in schools. Although able to administer these tests to a broad student population, testing conditions that do not represent actual classroom conditions threaten the accuracy of test results and often teachers are not able to use results to inform instructional decision-making at the classroom or student level. In Chapter 1, the author explains that a primary goal of Technology and Assessment is to help readers who may have a background in one area or the other – educational technology research or student assessment and testing – to walk in each other’s shoes and ultimately, align their focus. However, the book is laid out in such a way that readers having little or no familiarity with either field will be able to follow Russell’s argument and grasp his historical overviews as well as his central points about the challenges facing each field. Building on the introduction in Chapter 1, the next six chapters (Chapters 2 through 7) focus on student assessment and then, four chapters (Chapters 8 through 11) focus on educational technology. In the following sections, I provide a critical overview of these two interrelated and intersecting accounts. Part 1: The History of Testing, the Technology of Testing, and Computer-based Technology as Disruptive Applications Russell begins his discussion of student assessment by describing it as a “strategy for educational reform.” Chapters 2, 3 and 4 orient the reader to the historical context from which present testing issues stem and then, chapters 5, 6, and 7 discuss the relationship between testing and modern technological advances. For instance, chapter 2 describes how today’s present preoccupation with testing and accountability largely grew out of the “standards movement” of the 1980’s where “two avenues” arose “for the role of assessment and educational reform” (p. 11). One avenue advocated assessments, especially assessment alternatives to multiple-choice testing, that could capture higher order thinking and help teachers change their instructional practices. The second avenue advocated standardized testing to hold schools accountable for teaching content specified in state content standards. Reformers visions for these assessments ranged from norm-referenced to criterion-referenced exams. The 1980’s and 1990’s were characterized by such well known movements as the Coalition for Essential Schools which advocated students’ public “Exhibitions” to demonstrate mastery of “essential” skills and portfolio assessments used mostly at the classroom level but widely adopted as alternative forms of assessment by states such as Vermont in the 1990’s. Interestingly, Russell outlines four reasons why such non-traditional forms of assessment have failed to take hold at the state level even though they may more accurately represent the range of learning that occurs in classrooms. A primary reason, of course, is the failure to convince “the educational community and political leaders of the additional value added by assessments that provide diagnostic versus summative information about student performance” [emphasis added] (p. 18). He also suggests that educational leaders failed to see the additional value to assessment practices added by technology in schools beyond technology’s use as a tool to increase the efficiency of test delivery (e.g., putting paper based tests on computers). Observers, he argues, questioned whether such practices encouraged a severe underutilization of technology’s potential or create inequitable testing conditions between tech-savvy students and those who are not used to using computers. The issue of how to appeal both to educational policy makers and leaders while at the same time utilizing advances in technology and in testing to their full potential in educational reforms is a key issue today. Russell turns to history to illuminate it. Chapters 3 and 4, for instance, explain how neither “the influence of computer-based technologies on testing that we are witnessing today nor the accompanying concerns are novel…since the rapid expansion of standardized testing following World War I and the subsequent invention of the electronic scanning machine…computer-based technologies have impacted testing…[and] neither the political nor social influences” accompanying the rise in testing or the desire to apply technology to increase its “efficiency” are new. In chapter 4 especially, Russell describes the “technology of testing” – most notably electronic scanners -- and discusses how computer-based technologies influenced testing practices as well as the “promises made” by innovators and the accompanying concerns of those who sought to believe them. What emerges is a portrait of the ways in which these new machines propelled the testing industry into a certain values orientation, namely “decreased instructional value of testing for teachers” and “increased reliance on multiple-choice items” while decreasing the use of “open-ended essay items.” Chapter 5 describes the relationship between instructional uses of technology and testing practices as a “push-me pull-you” phenomena. As alluded to earlier, the chapter outlines areas of “misalignment” between the technological tools students use to learn and the tools they are allowed to use to test. The result is that classroom practices shift to mimic testing practices (e.g., open-ended, project-based inquiry using computers are replaced with using computers for test preparation) or testing programs change to incorporate learning tools. Both shifts can shortchange students. However, in chapters 6 and 7 Russell provides evidence of the “considerable progress” that has been made in applying computer-based technologies to testing (p. 96). Sophisticated automated scoring tools, tests that adapt to the user’s ability and reduce testing time, Web-based assessments that provide immediate results to test-takers and their constituencies are just a few examples. In particular, Russell argues that computer applications which act as “disruptive technologies”—those that “alter the current [testing] process and procedures and eventually make extinct those structures that do not adapt”—hold the most promise toward fulfilling the test-developer’s vision of using results to accurately inform and enhance the practices of teachers and their students on a large scale. Although at the end of chapter 7, Russell describes several challenges to enhancing assessment through disruptive applications (e.g., technical, political and practical challenges) the vision he presents is seductive: Computer-based technologies provide important opportunities to dramatically alter the technology of testing…the ability of computers to present complex, multi-step problems that may incorporate several types of media, can have different paths to reach a solution, can have multiple solutions, and can record the examinee’s every action, creates opportunities to learn about students knowledge, conceptual understanding, and cognitive development in ways that today’s technology of testing cannot (p. 101). Such applications of powerful, networked technologies would seem to do much to bridge the goals and values of the assessment and educational technology fields. Russell draws on examples of such applications currently being used in architecture (Tatsuoka, 1991), military medicine (Satava, 2001) and science (Vendlinski & Stevens, 2002). The Interactive Multi-media Exercises (IMMEX), in particular, are currently being used in K-12 education as both a diagnostic assessment tool for teachers and a way of documenting gains or losses in students’ learning over time. IMMEX is able to record and create both individual and group clusters of performances. As students work their way through the problem, the software is able to develop a profile or classification of the students’ performance as falling into one of several families of strategies (i.e. limited, efficient, prolific). Therefore, after students complete an exercise, information about where they fall on the continuum is made available to teachers who can adjust their instruction to provide targeted feedback and additional scaffolding. IMMEX cases have also been developed for language arts, social science, art (see http://www.immex.ucla.edu/ for more examples), and in my own experience, related applications of these multimedia exercises have been developed for use in standards-based teacher education (Dexter, Greenhow & Riedel, 2002; Greenhow, Dexter & Hughes, 2004). Part 2: The History of Technology in Education, Research, and Approaches to Assessing Technology In the second half of his book, Russell turns to focus on the interpretation of technology and assessment operating within the field of educational technology. Chapters 8 through 11 outline its central themes, current practices, and major developments in reform-oriented efforts. To orient the reader, Chapter 8 provides evidence of the spread of computers and the Internet in today’s schools and maps out the multiple roles that technology has played as an educational reform tool. For instance, reformers have promoted computers at the curriculum level to teach students computer-related job skills and as tools that teachers use to help students achieve standards-based competencies. Computers have also been touted as tools to transform pedagogy and to support communication and productivity outside of the classroom. Russell argues that the most recent “technology-related reform efforts” represent a “merging” of all of these earlier visions. Today’s reformers advocate for the use of technology in schools as tools essential to learning “new skills and knowledge believed to be important for the modern workplace” and demands of living in a knowledge society (p. 132): By establishing formal standards that are intended to become a component of instruction, technology is placed directly into the curriculum. But, by including standards that are best met through extended collaborative projects, the establishment of these standards also encourages teachers to employ instructional practices that are more student-centered…in this way, more recent technology standards attempt to fit into existing curriculums while promoting reform of traditional teacher-centered practices (p. 133). While we have seen an incredible investment in technology infrastructure in schools during the last decade, the current concern is with return on investment: “concerns about the impact of technology has shifted the focus of research from understanding how technology can change teaching and how to support these changes to a narrower focus on how technology is impacting student learning as measured by standardized tests in schools” (p. 135). As educational technology leaders appeal to calls for greater accountability from policy makers and the public by themselves advocating state standards for students’ and teachers’ technology use, they do much to build support for their own reform efforts. However, without plausible evidence that students and teachers are achieving standards-based competencies in technology-enhanced schools (or through the use of certain technologies in particular) such support and funding will most certainly erode. As mentioned above, the chicken and egg problem that has tended to slow down progress toward persuasively documenting gains since the introduction of “educational technologies” to public education in the 1800’s is of great concern to the field today. To provide perspective, Russell, drawing on Cuban’s accounts (1986; 2001), briefly chronicles the rise of technology in education from books and blackboards in the 19th century to film, typewriters, radio, television and computer-based applications in the 20th century to the ubiquity of the Internet today. He concludes, “there is no question that computers are the most recent technology that has penetrated the American educational system…an important question that remains largely unanswered is how computer use in schools is affecting teaching and learning.” To begin to address this, Russell provides an overview in chapter 10 of the kinds of questions educational technology researchers have focused their research agendas around, the kinds of methods and evaluation strategies they have used, and what some of the themes in the results have been. As researchers were able to gather figures about how many teachers or students were using a particular technology, how often, and to what effect on student learning (broadly speaking), they were then able to hone in the details of the learning process, such as who is using the technology, how, why, and what impact is it having on learners’ and teachers’ experiences. What emerged was a more complete picture of the complexities of integrating technology into schools, especially the conditions that must be established before meaningful learning can occur (e.g., access; technical support; time for teachers and students to learn to operate the tools and then, to work with them; a supportive professional community, etc.). Although Russell’s is a very quick overview (about 25 pages) of a legacy of research encompassing tens of thousands of articles and over eighty years, his “meta-analysis” of a select group of articles documenting a decade of original research that focused on the use of computers within specific subject areas is interesting and helpful in further illuminating the types of questions researchers have sought to understand and the methods they have used. This brief overview sets the stage for chapter 11 “Promising Approaches to Assessing Technology” in which he carefully navigates the argument that has recently divided the educational research community, namely, that the gold standard of scientifically based research should incorporate randomized experiments. Here, Russell does a good job of providing an explanation of the context within which randomized experimental methods might effectively be used. However, he also lays out “stages of development” for research that assesses educational technology – both effects of technology and effects with technology (see p. 167-172) – that warrant multiple methods to investigate various research questions as technologies are introduced, developed, learned, integrated, and refined in design, practices, and conditions within educational settings over time. This chapter helps the reader to understand why defining “outcomes” of educational technology is a complex task for the researcher and test-developer, alike. Russell concludes by citing previous multi-level analyses that suggest it can be done. He closes the chapter with a call to honor and preserve the complexity in both the research and assessment process: While it is important to understand how technology use affects student learning, it is equally important to employ measures of learning that are sensitive to the types of learning that occur when students use a given technology. It is also vital that the modes of assessment do not mask learning effects by requiring student to demonstrate their knowledge using tools that are unfamiliar to them [emphasis added] (p. 202). Conclusion: Strategies to Close the Gaps Between These Interpretations of Technology and Assessment In the final chapter “Bridging the Gaps,” Russell cautions against what he sees as a potentially widening gap of intentions and actions between the two interpretations of technology and assessment he has outlined. By developing the historical context for his argument, Russell has demonstrated that these uncoordinated and potentially conflicting reforms are firmly rooted in the soil of American education. To bring the two interpretations together and align their reformer’s aims, will likely take nothing short of educational technology researchers and leaders working with test-developers and measurement experts on federally funded projects to develop a coordinated line of research and technology-enhanced assessment instruments that are sensitive to measuring gains that both constituencies and policy makers care about. Russell argues that this might best be done through a two-tiered research program that would include a single national coordinating center and regional research centers to study both “well-established technology interventions” as well as smaller-scale proof of concept studies or issues of interest to educational agencies within the region (p. 216). I encourage the reader to review this final chapter with an open mind as well as a critical and engaged eye. Russell’s Technology and Assessment: A Tale of Two Interpretations is perhaps most importantly an incitement to join the discourse about how we ought to bring these two fields together and halt the reform trajectories that are working to negate each other at the expense of our students, our schools, and our educational future. References Becker, H.J. & Lovitts, B. (2000). A project-based assessment model for judging the effects of technology use in comparison group studies. Paper presented at SRI Design Meeting on Design Issues in Evaluating Educational Technologies. Retrieved December 4, 2002, from http://www.sri.com/policy/designkt/becker2.pdf Dexter, S., Greenhow, C. & Riedel, E. (2002, July).Instructional Tools for Teaching Technology Integration Decision-making: IMMEX-powered Online Cases. Paper presented at the Preparing Tomorrow’s Teachers to use Technology (PT3) Grantees Annual Meeting, Washington, DC. Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. New York: Teachers College Press. Cuban, L. (2001). Oversold & underused: Computers in the classroom. Cambridge, MA: Harvard University Press. Greenhow, C., Dexter, S., & Hughes, J. (2003, April). Teacher Knowledge about Technology Integration: Comparing the Decision-making Processes of Preservice and In-service Teachers about Technology Integration using Internet-based Simulations. Paper presented at the American Educational Research Association Annual Meeting, Chicago. Means, B., Penuel, B., & Quellmalz, E. (2000). Developing assessments for tomorrow’s classrooms. In W.F. Heinecke & L. Blasi (Eds.), Methods of evaluating educational technology (pp.149-160). Greenwich, CT: Information Age Publishing Russell, M. & Abrams, L. (2004). Instructional uses of computers for writing: How some teachers alter instructional practices in response to state testing. Teachers College Record, 106(6) 1332-1357. Russell, M. & Haney, W. (2000). Bridging the gap between testing and technology in schools. Education Policy Analysis Archives, 8(19). Retrieved October 7,2006, from http://epaa.asu.edu/epaa/v8n19.html Satava, R. (2001, July). Metrics for objective assessment of surgical skills workshop: Developing quantitative measurements through surgical simulation. Draft report for meeting in Scottsdale, AZ. Tatsuoka, K.K. (1991). Boolean algebra to determination of the universal set of knowledge states (ETS Technical Report ONR-91-1). Princeton, NJ: Educational Testing Service. Vendlinski, T. & Stevens, R. (2002). Assessing student problem-solving skills with complex computer-based tasks. Journal of Technology, Learning and Assessment. Retrieved July 10, 2002, from http://www.bc.edu/research/intasc/jtla/journal/v1n3.shtml About the Reviewer Christine Greenhow Senior Research Fellow University of Minnesota, Twin Cities Christine Greenhow is a Senior Research Fellow at the University of Minnesota and advanced doctoral student at Harvard University, Graduate School of Education. Her research, teaching and consulting work focus on technology integration in K-12 and higher education settings, teacher education and professional development, research and assessment methods, and educational uses of emerging online technologies. She is a former public school teacher, coordinator for a Preparing Tomorrow’s Teachers to Use Technology (PT3) grant, and co-founder of a Minnesota-based nonprofit that makes college admission possible for low-income urban students. Copyright is retained by the first or sole author, who grants right of first publication to the Education Review.