Goodman, Greg S. & Carey, Karen T. (2004). Ubiquitous assessment: Evaluation techniques for the new millennium. Reviewed by Susan K. Green, Winthrop University

Goodman, Greg S. & Carey, Karen T. (2004). Ubiquitous assessment: Evaluation techniques for the new millennium. New York: Peter Lang. Pp. xii + 254 ISBN 0-8204-7069-4 Reviewed by Susan K. Green Winthrop University August 29, 2005 The purpose of this book is to introduce or enhance knowledge of individual assessment for educators and psychologists (including counselors, teachers, administrators, school psychologists and other clinicians) using a “new paradigm” that envisions evaluation of the student or client as “enmeshed within the learning or therapeutic process.” The authors emphasize that assessment should be seamlessly woven into the other activities of clinical and school professionals. Chapter 1, Critically Situating Assessment, describes the history of assessment, with a focus on categorization of individuals, from the Greeks through the twentieth century and No Child Left Behind. The authors explain that rather than relying on the formal, standardized tests that have evolved in the twentieth century from this tradition of sorting and labeling, ubiquitous assessment emphasizes nonstandardized and informal techniques. In particular, the authors reject reliance on published norm-referenced tests. They believe that discrepancies found between the typical white, middle-class groups often used for norming these tests and other more marginalized groups are used to reinforce mainstream dominance and “superiority” and contribute to institutional racism. The authors place ubiquitous assessment within the intellectual tradition of postmodernism, which “challenges hierarchical structures of knowledge and power,” eschews rigid conceptualizations of what is normal, and pursues social justice. The second chapter, entitled, Critically Situating Reliability and Validity, provides standard (not critical) information about traditional types of reliability (e.g., test-retest, internal consistency) and validity (e.g., criterion-related, construct) that are primarily statistically determined. The authors then go on to offer additional perspectives on validity and reliability from researchers using qualitative methods, who emphasize triangulation (looking for similar patterns from different sources), credibility (accuracy), and dependability (an analog to reliability). Other types of reliability and validity are then discussed (decision validity, social validity, ethnic validity). Providing illustrative examples from typical educational or clinical assessments could have been useful for readers to be able to apply both the traditional psychometric and the qualitative concepts to familiar evaluation contexts. The authors could have been more selective about the amount and complexity of information in this chapter. This material is daunting, especially for novices, and it is not placed in any specific context. Most of the descriptions of types of validity and reliability are limited to one or two paragraphs. The authors then go on to discuss reliability and validity in terms of observations, interviews, and treatments, emphasizing the need to corroborate evidence with other sources. As the authors summarize, “Assessments will lead to the development of valid treatment and intervention plans if the clinician is certain to have fully explored the problems from all perspectives.(p. 43).” The following five chapters address the use of ubiquitous assessment in the specific contexts of school counseling, classrooms, marriage and family counseling, school psychology and mental health settings. In the school counseling chapter, the authors emphasize the counseling relationship as a partnership. They suggest that administering formal tests can interfere with forming meaningful connections with clients and therefore recommend interviewing, observing and other nonstandardized methods of data gathering. They briefly summarize several perspectives on counseling and show how their ideas about ubiquitous assessment are compatible with them. Basically, they point out that in each of these perspectives, the therapeutic process is consistent with ubiquitous assessment because the therapist continually integrates new and unique information shared by the client. The next sections of the chapter describe the counselor’s daily practice and types of assessments and interventions in which counselors engage. These sections seem uneven in depth and content, perhaps because the intended audience appears to vary. For example, there is a one paragraph description of the structured interview with no discussion of interviewing technique (e.g., developing rapport), suggesting an audience quite familiar with the interviewing process. Yet there are also several very basic pieces of advice such as how to introduce yourself to an at-risk child or the admonishment not to put up roadblocks to parent calls for help. The section on suicide intervention is thorough and includes specific questions to ask. But then the authors stress the importance of stating the limits of confidentiality to the client without explaining what those are. A different issue arises about the philosophy of ubiquitous assessment in the sections on assessing depression and attention deficit hyperactivity disorder. These sections feature the lengthy diagnostic criteria from the Diagnostic Manual for the American Psychiatric Association. To me, the concept of ubiquitous assessment implies that individual children be evaluated for interventions unique to their needs in their own context based on pressing concerns that emerge in their particular case. Relying on DSM characteristics as an important element of assessment seems to imply shoehorning students to fit definitions imposed by “utilitarian conceptualizations that, in effect, dehumanize individuals (p. 17),” a process that would contradict the authors’ postmodern critique in the first chapter. Early on, the discussion of ubiquitous assessment needs to be expanded to include a focus on its ultimate purpose, which would logically seem to address solving problems rather than labeling children. In Chapter 4, Assessment in Classrooms, the authors laudably point out the importance of observing referred children in the classroom and assessing their learning environment. However, they spend the first 8 1/2 pages of the chapter discussing traditionally used measures such as standardized group achievement tests, pointing out problems with these measures, such as the lack of congruence between what is taught and what is tested. They also describe teacher-made tests with a brief section on how they can be improved. In this section, I was surprised that they did not mention Black and Wiliam’s (1998a, 1998b) important work on formative assessment, which has deeply influenced thinking on classroom assessment in recent years. Black and Wiliam’s conception of formative assessment is a particularly good example of developing “seamless” weaving of assessment and intervention (in this case, instruction). The next section describes several individually administered achievement tests (e.g., California Achievement Test, Iowa Tests of Basic Skills), which the authors suggest are given by mental health professionals “because of the inherent problems with standardized group tests and teacher-made tests (p. 86).” But these individually administered tests, too, are subject to the same critiques they make about group tests (e.g., lack of representative norms, lack of match between curriculum and test, lack of information for modifying instruction). The authors do point out that many of these tests do not have appropriate levels of technical adequacy. The second part of the chapter addresses more useful methods for gathering information about students consistent with “ubiquitous” assessment. The authors begin this section by describing the importance of consultation with teachers and parents and by providing questions that could be used in a consultation or problem-solving interview. They stress such an interview as a first step, to be followed by classroom observations of the child across several days and classes. The detailed descriptions and examples of the different kinds of observations (e.g., real-time observations, antecedent-behavior-consequence analysis, time sampling techniques) are clear and thorough. I would have liked to see more information and an example of a data sheet for observing more than one child, as peer comparisons were extremely useful in my own assessments as a school psychologist. As the authors point out, establishing norms accepted by different teachers can be very informative in understanding the context in which a referred child functions. Similarly, the section on curriculum-based assessment could have been expanded because of this assessment tool’s capacity to provide information to monitor academic progress and provide information to modify instruction. The chapter ends by emphasizing the necessity of understanding referred children in the context of their natural classroom environments, although describing how to put the information gathered from these different contextual sources together to develop an effective intervention is not addressed at this point. The chapter on assessment in marriage and family counseling points out that this type of assessment is more complex than assessment for individual counseling because understanding the dynamics of the interpersonal relationships of all family members is required. The purpose of the chapter is to introduce several conceptualizations of family dynamics that will suggest appropriate therapy rather than to describe specific assessment techniques. The authors combine the approaches of Bowen and Minuchin. Bowen’s systems theory aims to help clients’ understand their family dynamics. In contrast, Minuchin’s structural approach challenges the presenting symptoms and the family structure. The first element of family dynamics introduced is cultural norms. The authors point out that the clinician must ask questions to examine the impact of culture within each family. They provide useful examples from Hmong families who came to the United States after the Vietnam War. The next element discussed is functional components within family interaction. The authors recommend that the clinician identify which behaviors work for each family and then which behaviors are dysfunctional in their communicative intent. Birth order and substance abuse are then discussed as factors that also influence family interaction. The genogram or “representational map of the family” is then discussed as a tool for assessment and as a method for helping family members understand their situation. The map is to depict “the flow and changes within the family,” but there were no clear directions for how to construct such a map. Instead the authors explained that “the essential element of the genogram involves the telling of stories (p. 117),” and they provide an example of one such story. The chapter ends by pointing out that ideally there is no distinction between assessment and therapy, but again no methods of integrating the different types of assessment described in the chapter within the ongoing counseling process are addressed. Chapter 6 addresses assessment in school psychology. Goodman and Carey first point out that school psychologists have a broad range of training that equips them to do ubiquitous assessment and not just testing for eligibility for special education. They suggest that the primary role of the school psychologist should be problem-solving consultation with educators and parents, who then provide direct services to students. Regardless of the model of consultation, the purpose of the consultation is to provide consultees with assistance in addressing a child’s academic, behavior, or social skills problems. The authors then briefly describe the stages of consultation, from problem identification through evaluation of the intervention plan. They suggest that ubiquitous assessment undertaken during the consultation process will produce interventions tailored to and realistic for the student’s circumstances. The authors then briefly discuss other roles for school psychologists including development of prevention programs, providing staff development, and research and program evaluation. The next section of the chapter is a brief history of special education law explaining how school psychologists have become involved over the years in testing children for special education eligibility. The authors rightly point out that states have gone far beyond federal law in mandating excessive testing requirements. They then explain that the most frequent test psychologists are asked to administer is the IQ test, and they describe the problems in using these: intelligence has never been adequately defined; scores fluctuate; each test is merely one sample of behavior; and these tests do not provide information useful for developing interventions. Even though they obviously look with disfavor on such tests, the authors then describe three such tests in detail. They then move on to describing what they believe school psychologists should do when undertaking assessment, but this section is brief and weak. First, a broad definition of assessment quoted from the National Association of School Psychologists is given. Next, the authors point out that school psychologists should use information from other chapters of this book. Finally, they discuss what others have suggested be done when the practitioner is asked to administer individual intelligence tests (Barnett & Carey, 1992). Assessment in mental health settings is addressed in Chapter 7. Mental health settings are defined as all public and private mental health organizations where marriage and family therapists or licensed clinical social workers serve as part of a psychiatric treatment team. These teams in many agencies begin contact with an initial assessment and diagnosis process that is distinct from but directs later therapy. This assessment is based on the multiaxial diagnostic system described in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition-Text Revision, which is described in some detail at the beginning of the chapter. Part of the assessment process includes a mental status examination, which the authors believe can be conducted in the spirit of ubiquitous assessment. This would involve a more “enmeshed” connection between assessment and therapy. For example, helping a client feel that they are in a trusting, therapeutic relationship rather than merely being examined and assessed can make the initial interview both more useful and more pleasant. The authors provide quite a bit of detail about conducting this examination, including a helpful rubric for assessing the general appearance and behavior of the new client. In this chapter for the first time, they also discuss the challenging task of interpreting and integrating assessment information, but only briefly. Because ubiquitous assessment is based on observations, interviews, and other non-standardized methods, it is crucial that novices be guided on how to appropriately sort through and interpret all of the information from a variety of sources. Forming a useful picture of a client in any setting that can lead to effective interventions requires thoughtful integration of a wide range of material. Such efforts are much more difficult than merely reporting scores from a battery of standardized tests, and methods for doing so could profitably be expanded in future works by the authors. Chapter 8 cogently describes the elements of the written report, beginning with the key point that the report should be written with the purpose of converting assessment data into appropriate intervention related to the reason for referral. In discussing the basics of report writing, Goodman and Carey point out that one of the most important tasks is to thoughtfully integrate information, especially when inconsistencies occur across situations or sources. The practitioner must analyze and explain why discrepancies occur. Although not stated by the authors, this information can be useful in generating hypotheses that can be tested in interventions designed for the client (e.g., if student X disrupts Mr. Y’s class, but not Ms. Z’s, the practitioner could suggest elements of Ms. Z’s classroom management plan to Mr. Y). The authors also suggest that the clinician not take the reason for referral at face value, but engage in collaborative consultation with the parent or teacher to determine what the real problem is. The next section of the chapter examines each element of the report. In the Reason for Referral section the authors emphasize that the reason for referral should not be to determine eligibility for special education services, because merely focusing on that decision does not assist in developing interventions. In the Background Information section they point out that only information related to the reason for referral should be included. This section should include educational history and a description of any previous interventions. The Behavioral Observations section should provide a picture of clients’ functioning in their natural environment, including the classroom. Classroom functioning should be described in terms of peer comparisons. The authors suggest several important questions that should be answered in this section related to school and home functioning (e.g., How much practice does the student need to complete a task? Does the student respond to feedback? Does the student ever experience success? When? Are rewards provided to the client and siblings for appropriate behavior?). Under the Results and Interpretation section, the authors provide a Traditional Assessment section. They mention again that they do not promote the use of traditional norm-referenced tests, but they describe a method of reporting of test results if these are required. They recommend reporting standard scores and percentile ranks but not age and grade equivalents. I am surprised that they advocated subtest analysis and did not advocate reporting confidence intervals or ranges instead of standard scores, given their earlier critiques of published norm-referenced instruments. They also provide an example of the way results could be written from more informal (ubiquitous) assessment sources. The authors point out that the final section of the report, Recommendations, should be written to “require ongoing consultation” and should suggest interventions related to the reason for referral. This section was brief and could have benefited from examples of the range of recommendations that could be possible. Otherwise, this chapter was a good overview of issues that should be addressed when writing reports. Chapter 9 addresses ethical considerations. The authors suggest a continuum of ethical transgressions with three levels of impropriety. The lowest level “impugns the therapist’s credibility,” and the only example given is not returning a phone call. The middle level involves failure to identify boundaries between professional and personal behavior, with examples such as inappropriate jokes or shopping on the Internet during work hours. The highest level “includes all activities for which the counselor or therapist will be terminated,” with examples such as failure to report child abuse, or sexual contact with clients. The remaining sections of the chapter address several topics that are not directly tied to these three levels of transgressions. These include standard issues such as working within one’s scope of practice and competence, abiding by confidentiality rules and limitations of client privilege, protecting client privacy and using informed consent, and appropriately communicating assessment results. Noteworthy inclusions are the sections on avoiding bias in working with underrepresented groups and avoiding using single measures for making high-stakes decisions. Chapter 10, entitled Ubiquitous Assessment, puts the authors’ perspective in a larger societal context and showcases support for their perspective from other disciplines. Goodman and Carey first report signs that our culture is moving toward values consistent with ubiquitous assessment. They point to the Supreme Court case Grutterv. Bollinger, which supported the use of qualitative assessment in college admissions to encourage affirmative action. They also mention recent changes recommended for the reauthorization of the Individuals with Disabilities Act that emphasize a process of intervention and progress monitoring rather than a focus on special education eligibility. They also discuss a shift from a sickness/medical model to a health/wellness model in the mental health field. Although somewhat of a stretch, they next suggest that “ubiquitous assessment is a unifying theory” connecting the assessment and intervention fields much as string theory links quantum physics and Einstein’s theory of relativity. They also describe parallels between ubiquitous assessment and some aspects of recent feminist theory. The next section of the chapter discusses the importance of conceptual validity in ubiquitous assessment. Conceptual validity “posits that the truth or proof of the veracity of one’s experience is dependent upon the individual perspective (p. 195).” And the authors suggest that the validity of ubiquitous assessment is best evaluated through this type of qualitative experience because they reject more standardized, objective older paradigms. I agree that the goal of ubiquitous assessment is to address the individual needs of clients, and an “immediate, gestalt connection” is a positive thing. However, I would hope the authors would not want to throw out the baby of individually tailored objective measures with the bathwater of published standardized aptitude and achievement tests in assessing the impact of therapeutic interventions or assessing the effectiveness of ubiquitous assessment itself. I believe that the ubiquitous assessment perspective owes a great deal to the behavioral approach in many therapeutic fields, even though it is often considered part of the tradition that postmodernism and these authors reject. The behavioral approach has long championed the value of individually tailored problem-solving assessment and the relative uselessness of published standardized measures for designing interventions (e.g., Bergen, 1977). The behavioral methodology, focusing on publicly observable behavior as the dependent variable (Kendler, 2005), can also be very helpful in generating convincing data because it is verifiable by others. Gathering classroom observations that show that a student’s appropriate classroom participation has increased over time, or allowing a student to graph her reading progress using Curriculum-Based Measures can have a powerful impact. Relying solely on qualitative experiences like degree of rapport and connectedness is not enough evidence to persuade others of the value of ubiquitous assessment. Later in the chapter the authors do encourage quantitative as well as qualitative studies to validate the ubiquitous assessment process, because of the current lack of supporting data. Integrating the section on conceptual validity into the section on the need for research could have provided more clarity about how a variety of qualitative and quantitative sources can be used to validate a ubiquitous assessment perspective given the complex and “nonlinear” nature of typical practice. The authors then step away from assessment to larger issues of professional practice to discuss the importance of a spiritual dimension in one’s work. They suggest such an element is important to provide balance in life, to enhance connectedness with clients, and to engage in self-examination to enhance personal and professional growth. Overall, this book provides a snapshot of the field of clinical assessment in transition. Goodman and Carey have detailed the problems of traditional, standardized published measures, but their articulation of ubiquitous assessment doesn’t go quite as far as it could in laying out the purpose and especially the processes required for the new approach. Their use of the term “ubiquitous” seems clumsy and less than compelling. They sometimes seem to emphasize the importance of “seamlessness” between assessment and intervention almost for its own sake rather than as a means to the end of designing effective interventions tailored to the individual. They also devote a surprisingly large part of the book to tests and procedures of which they are highly critical. But they have brought together several lines of thinking in an effort to move the field forward to better serve clients and help create a “more psychologically supported community based on principles of social justice and meaningful connections…(p. 204).” References Barnett, D. & Carely, K. (1992). Designing interventions for preschool learning and behavior problems. SanFrancisco: Jossey-Bass. Bergan, J. (1977). Behavioral consultation.Columbus, OH: Merrill. Black, P., & Wiliam, D. (1998). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80 (2),139-44. Black, P. & Wiliam, D. (1998b). Assessment and classroom learning. Assessment in Education: Principles, Policy and Practice, 5, 7-75. Kendler, H. (2005). Psychology and phenomenology. American Psychologist, 60, 318-324. About the Reviewer Susan Green is an associate professor at Winthrop University in the Center for Pedagogy. Before coming to Winthrop, she worked as a practicing school psychologist for ten years in Oregon and South Carolina. Previously she had served on the psychology faculty at George Washington University for ten years. Her current areas of interest include classroom assessment and instructional and motivational strategies to enhance student achievement. She holds Ph.D.s in Educational Psychology and Social Psychology. Copyright is retained by the first or sole author, who grants right of first publication to the Education Review.