Biddle, B. J., & Saha, L. J. (2002). The untested accusation: Principals, research knowledge, and policy making in schools. Reviewed by Ruth Rees, Queen's University

Biddle, B. J., & Saha, L. J. (2002). The untested accusation: Principals, research knowledge, and policy making in schools. Westport, CT: Ablex Publishing. P. xvi + 302. $71.95   ISBN 1-56750-622-4 Reviewed by Ruth Rees Queen’s University December 19, 2005 The purpose of this book is to address the concerns that the public has raised against educational research, with its supposed deficiencies and alleged lack of impact. This book documents a comparative study of 120 schools principals, representing different types of schools in Australia and the United States, about their use of educational research knowledge: the techniques they employed to acquire their knowledge how they evaluated it, and what they know about it, and how that knowledge might have been used in their schools, if at all. To explore whether research knowledge use differs among various types of schools or experiences that impinge on the lives of principals in both countries, qualitative and quantitative data collection help to answer the general question: Does research on education have an impact? First the authors, Bruce J. Biddle and Lawrence J. Saha, reviewed related research on educational research. They defined ‘research’ as empirical investigations only, focusing either on education or having implications for educational practice. Second, they also distinguished between two forms of understanding: salient ideas, in which elements of knowledge are ‘up front’ and appear readily in the thinking of the person; and ‘latent ideas, where such elements are ‘further back.’ Salient ideas are said to reflect recent experiences and matters of current concern to the individual, but latent ideas are said to include both thought elements that were once salient and others the individual has perceived from the beginning to be peripheral to his or her interests. The authors contend that answers volunteered in response to open-ended questions are assumed to reveal salient ideas whereas responses to closed-ended structured questions may reveal both salient and latent thoughts. (pp. 9-10) Through research on principals, they investigated: the ways principals acquire knowledge from research bearing on education; the opinions they hold about that knowledge; the research knowledge they are familiar with, as well as the depth and breadth of their understanding of that knowledge; and how they use that knowledge in their professional lives. The authors chose principals in the United States from the state of Missouri; in Australia, the principals came from the Australian Capital Territory and the state of South Australia. Both groups represent schools of diverse populations, according to the authors. The schools from which the principals were selected represented public schools, parochial schools (Roman Catholic), and independent schools (e.g., private schools). In the United States, the sample came from metropolitan, suburban, small-city, and rural communities. In Australia, the sample came from metropolitan and rural communities. They represent 37 primary schools (46%), 44 secondary (54%) in US schools, totaling 81; in Australia, the sample was from 16 primary schools (41%), and 23 secondary schools (59%), totaling 39. All principals had to be in their jobs for at least 12 months (i.e., to have had some experience) in order to be selected. The demographics revealed that the Australian principals are older, more experienced, and have less formal education than American principals, but work less hours than their American counterparts. These points come up over and over again in the writing. The one demographic variable that I do not understand the reason for including was the sex of the principals: the US principals were comprised of 43 women (53%); in Australia, 12 women (31%). I fail to understand the reason for including this demographic: are these proportions representative of the state or region, or of the two countries? The authors in their final chapter claim that their comparative findings could be applied to principals in either the United States or Australia. Yet before this can be done, I would suggest that a more thorough discussion about the representativeness of the samples be provided. The authors developed their own questionnaire, or rather two versions of the questionnaire adapted for relevance from principals from either the US or Australia, comprised of six sections. Section one explored research-associated knowledge that was salient in the thinking of respondents. Respondents were to provide examples of research knowledge that they had found useful in some aspect of their job as a principal. Section two was designed to explore whether respondents were expected to innovate or to resist innovations as principals. Section three asked participants to describe a recent event in which a decision had been made about policy issues in their schools, the role that they played, and whether that decision had been influenced by research knowledge, and if so how. In Section four, respondents were asked about their recent involvement with sources that can provide access to research-based knowledge (from a given list of ten such sources) which were put into Appendix A3. Principals were then asked to provide details of their access to these sources, to volunteer examples of research knowledge they had recently acquired (if any) from each source, and to judge the usefulness of each source for acquiring research knowledge. Section five explored the range of respondents’ latent familiarity with a number of different phrases representing example of important research traditions: phrases denoting technical concepts with which they might be familiar, phrases individual some theoretical explanation they might know about, and phrases alluding to a theoretical explanation they might understand. Finally, Section six explored opinions about principals’ research knowledge: their attitudes about the value of research knowledge, whether research generates knowledge useful for educators, and how they characterize their own professional use of research knowledge. As well, the participants were asked to complete a document about their background and experiences. Variables between groups were considered statistically significant if there was a difference of at least 2.0 between results. Six following chapters explored the findings from the different sections in the questionnaire. Detailed descriptions of the data analysis methods are described in each of those chapters, complete with discussions of how the data categorizations were developed. Overall results were provided in each chapter, and then presented again in the final chapter. Some interesting findings are documented below, followed by the entire list of findings quoted from that final chapter. It was interesting to me that principals reported that they received their research knowledge most commonly from a variety of journals. Interestingly, US principals reported workshops by others as their next most common source; Australian principals said post-secondary courses. The Australian principal participants cited The Australian Journal of Education and the Journal of Educational Administration, primary sources of research, as the most popular journals. The others were secondary sources, to include the American journal Phi Delta Kappan. The American principal participants sited secondary resources as being most common, with the two most popular journals being Educational Leadership and Phi Delta Kappan. No single journal or group of journals has a monopoly on providing these principals with their research information. They all indicated that they read at least one professional book per year. Another finding suggested that principals in both countries tend to have more command of salient research knowledge when they are regular readers of professional books, have more scholarly interest, and lead a school where teachers have more academic qualifications. Moreover, principals in both countries tend to have greater command of the auxiliary details of salient research knowledge when they have acquired the detailed habits of scholarship and spend more time in settings where those details are discussed, but the Australian and American principals differ, in part, in the degree to which they are in command of this knowledge. While principals are more often familiar with salient research knowledge when they read professional bulletins regularly, the kinds of bulletins varied. In the US where schools are structured by districts, principals read district and system bulletins regularly. In Australia, where it is state-run, the Australian Council for Educational Research distributes broadly educational research reports called SET which are said to have been widely read by Australian principals. (Note. I went to the ACER website to try to locate some examples of these SET reports, but was unsuccessful finding reports with that overall name. Perhaps they are only available to principals within Australia. I only mention this because the authors referred to these reports many times.) I reaffirm the authors’ contention that the results of the study indisputably indicate that the accusation that educational research has little or no impact on those educational practitioners is incorrect with respect to “typical principals” in both United States and Australia. Their many findings as quoted from the authors (pp. 223-241) are reported under the following headings provided by the authors: A. Educational research truly feckless? Most principals are frequently exposed to information sources where knowledge from research is regularly displayed and discussed. Most principals also learn about research knowledge from the sources to which they are exposed. Most principals hold positive opinions about research on education; surprisingly few think that it is valueless or are hostile to its use. Most principals are rarely concerned about supposed serious flaws in educational research; rather, they are more often concerned with problems associated with the transmission and use of the knowledge that research generates. Most principals find it easy to volunteer examples of useful educational research knowledge that are salient in their thinking. Most principals are also familiar with a surprising range of educational research topics that are latent in their thinking. Most principals serve in schools where research knowledge has an impact on policy decisions and educational practices. B. Secondary sources and depth of knowledge. Most principals are not regularly exposed to primary sources for research knowledge; rather, the sources…are secondary. Most principals retain details about salient research knowledge examples that are appropriate for a user population. Most principals do not retain other details about salient research knowledge examples that are more useful for researchers or scholars concerned with creating and interpreting that knowledge. Most principals retain fewer details for research knowledge examples that are latent in their thinking than for research knowledge examples that are salient for them. C. Breadth of research knowledge and its effects. Most principals are in regular contact with many types of information sources that provide access to research knowledge about education. Most principals retain various examples of salient research knowledge in their thinking that cover different realms of their responsibilities. Most principals retain many examples of research knowledge, covering multiple topics that are latent to their thinking. Most principals serve in schools where research knowledge affects policy decisions and educational practices for many types of events. D. Forms of utilization. Most principals do assimilate and ponder the implications of research knowledge that may (or may not) be directly and immediately applied. Most principals also believe that research knowledge is used directly in their schools, particularly to affect professional colleagues and target persons. In some cases, principals play key leadership roles when research knowledge is used directly, whereas in others, complex event histories appear in which key roles are played by other actors, particularly authorities and other professionals in the schools. E Principals’ opinions about innovation Most principals favor educational innovation, an active role for themselves in innovative processes and shared decisions concerning innovations. Most principals’ opinions about innovation are not tied closely to their opinions about research knowledge and its application in any simply way. Next the difference findings between the Australian and US principals are detailed below: F. Effects of national context. The ways the principals learn reflect the structure of education in their country: U.S. is state-wide education and through school districts; Australia has no school districts. By comparison, Australian principals are more likely to learn about research knowledge through face-to-face contact; Americans are more dependent on secondary sources. By comparison, Australian principals are more likely to be thoughtful about research knowledge and its application than are American principals. By comparison American principals retain more details about and examples of research knowledge in their thinking than do Australian principals. By comparison, American principals retain more details about and examples of research knowledge in their thinking than do Australian principals. G. General effects of environment and experience. Principals who read professional books more often favor the use of research knowledge, know more about that knowledge, and are more likely to serve in schools where that knowledge is used. Principals who have higher levels of professional education and who desire to obtain a university position more often know about examples of salient research knowledge and are more likely to serve in schools where that knowledge is used. Principals who attend religious services frequently are less likely to know about the auxiliary details of research knowledge. H. Effects of environment and experience that appear in one country. Australian principals who are younger are more likely to know about examples of salient research knowledge. Within American schools where teachers hold high levels of academic qualification, principals are familiar with more examples of salient research knowledge, and applications of research knowledge more often appear. American principals who make use of district/system bulletins more often know about examples of salient research knowledge and the auxiliary details of latent research knowledge. The authors close with three strategies for improving research impact. They are, and I quote: Improve user understanding. If principals play a key role about educational policy and policy making in school, they should be made aware that they have a responsibility in keeping up ad using educational research. Moreover, their training should include, among other things, assessment in the quality of educational research and how to best disseminate that information to staff. Promote better tertiary reports on and reporting of educational research. Journalists and members of the media have a key role to play in this regard. Revise the knowledge distributing system particularly in the United States, to try to emulate something like the Australian/New Zealand SET (with the understanding that they did not think the SET idea would work within the US, and indeed seemed to be on the decline in Australia, but not in New Zealand). I have some suggestions for improvement for this book in its second edition. First, the references should be updated. Most of the references are in the 1990’s, when the bulk of the literature review for this study took place; to ensure its applicability well into the twenty-first century. Second, I also suggest eliminating such words as “recently published…study” to work that is now ten years old; this was in reference to Weiss’s 1995 study of principals. It is especially annoying when it is mentioned not once, not twice, but several times. Third, another revision of this book should attempt to eliminate redundancy as well; it is quite repetitious in parts, unnecessarily so. Four, the authors should consistently present the data both in figures and in the body of the writing in the same order. For example, all but the first two figures have presented the data for Australian principals first, followed by American principals. I would suggest reversing this for consistency sake in figures 4.1 and 4.2. Five, while I like the title of the book, I think that it would make more sense if the title was changed to something for explicit such as: The untested accusations about educational research. One, there are many accusations or fallacious assumptions in the public purview about educational research, many of which addressed in this book. Two, adding the words ‘educational research’ to the title would allow it to be more easily accessed through general topic searches, and hence increase the probability of its being used. I have two more suggestions. Six, the word ‘feckless’ is overused in this book. I’d suggest using synonyms such as ‘weak,’ ‘inconsequential,’ ‘ineffectual,’ as a few examples. Finally, my seventh and last suggestion is for the authors to rethink sentences starting with the word “And.” Too many sentences starting this way, and I for one find it unnecessary and inappropriate. All that aside, this is a very informative book to read. The book is well-written, very clear in both its structure and content, with lucid introductions to and summaries of the book in its entirety, and of each chapter. It should be useful to both academics and practitioners in education. Much thought and care have been taken in developing the different response categories. It is these categories themselves that could be open to criticism because they force responses from participants. However, the authors have remained open to emergent categories, and have provided some explanations which should anticipate and then minimize others’ criticism about how the research was carried out. Indeed, two of the main responsibilities for academics should be to ensure that what research we carry out is carried out properly but also is disseminated broadly to the correct target audiences. While many researchers focus on the first responsibility, we are said to neglect the second. Yet, as was pointed out by Biddle and Saha, the utilization of our research in schools is what really demonstrates it impact. Both researchers and practitioners take note! About the Reviewer Ruth Rees, PhD, is a Professor in the Faculty of Education at Queen’s University, Kingston, Ontario, Canada. Her expertise is in the area of school leadership, and specifically principals’ training in Ontario, Canada. She is a regular reviewer for Education Review. Copyright is retained by the first or sole author, who grants right of first publication to the Education Review.