Biddle, Bruce J. and Saha, Lawrence J. (2002). The Untested Accusation: Principals, Research Knowledge, and Policy Making in Schools

map>
	Biddle, Bruce J. and Saha, Lawrence J. (2002). The Untested Accusation: Principals, Research Knowledge, and Policy Making in Schools. Westport, CT: Ablex Publishing. Pp. xvi + 302 $71.95 ISBN 1-56750-622-4 Reviewed by Paul D. Hood WestEd October 21, 2003 Scientifically based research has recently become a familiar term in the educational research community. However, the perennial concern is whether educational research, no matter how scientifically based, can actually find its way into use in schools. The Untested Accusation provides us with a qualified, but generally strong, affirmation that school principals in both the United States and Australia are indeed aware of a broad range of educational research and they use that research knowledge in their schools. In this book Biddle and Saha flesh-out in substantial detail the findings from analyses of 120 in-depth interviews with U. S. and Australian school principals that were conducted in 1991 and 1992 and initially reported in Biddle, Saha & Anderson (1993) and Saha, Biddle, & Anderson (1995). Their study focuses on how principals acquire, evaluate, and think about knowledge derived from educational research, and how that knowledge is used to influence policies and practices in their schools. Their findings challenge common claims about the supposed faults and weak impact of educational research¾thus the primary title, The Untested Accusation. This decade-long effort was stimulated by an edited synthesis of research (Saha & Keeves, 1990) and an edited book focused on the use of research in education (Anderson & Biddle, 1991). The experiences in editing these books led the study investigators (Anderson, Biddle and Saha) to the realization that relatively little research was available on the impact of research knowledge on education. Following the earlier publications of reports of the study, Biddle and Saha continued to analyze the data and authored this volume. Chapter One, Does Research on Education Have an Impact? This very short chapter lays out the rationale for the study. The authors note that research on education has been criticized as having little impact. “Was it actually true that knowledge from research had few to no effects on users in the education community? And if it was not true, how did such users . . . actually view and utilize knowledge from research on education?” (p. 3) Chapter Two, Thinking Out the Problem. The first section of this chapter defines a number of key terms such as research, research on education, knowledge, and knowledge use. A second section discussed social research and its problems, especially those due to differences in ideology. Knowledge in social research is “always more complex, more tentative, and inevitably, placed within social and historical contexts. ¼ [Thus] applying social research knowledge is always more problematic than applying knowledge from physical or biological research.” (p. 11) Social researchers and users of research live in separate worlds with different concerns (p. 11) A sampling of some of the dilemmas of research on education discussed by the authors include the following: Critics say educational research “is uniquely flabby, weak, trivial, poorly conceived. or inappropriate for solving problems¼” Research has played a weak role in education reforms. All kinds of competing (and sometimes antithetical) activities, methods, and standards may appear under the educational research umbrella and this generates confusion. Advocates have sometimes justified support for research by claiming that research alone can solve problems. Education research is plagued by weak studies and this impression is enhanced the extremely broad coverage in ERIC. Researchers have been unwilling to defend their craft. (p. 15) Yet some research has had huge effects on users, such as measurement of intelligence, teachers’ expectations and the differential treatment of students, or the studies of prejudice that influenced the Supreme Court’s Brown vs. Board of Education decision. (p. 15) A short section on prior studies reviews work related to the study. The authors state that they found it difficult to locate systematic studies of effects of research knowledge on users in the educational community. However, extensive research had already appeared concerning principals, their roles, forces that affect them, and consequences of their behavior. Thus they decided to focus their study on school principals. Because the authors were located in the United States and in Australia, it was convenient to design the study to include a comparison between school principals in both countries. The chapter continues by laying out theoretical assumptions underlying the study. Some of the key items are the following. Knowledge is defined as “organized packets of information about events appearing in oral, written, or electronic form.” Knowledge resides in the symbolic representations of a society rather than in the thinking of the individual. This knowledge exists in various forms, but most knowledge is verbal knowledge. Most knowledge associated with research is also expressed verbally. Thus the core claims of research knowledge involve statements about events that are presumably backed by evidence. The authors call these core claims empirical propositions. These empirical propositions rarely appear in isolation. “Rather empirical propositions are normally bundled together within traditions of effort that involve a specific topic of interest and similar techniques, operational definitions, ways of interpreting evidence, and underlying assumptions about the nature of events.”(p. 21) The authors distinguish among three forms of transmitted knowledge: Primary sources, such as papers, articles in refereed journals, and monographs; Secondary sources, such as review articles, textbooks, and workshops; and Tertiary sources, such as newspapers, magazines, TV, radio, trade books, and unreviewed reports on research. They note that problems with secondary sources are that research knowledge is simplified, stripped of details, incomplete, and perhaps biased. A further problem with tertiary sources is that they are open to partisan misrepresentation. The authors decided to ask respondents to provide examples of education research in their own words. They restricted the phrase “research knowledge use to potentially observable changes in the conduct of principals or others associated with the schools whom the principal can influence.” And they required that use of research knowledge be linked to a causal trail. Thus the study focused on four related areas: ways principals acquire knowledge from research, opinions they hold about that knowledge, research knowledge they are familiar with, as well as depth and breadth of their understanding of that knowledge, and how they use that knowledge in their professional lives. An important additional assumption of the study was that knowledge use would be promoted if principal worked in a school environment that favored innovation. Chapter Three, A Design for Investigation. The investigators considered various education user groups, but decided to focus on school principals. They studied the use of research knowledge by 120 school principals; 81 in the United States (all in Missouri) and 39 in Australia (all in the Australian Capital Territory (ACT) or the state of South Australia). The main focus was on how school principals acquire, evaluate, and think about knowledge from educational research, and how that knowledge is used to influence policies and practices in their schools. Their sampling included principals in primary and secondary public, parochial, and independent schools. Using structured interviews and a background questionnaire, data was collected and interviews were transcribed and coded during 1991 and 1992. Given the small sample size, findings are presented as data displays and simple statistics, primarily t-tests and correlations. The design assumes that four processes¾acquisition, opinions about research, familiarity with research, and use¾were likely to form a causal sequence and all four processes would be affected by environmental conditions (33 variables regarding background experiences, professional experiences, initial career stances, present career stances, school characteristics, and innovation forces). In the first stage of analysis, these environmental variables were correlated with knowledge source variables (exposure to sources, evaluation of sources, knowledge learned from sources). In the second stage, environmental and knowledge source variables were correlated with opinion variables (regarding research and regarding innovation). In the next stage environmental, knowledge source, and opinion variables were correlated with familiarity variables (volunteered examples of research knowledge, and recognition of a set of 20 research examples presented by the interviewer). And in the last stage, all previous sets of variables were correlated with knowledge use variables (volunteered examples of use, descriptions of policy decisions, and set of recognized topics). Throughout all the analyses, separate computations were made for the U.S. (Missouri) and Australian (ACT and South Australia) samples. Given the very large number of variables and the relatively small sample sizes, the investigators adopted a matching strategy in which relatively large and apparently significant correlations were ignored unless they appeared in meaningful clusters. Extensive quotes from respondent interviews are provided to support the statistical evidence. Chapter Four, Principals’ Exposure to Knowledge Sources. Two strategies were employed to investigate exposure to sources. In the first section of the interview, respondents were asked to volunteer examples of useful research knowledge and to provide details about those examples. In this section, respondents were asked where they had learned about the research they volunteered. In a fourth section respondent were handed a list of nine types of “sources for expert knowledge” plus an “other sources” category and asked a series of questions about each source, for example, How often was the respondent exposed to the type of source during the previous year? The nine types of sources were: Professional meetings, Workshops organized by others, Workshops organized by the respondent, Professional journals, Professional books, Professional bulletins from regional or national sources, Professional bulletins from district or system authorities, and University or college courses. A residual tenth category, Other sources, was replaced during the coding process with two additional categories, Other Professionals and Long term personal interests. Respondents were asked about their uses of the sources, details about each type of source, and usefulness of the source. The responses to both the volunteered sources and the pre-categorized sources were coded into 11 types of sources listed above. Most principals cited secondary sources. Americans were more likely to name professional journals. Australians were more likely to name professional meetings and professional books. The most important source type for everyone was professional journals. The authors note that prior studies reported that for principals, face-to-face interaction was the most important source for ideas about innovation. “To say the least, our findings do not support such conclusions.¼the most important source type for our respondents was not other professionals but professional journals and [bar chart displays] reveal substantial citations for workshops, bulletins, and professional books. (However, this reviewer notes that examination of the interview schedule makes it clear that only after asking an extensive set of questions about nine types of sources was the respondent asked, “How about other sources of professional knowledge?” It was only here that the respondent might mention other professionals. The category “other professionals” was created as part of the post interview coding process. Thus, the failure to identify face-to-face professional interactions as an important research knowledge source seems most likely due to the way the interview was designed.) Chapter Five, Principals’ Opinions About Research and Innovation. Principals hold generally positive views of research. Typically, respondents were not highly critical of research and considered the knowledge it generated as having a positive impact on their professional work. Critics have charged that educational research is badly flawed, and educators are hostile to it. “Our evidence implies that the typical principal is not hostile to educational research and its knowledge, thus the critics have been dead wrong about the opinions of at least this key group of educators.” (p. 121). Most principals say that they learn personally from research knowledge and that they use it to influence their colleagues and others. Moreover, most principals hold positive views about innovation. However, the authors found no significant relationships between attitudes toward research and attitudes toward innovation. Moreover, the authors found that most of their demographic, background, and prior experience variables failed to predict principals’ views toward research or toward innovation. The major exceptions were that reading professional books predicted positive opinions about research knowledge in Australia and about innovation in both countries. The chapter concludes with a discussion of some of the differences found between Australian and American principals, for example, American principals were more likely to talk about potential flaws in the conduct of research and ways in which research can be misused. Chapter Six, Research Knowledge That Principals Volunteer. When asked about examples of research knowledge they considered useful, respondents provided a very rich range of information. Contrary to some critics, who have argued that educators are disinterested in educational research knowledge, most of the principals in this study found it easy to recall and talk about their knowledge of research. The typical principal volunteered four to eight examples, usually covering various domains of professional responsibility. The authors conclude that the typical principal commands a broad range of salient knowledge of educational research. Moreover, the findings indicate that most principals can recall some details associated with the examples they volunteer. However they generally remember only the core contributions of their examples. In terms of predictors of this volunteered knowledge, principals in both countries tend to have greater ability to recall research knowledge when they are regular readers of professional books, have more scholarly interests, or lead a school where teachers have more academic qualifications. Moreover, principals in both countries tended to better recall auxiliary detail of the examples they volunteered when they acquired habits of scholarship or spent more time in settings where details are discussed. Thus, differences in their command of salient research knowledge are related to personal intellectual interests and exposure to specific experiences where details about research are made available. The chapter again concludes with a discussion of national differences. For example, American principals retain slightly more information about the knowledge they volunteer than do Australian principals, which is perhaps explained by the fact that the American principals have generally completed more postgraduate work. Chapter Seven, Research Knowledge That Principals Recognize. While chapter six examines knowledge the principals volunteer, this chapter focuses on principals’ ability to answer questions about a broad range of research topics. Twenty phrases selected from a much larger list of “important” research topics suggested by eminent researchers were listed on a sheet of paper entitled, “Examples of Research Generated Knowledge” and handed to the respondent at the start of this part of the interview. Respondent were asked, in turn, whether they were familiar with the research knowledge associated with each phrase. If the respondent was familiar with the phrase, follow-up questions were asked to assess depth of understanding. The typical principal was able to recognize and talk about findings associated with about half of the 20 topics. However, there were national differences. Of the 20 topics, all but six topics were recognized by at least half of the respondents interviewed in either county, but only three topics were recognized by most of the respondents in both countries. In contrast to the salient knowledge regarding volunteered examples, the knowledge concerning recognized items on this list of topics was much less rich in detail. Regarding predictors of core knowledge about the recognized topics, principals in both countries could provide more salient examples when they are regular readers of professional books, have more scholarly interests, or lead a school where teachers have more academic qualifications. When recall of auxiliary aspects of that research knowledge are involved, principals had more command when they have acquired detailed habits of scholarship or spend more time in settings where research details are discussed. American principals retained slightly more information than did Australian principals, perhaps because American principals are more likely to have completed postgraduate courses or because they were exposed more often to news about educational research. Chapter Eight, Principals’ Reported Use of Research Knowledge. The investigators used several strategies to investigate the issue of research use. Early in the interview respondents were asked to volunteer examples of useful research knowledge. About midway in the interview respondents were asked to describe a recent event in which a decision about a policy or program issue had been made in the respondent’s school, the part they had played in the event, whether research knowledge had played a role in the decision, and, if so, how that role had played out. Finally, when respondents were asked whether they were familiar with any of 20 phrases representing research examples, respondents who recognized a phrase were asked about the knowledge it represented and whether they or others planned to use or were already using that knowledge in their school. When asked about research knowledge they volunteered, more than three-fourth of the principals described applications for these examples. When asked about a policy-relevant event that had recently taken place in their schools, roughly half explained how that event had been influenced by research knowledge. And when asked about the 20 research topics, more than half the principals provided details of applications for one or more of those topics. However, it should be noted that not all principals reported applications of research knowledge. Some principal reported more applications than others. And respondents who reported more applications tended to also report reading more professional books and exhibited other qualities associated with scholarly interests. The investigators had originally assumed that because principals play a central role in making policy decisions for their schools, they would also be major players in applying research knowledge. However, the evidence presented suggests that this assumption may be only partly correct. When respondents spoke about the research knowledge they volunteered they did describe major roles for themselves. But when they spoke about decision-making events involving research knowledge use, they often spoke about other actors, including authorities representing the state or school system, or other professionals, primarily teachers. Another significant result of this part of the study suggests that applications of research knowledge use are likely to take various forms and to involve many different research topics. Respondents mentioned many applications such as new types of classroom processes, innovations in staff development, student assistance programs, procedures for encouraging teachers, and techniques for classroom supervision. Regarding the role research knowledge plays in affecting school-level policy, the evidence advanced in this chapter suggests that some research knowledge is applied directly to policy decisions. However, in other cases, especially when asked about major uses of research knowledge, respondents frequently stressed what they had learned personally from that knowledge and how that personal knowledge may have been used. Generally, the finding suggest that Australian principals seem to be slightly more thoughtful than American principals. Australian principals reported significantly more propositional and slightly more nonpropositional applications for volunteered examples of research knowledge and they were more likely to say that research knowledge initiated the decision-making event they described. The major exception to this generalization appeared for the 20 research topics where American principals demonstrated greater familiarity with those topics. If there is perhaps greater thoughtfulness among the Australian principals, a likely explanation is suggested. Australian principals work fewer hours per week than do American principals. Chapter Nine, Findings, Conclusion, and Implications. This chapter contains an unusually well crafted summary of the main findings and conclusions drawn from the study. (After scanning chapters one, two and three, a reader pressed for time could skip to this chapter to get a cogent overview of the entire study.) The following statements repeat or paraphrase the major findings, which are, of course, described in much greater detail in the chapter. Regarding the ways the school principals acquire knowledge about research: Most principals are frequently exposed to sources where research knowledge is regularly displayed and discussed. Most principals learn about research knowledge from the sources to which they are exposed. Most principals hold positive opinions about research on education; few think it is valueless or are hostile to its use. Most principals are rarely concerned about supposed flaws; rather they are more concerned with problems with transmission and use of research knowledge. Most principals find it easy to provide examples of useful research knowledge. The typical respondent volunteered examples from four to eight different research traditions. Most principals are familiar with a surprising range of research topics that are latent in their thinking. The typical respondent recognized about half of 20 topics presented to them. Most principals serve in schools where research knowledge has an impact on policy decisions and educational practice. Biddle and Saha note that, “Taken together, the above seven conclusions suggest that much of what has been written about the supposedly unique problems of educational research and its lack of impact should be challenged.” (p. 225) The authors then review findings concerning secondary sources and depth of knowledge: Most principals are not regularly exposed to primary sources; rather the bulk of the sources are secondary. Most principals retain memory of details about salient research examples that are appropriate for a user population. Most respondents could explain the implications of the research they cited clearly. Roughly 90 percent provided precise details about information sources for at least one research tradition they had chosen to talk about. [However] most principals do not retain details about knowledge of the kind that are more useful for researchers or scholars concerned with creating and interpreting research knowledge. Most principals retain fewer details for research knowledge examples that are latent in their thinking than for those examples that are salient for them. Biddle and Saha observe: “Yes, principals depend largely on secondary sources for acquiring research knowledge, and this means that they are often exposed to versions of that knowledge that are incomplete and may be biased.¼most of the knowledge details they command would appear to be those needed to make efficient use of that knowledge in their schools.” (p. 227) Regarding breadth of research knowledge and its effects: Most principals are in regularly contact with many types of information sources that provide access to research knowledge. Most principals retain various examples of salient research knowledge that cover different responsibilities. When asked to nominate examples of useful research the typical respondent came up with four to eight examples. [Moreover,] most principals retain knowledge of research covering many topics that are latent in their thinking. When asked about 20 preselected topics, most principals recognized five to fifteen of these topics. Most principals serve in schools where research knowledge affects policy and practice for many different types of events. Biddle and Saha state” “These results suggest that the typical principal has at least some familiarity with a broad range of research knowledge topics. These principals are generalist when it comes to research knowledge and should not be expected to have the same command of details for specific research topics that we would expect to find among investigators or scholars whose work specialized on these issues.” (p.228) Regarding forms of research knowledge utilization: Most principals do assimilate and ponder the implications of research knowledge that may (or may not) be directly and immediately applied. Most principals also believe that research knowledge is used directly in their schools, particularly to affect professional colleagues and target persons. In some cases, principals play key leadership roles when research knowledge is used directly, while in other complex event histories key roles are played by others, particularly authorities and other professionals in the school. Opinions about Innovations: Most principals favor educational innovations, an active role for themselves in the process, and shared decisions concerning the innovations. Most principals’ opinions about innovations are not tied closely to their opinions about research knowledge in any simple way. Conclusions about national differences among principals: Australian principals are more likely to learn about research knowledge through face-to-face contact; Americans are more dependent on secondary sources. Australian principals are more likely to be thoughtful about research knowledge and its applications than are American principals. (American principals work longer hours and they depend more on secondary sources.) [However,] American principals retain more details about and examples of research knowledge in their thinking than do Australian principals. (Perhaps because more American principals are exposed to postgraduate courses.) (It should be noted that the interview samples were evidently too small to generate many clear findings about differences among principals in urban, suburban or rural schools, or among levels or types of schools.) General effects of environment and experience: Principals who read professional books more often favor use of research knowledge, know more about that knowledge, and are more likely to serve in schools where that knowledge is used. Principals who have higher levels of professional education and who desire a university position more often know about examples of salient research knowledge and are more likely to serve in schools where that knowledge is used. Principals who attend religious services frequently are less likely to know about the auxiliary details of research knowledge Several conclusions are also drawn about effects of environment and experience that appear in one country but not the other. Reviewer Commentary The Untested Accusationis an extraordinary contribution to our understanding of how school principals in the United State and Australia acquire, evaluate, and think about knowledge derived from educational research, and how that knowledge is used to influence policies and practices in schools. Although the data examined in the study are now more than a decade old, we know from other studies of information search and use behavior of educational information users, including school principals, that patterns of information search and use tend to be highly stable. Therefore, there is good reason to believe that most of the general findings, if not all the details, from this study are likely to as true today as when the interviews were conducted. The one remarkable exception is that the Internet was not a prominent source in 1991 and 1992. It surely would be an important source today. One might also ask whether we can generalize safely from a sample 81 principals in one state (Missouri) to the entire United States or from a sample of just 39 principals in the ACT and the state of South Australia to all of Australia. We know from 1990 Census data that school age children in Missouri were less ethnically diverse and that there were proportionately fewer limited English speakers than for the entire U.S. However, it seems quite unlikely that these small demographic differences would have much, if any, bearing on the variables considered in this study. Perhaps more troubling might be the size of the samples. They are certainly useful for an in-depth exploration. But generalization to the entire populations of school principals in the United States and Australia may be problematic. These quibbles aside, this study represents a major contribution to our understanding of how research knowledge is acquired, evaluated and used by school principals. The study was carefully designed and conducted. The data were thoughtfully analyzed and reported. Bruce Biddle and Larry Saha deserve our commendation for persisting with such dedication to make their findings available in this book length report. References Anderson, D. S. & Biddle, B. J. (Eds.) (1991). Knowledge for policy: Improving education through research. London: Falmer Press. Biddle, B. J., Saha, L. J., & Anderson, D. S. (1993). Research knowledge and educational policy. (Award Number R117E00111-90) [Final Project Report submitted to the Office of Educational Research and Improvement]. Washington, D.C.: U.S. Department of Education. Saha, L. J., Biddle, B .J. & Anderson, D. S. (1995). Attitudes towards education research and policy making among American and Australian principals. International Journal of Educational Research, 23(2), 113-126. Saha, L. J. & Biddle, B. J. (1998). Forskning betydelsefull főr rektorer. Pedagogiska Magasinet, 2(98), 6‑11. Saha, L. J. & Keeves, J. P. (Eds.) (1990). Schooling and society in Australia: Sociological perspectives. Canberra, Australia: Pergamon Press/ANU Press. About the Reviewer Paul D. Hood WestEd 300 Lakeside Drive, 18th Floor Oakland, CA 94612-3534 Paul Hood is Director of Planning at WestEd. His doctorate is in social psychology. Since joining the Far West Laboratory in 1966, now WestEd, he has directed several R&D programs concerned with development of dissemination support systems, and development and evaluation of products and systems to enhance communication and effective use of educational R&D information and products by school personnel. He has also conducted surveys of information users, policy research, and syntheses of evaluation studies of R&D use.