Chapter 1 The Evidence Agenda - OECD.org · 2019-04-24 · Chapter 1 The Evidence Agenda Tracey...

1. THE EVIDENCE AGENDA – 15

EVIDENCE IN EDUCATION: LINKING RESEARCH AND POLICY – ISBN-978-92-64-03366-5 © OECD 2007

Chapter 1 The Evidence Agenda

Tracey Burns and Tom Schuller, OECD Centre for Educational Research and Innovation

In this chapter, we examine the resurgence on interest in “evidence”, outline the roles and relationships between major stakeholders, and provide an overview of this publication.

Part One: Setting the Stage: The Evidence Agenda and Methodological Issues

In recent years a number of public crises have seized the attention of the world and required rapid responses from governments to ensure the health and safety of the public and maintain their confidence in policy makers. The 2001 UK foot and mouth crisis and the emergence of SARS in Asia and Canada highlight the difficulties of decision-making for policy makers and also the necessity for time-sensitive information on which to base those decisions. In each of the above examples, dramatic action needed to be taken urgently. These decisions resulted in substantial economic and societal losses, as well as worldwide reaction to piles of scorched carcasses, delayed elections, passengers wearing masks while taking public transport, and restricted movement. The dangers were contained and the emergencies passed, but post factum evaluations revealed that perhaps the policy decisions taken were not, in fact, the most effective or efficient ones available (House of Commons Science and Technology Committee, 2006).

An even bigger issue is that of climate change. Here the debate has been very public, including much on the nature and reliability of the evidence presented, notably in respect of the report by Sir Nicholas Stern (Stern, 2006). Here the contestability of research and the difficulty of matching timeframe to analysis and action were very much in evidence.

Although extreme examples, these real-life instances illustrate the dilemma facing policy makers of all stripes, including education, who must make swift, time-sensitive decisions based on the information they have available. Yet often the information that is readily available is not “perfect” research on the subject. This could be either because the rigorous research relevant to policy needs has not been conducted; or because there is a disjoint between policy and research communities such that the relevant information is not widely disseminated and so overlooked by the policy maker; or simply that the research that is available is contradictory and so does not suggest a single course of action that could be reflected in policy. Yet clearly it is crucial that policy decisions be made

16 – 1. THE EVIDENCE AGENDA


with the best available evidence, as the decisions made can, as the above examples demonstrate, have far-reaching impacts on all members of society.

Interest in and discussion on how educational policy is aided by research, and specifically on what kinds of evidence from research count or should count in policy and practice, have grown dramatically in the last several years. Discourse on the nature of scientific evidence, challenges for raising awareness of policy makers, and pleas to bring research into classrooms all trigger intense and sometimes heated debate on what constitutes “evidence-based” or “evidence-informed policy”, the terms which have come to denote this field, and which we define as “the conscientious and explicit use of current best evidence in making decisions and choosing between policy options”.

The activity from which this book (see below) comes was originally titled “evidence-based policy research” (EbPR). However, as the debate progressed this term became increasingly unsatisfactory for many of those participating. This is principally because it is seen to imply too tidy and rational an image of policy-making, as some kind of clinical and objective operation. “EbPR”, moreover, can be ambiguous: is it the research or the policy which is evidence-based? The balance swung more towards the use of research to foster evidence-informed policy, which leaves more open the actual extent of the evidence use. So this report settles for evidence-informed policy research (EIPR) as its focal term. In very broad terms the research that is used to produce evidence-informed policy can be distinguished from purely scientific research in that the former is oriented to informing action while the latter is oriented to developing theory and testing hypotheses (although these are not mutually exclusive categories). The distinction is important, as burdens and standards of proof of causality are very different, and in many cases those responsible for evidence-informed policy are obliged to use the best available evidence at a given moment in time, whatever its strict epistemological status.

In brief, our concern is with the most appropriate forms of evidence to assist public policy-making, and the effective mechanisms for developing and utilising that evidence.

The (re)emergence of “evidence”

Despite the recent resurgence of interest in evidence-informed policy research (EIPR), the issues underlying the discussion are not new. More than a decade ago, the 1995 CERI report Educational Research and Development: Trends, Issues and Challenges raised the question of why the role of governments in promoting and using educational research had emerged as a prominent issue, and suggested several possible answers. These included:

• A belief that education and knowledge are increasingly important factors in innovation and economic growth in OECD countries.

• A growing concern with accountability in respect of educational expenditures.

• A concern about the quality and effectiveness of current educational research.

CERI’s work on knowledge management pointed to the key role of knowledge-based innovation in education (OECD, 2004). However the country reviews of educational R&D have confirmed the following features as commonly (though not universally) characterising OECD systems (OECD, 2003):

• Low levels of investment in educational research.

• Generally low levels of research capacity, especially in quantitative research.

• Weak links between research, policy and innovation.



In most if not all countries, therefore, the issues of effective relationships between research and policy makers, capacity-building within those domains, and importance of allocating scarce resources in the most efficacious manner remain as important as they were ten years ago. So what has changed? Why revisit a discussion which was already well delineated ten years ago?

Key factors underlying this change are a greater concern with student achievement outcomes; a related explosion of available evidence due to a greater emphasis on testing and assessment; more explicit and vocal dissatisfaction with education systems, nationally and locally; increased access to information via the Internet and other technologies; and resulting changes in policy decision-making. These are accentuated by broader issues to do with the perceived legitimacy of policy-making in general.

A rising concern with the outcomes of education (e.g., student achievement) is one of the most significant overall shifts in educational policy orientation. Much previous work, including the OECD’s, concentrated on inputs (financial or otherwise) and participation rates. Today there is a mounting preoccupation with what happens as a result of these investments and activities. Outcomes are interpreted not only in terms of course completion or qualifications, but also in terms of skills and competences (as with the PISA study), access to and success in the labour market, and wider social outcomes such as health and citizenship attributable to education. In other words, policy makers are increasingly interested in what education actually delivers – and therefore with what educational research can tell us about that. A consequence of this has been the explosion of evidence of different kinds resulting from the enormous increase in testing and assessment.

A significant force behind this orientation to outcomes has been the greater interest shown by treasuries and finance ministries in the effectiveness of educational expenditure, as a major component of public expenditure generally. Where annual spending rounds have a strong grip on policy, requests from treasuries for evidence of results present challenges to their counterparts in educational ministries and to policy makers at other levels in the education system. The challenge is to gather evidence which is both appropriate and convincing. This is especially the case where the request is that impacts and effectiveness be given monetary values.

Increased access to information via the Internet and other technologies (including the easily digestible and publicity friendly information arising out of testing and assessment) is potentially the great equaliser, allowing as it does a greater number and wider breadth of individuals with access to all available information. At the same time, however, this process has effectively removed many of the established gate-keepers or quality controls for this information – a process sometimes called “disintermediation”. More information is available, yes, but is it good information? And is it presented accurately and in an understandable fashion? Can the reader use it in a comprehensible and useful manner?

These twin effects (greater access with less quality control) have had great impact on policy-decision making, requiring as they do the policy maker to weed through immense amounts of information of unclear quality in order to make decisions on behalf of a more informed constituency. At the same time, there has been a shift across most OECD countries to de-centralise decision-making in education (unlike many of the other areas covered by the social sciences), giving more responsibility and mandating to local authorities. Given greater information, less quality control, a more informed public, and a greater diversity of policy makers, the need for clear, reliable, and easily available evidence on which to base policy-decisions has become more important than ever before,



as has the need to find mechanisms to obtain reliable answers to pressing policy questions. The role of research for evidence-informed policy, then, becomes newly important.

How do OECD countries understand and engage in EIPR? Our perception of the debate thus far suggests that the approach to this issue can be structured around the following dimensions:

• who are the players and the quality of communication/interactions between the different sets of agents involved in commissioning, executing and implementing EIPR;

• the kinds of methodologies and epistemological paradigms which dominate within policy and research communities; and

• the kinds of mechanism available to resolve the tensions and difficulties involved in the process.

Naturally, these interact with each other and need to be related to the structure of governance, notably the levels at which policy goals are set and policies fashioned and implemented, the availability of the evidence, and the substantive policy issues which happen to be prominent at any given time.

Further dimensions could certainly be added, but we suggest, as a basic organising framework, that it is the interplay between these dimensions which defines the different approaches to EIPR that are found across OECD member countries (and most likely beyond); and it is this interplay which needs to be explored in order for good practice to be understood and a useful agenda to be set.

This publication explores these dimensions and then pushes the discussion into the drivers and facilitators underlying the interaction between them, with the aim of investigating new challenges and new opportunities for EIPR in education. It arises out of a 2004-2006 CERI seminar series that brought together researchers and policy makers from all OECD countries to review the main aspects of evidence-based policy research – methods, costs, and capacity – and discuss what constitutes evidence for research in education, how that evidence can best be utilised, and possible solutions to challenges observed by participating countries.

The four seminars in the series were hosted by the United States (April 2004 in Washington DC), Sweden (January 2005 in Stockholm), the Netherlands (September 2005, at The Hague), and the United Kingdom (July 2006 in London). The contributions to this publication come from a selection of these attendees, and represent the diversity of roles, perspectives, and experiences that contributed to the seminar series.

In addition to the Secretariat discussion on the evidence agenda and the roles and relationships among the major stakeholders, this publication includes contributions on:

• methodological issues and what counts as evidence;

• the policy/research interaction: the role of brokerage agencies;

• research for evidence-informed policy in practice: examples from the field;

• the politicians’ perspective.



Policy-research interaction: who are the players and how do they interact?

In order to understand the policy-research interaction we need to focus on:

• the quality of the relationships between the primary stakeholders (e.g., policy makers, researchers, practitioners, etc);

• how well they function in terms of information flows and trust levels;

• what could be done to improve the quality in each case.

As a starting point, Figure 1.1 summarises the potential lines of communication and interaction between these different agents. Given the focus of this discussion it privileges policy makers, researchers, and practitioners. Its function is primarily heuristic, i.e., it is designed to prompt reflection and analysis in respect of specific country or other situations (see Levin [2004] for a more complete discussion of the various stakeholders and relationships between them).

Figure 1.1. Primary lines of communication/interaction

Source: OECD.

Ideas for education reform are often perceived to come from within the system of policy makers and, to a lesser extent, researchers working on education. These are the primary, but not the only, agents in the game. In the literature on evidence-based policy making there has been a good deal of discussion on how to bridge the gap between the researchers and policy makers, both in terms of communication of needs and priorities and in terms of coming to an understanding of the different timescales required by each community (e.g., the mandate-limited scope of the policy maker versus the length of time it takes to do good research and pilot implementation as part of that research). This discussion implies that there is a unified concept and community both of researcher and policy makers and that the challenge lies in bridging the gap between these two disparate, but internally homogenous, communities. In real life, of course, nothing is ever that simple.

School boards/ parents

Policy makers

Researchers

Teachers/ practitioners

Media Leaders



The researchers

Educational researchers are to be found in many different locations and roles, from university-based academics to hired contractors working for government or private sector. The latter group in particular has grown dramatically in recent years, particularly in countries such as the United States and Great Britain, as a response to increased need both for answers to particular questions and for researchers capable of conducting sophisticated quantitative analyses. Private sector research firms market themselves as capable of producing relatively swift analyses to focused educational research questions, and demand for their services has grown with increased desire to direct funding towards programmes that have proven effectiveness in education.

Educational researchers may be educationalists, other social scientists, computer scientists or architects, among other things. While comfortable with this diversity of identification, education researchers are less comfortable with the diversity of perspectives and backgrounds. The EIPR debate has been characterised by differences of opinion on these matters that have turned the discussion into a battleground between groups of researchers that seem incapable of communicating with each other. The current state of affairs could reasonably be described as mutual antipathy between those researchers deemed to favour “quantitative” approaches versus the “qualitative” researchers (although these labels do not satisfactorily capture the differences that exist between the different groups, see Gorard and Taylor, 2004). This antipathy is generally evidenced by mutual ignoring, with only occasional periods of interaction between the groups. The increased attention paid to education in research has resulted in a paradoxical situation where people seem more willing to talk and somewhat less willing to listen than before. For this and other reasons, aspirations to combine multiple methods are more often voiced than achieved (Chatterji, 2004).

The policy makers

Who are the policy makers? In any context and in every country, policy makers can be found at national, state, or local levels. Education, as a central component of citizenship building and cultural expression, can be a hotly contested and jealously guarded jurisdiction. In an international context, countries have dramatically different education systems and ministerial models, from the decentralised structures of the federated countries (e.g., Canada, Germany), to the highly centralised (e.g., France, Korea). These different systems have implications for decision making and locus of control and uniformity of policy across geographic areas. They also have an impact on the amount and uniformity of funding available for various initiatives for change, including the support of research directed at providing evidence-based policy in education. Unified support on policy lines translates into very real capacity-building of research programmes and implementation of reforms in the classroom.

In recent years there has been a shift across most OECD countries to de-centralise decision-making in education, giving more responsibility and mandating to local authorities. Non-centralised leadership of course results in a number of different policy contexts and priorities within each country, and in some countries a concurrent fragmentation of the solicitation and funding of educational research. Nations that do not have strong central planning may see evidence-based policy issues as unaffordable, both financially and conceptually. Regional policy makers might have less incentive to contribute to the EIPR dialogue and less confidence that their knowledge needs might be met. For regions with very specific short-term knowledge needs (e.g., improving student



performance on standardised tests and/or working to reduce inequities within the system without damaging the performance of the most privileged), the pooling of resources required to engage the national research community might be perceived as a time-consuming and fruitless procedure. Even if particular regions or individual policy makers were convinced of the importance of evidence-based policy, the lack of generalised agreement on policy priorities coupled with long-standing national rivalries and greater or lesser willingness to share information could very well lead to a situation where policy makers decide the time and energy needed to invest in an evidence-based policy approach make it simply not practical.

There is a very real limit to what policy makers (national, regional, and local) can influence in their respective systems, and a very real concern among policy makers with choosing their battles so as to maximise effectiveness. Ideas for education reform that originate from policy makers will be as varied as the system itself. Whether or not those ideas are based on evidence and research will depend, in large part, on the priority given to evidence-based policy research in education in that particular country and/or region, as well as pressures from and connections to the stakeholders (researchers, community members, parents, etc). It will also depend on the ability of policy makers to hear and absorb the information that is being presented to them for use in evidence-based policy, and in their ability to understand the fundamentals of evaluation.

Institutional leaders, practitioners, and the community context

In addition to professional researchers and policy makers, a number of other players appear who are involved at different stages in educational reform, from producing ideas to gathering evidence to assessing the results. The role of practitioners – teachers, other educational staff and their unions – in the production, and interpretation of research evidence has been attracting increasing attention. This is a recognition both of the potential contribution which they can make, and of the need to maintain their confidence in the reform process if it is to be successfully implemented. School and college leaders wield considerable power, and can support a culture which favours the production and the use of research evidence, or inhibit it. One of the best ways to gain the support of school leaders is to involve them in the research process and thus give them a sense of ownership over the initiative (Slavin, 2004).

Until recently, there has not been a great deal of support for the production and use of research evidence in the classroom. The teaching profession by and large do not see themselves as practitioner-researchers, learning on the basis of research into their own activities by their peers. This is in contrast to doctors, who are trained to use and contribute to the medical research agenda. Encouraging the understanding and use of research by school leaders themselves could be particularly important for pedagogical research, both in terms of the validity and generalisability of field-based studies, and also as a way to encourage implementing research-based reform.1 If the research itself is valued and used (and conducted) by school leaders, the shared ownership will encourage its implementation in a way that something imposed externally by researchers on teachers will not. This requires a great deal of capacity building.

1 A current activity in the Education and Training Policy Division of the OECD’s Directorate for Education deals with school leadership in depth.



Bottom-up campaigning for change on the part of parents, students, and local communities, stemming from observations and effective practices in the classroom and at home, also has the potential to drive educational reform. School boards in particular have been highly active in campaigns aimed at changing educational policy and thus educational practice (e.g., reducing class size, use of formative assessment, etc). Many if not all of these ideas are based on personal observation and experience and have not been tested empirically, a fact which does not reduce the conviction of the observer regarding the efficacy of these ideas.

Ideas which are generally perceived as “intuitively reasonable” gain power and support of public opinion. This is especially the case where they are promoted by the media, who often play a major role in shaping, or stunting, the policy agenda. They can then be used as a basis for policy change and educational reform regardless of whether there has been any empirical testing. How then, to go from the power of intuitive knowledge that resonates with established observations and practices towards a strategic attempt to capitalise on the ideas generated in the field, test them, and then, if proven to be effective, implement them in policy? How to encourage practitioners, the community, and media to look for the evidence supporting a reasonable idea (and its corollary: how to encourage researchers to communicate results in a broad and accessible fashion)? Given the scarce resources for education reform, the high level of public interest in education and the importance it holds for national and regional policy makers and the diversity of opinions within the education research community, the ability to assess what works in education is critical. The rise in the use and priority given to evidence-based policy research is a direct result of this understanding.

Methodologies and epistemological paradigms: what counts and should count as evidence

There are various practical reasons why countries choose to involve themselves or not in work on evidence-based policy. These include the length of time required to obtain evidence, the funding required to support the research, and the possibility of jurisdictional wrangling. In addition to these practical considerations, the dominant research tradition within education and the social sciences may prevent serious engagement in the debate. There is little sense in operating with a model which assumes a linear and rational process that translates good evidence into effective policy, as the reality is much more complicated (see Nutley and Webb [2000], for a discussion). We can even ask whether good research is more likely to be the product of an effective system than vice-versa.

Practical issues aside, the debate on EIPR opens up some quite basic philosophical issues to do with the nature of knowledge, and how different methodologies are suited to different knowledge claims. Causation is a particularly problematic concept, but one that demands attention from policy makers who are responsible for allocating resources and accountable for the effects of these allocations. The debate reaches into OECD’s own work: OECD, and the Education Directorate within it, would certainly claim to base policy recommendations on evidence, but the nature of the evidence varies considerably. It includes large-scale primary research, notably the PISA study; national and thematic country reviews which draw on quantitative evidence but draw their originality from the expertise of the examiners and the quality of the examination process; comparative case study work which necessarily adopts a quite flexible framework; and secondary analysis or synthetic research at different levels of scale and ambition. It would be fair to



acknowledge that there is no unanimity within OECD on where exactly to draw the lines around what counts as evidence, nor on how it might be best used.

To countries with research traditions that are less used to empirical and quantitative methodologies in the social sciences, the discussion may seem completely divorced from the national reality. Evaluation as a component to policy (in the form of both pilot trials and ongoing assessment after the implementation of reform) may not seem necessary or helpful. OECD countries vary in the degree to which they attempt to objectively assess policy effectiveness. Even the idea of objective verification can be seen as unnecessary in the traditional context of a lack of understanding of research but clear intuitive agreement among the population (which of course includes teachers and policy makers).

In addition, some countries have a strong and rich tradition of qualitative methodologies such as action research and case studies, and education research has traditionally been thought of as their domain (St. Pierre, 2002). Case studies have been conducted as pilot projects for a variety of initiatives with the intention of scaling up the project if the experience was evaluated as favourable. Experience with the project in some of these cases is evaluated through interviews, questionnaires, and a variety of other qualitative research methodologies, which are then synthesised into a general evaluation of the project, including recommendations for change (Bogdan and Biklen, 2002). One of the strengths of qualitative research is precisely that it can give the depth of information required for, for example, recommendations for change or possible explanations of why something does or does not work. The policy research debate, dominated as it has been by disagreements on methodology and whether qualitative research is permissible at all, might be incomprehensible given national standards of research and education reform.

The debate regarding appropriate methodology stems from a very real concern about the overall quality of education research. It is unfortunately not difficult to find low quality education research with poorly derived research questions, inappropriate methodology and analyses, and misleading interpretation of the data. A low threshold for research quality has led to rifts within the community as well as damaging the reputation of education researchers in the eyes of other researchers (for example, social science researchers) (Feuer, Towne, and Shavelson, 2002). This in turn has led to difficulties for funders and confusion on the part of the policy maker.

In methodological terms one clearly identified area of weakness is in quantitative skills and the use of large data sets. This applies to assessment and evaluation data as well as experimental research design. Randomised controlled trials (RCTs) have received a great amount of attention and are clearly useful for causal questions and provide the rigour appreciated by funders. However, it seems clear that, as with all other methodologies, RCTs have both strengths and weaknesses and can be proposed as one of a set of appropriate methodologies for education research.

The urge to improve the overall quality of education research is one of the fundamental drivers of the EIPR discussion. The need for quality research and the appropriate methodology for questions of a causal nature (e.g., does a programme have the intended effect?) has been thoroughly spelled out (e.g., Angrist, 2004; Boruch, DeMoya and Synder, 2002; Cook, 2003). Hierarchies have been proposed to help guide evaluation of research aimed at addressing effectiveness issues, with preference being given to robust experimental and quantitative designs (Cook, 2005). For non-causal questions (e.g., how does a programme work and why?), another set of methodologies is required, again with emphasis on the rigour of the investigation. The importance of deciding first on a research question and then choosing the appropriate methodology with



which to investigate the question is clear (Shavelson and Towne, 2002), yet often this point is overlooked by researchers and policy makers alike (Berliner, 2002). All this supports our basic proposition that there is no single best method for or type of evidence-based policy research. A variety of proposals have been advanced regarding how best to combine methodologies in education and other social sciences, with the view to providing concrete proposals and explicit strategies (Gorard and Taylor, 2004).

Part One of this publication takes a look at the issue in the form of a dialogue between two well-respected methodologists: Tom Cook from Northwestern University (United States) and Stephen Gorard from York University (United Kingdom). Each author first states his position in terms of the key concerns for education research methodology, and what counts and should count as evidence. For policy-making, is it a question of using the best available evidence, or should only the best evidence be used (which would imply that evidence that does not meet the gold standard could not be used)?

After positioning their arguments, the authors engage in a dialogue around areas of disagreement and unresolved issues. Much of the unresolved issues can be conceptualised under the heading of capacity-building in educational research and methodologies. The discussion can be boiled down to two main unresolved issues:

• Which forms of capacity are most in need of expansion/strengthening;

• How and by whom should this be done.

Capacity-building

As Cook and Gorard point out in Chapter 2, capacity building is required to encourage more participation in the evidence-based policy research discussion. This is true in a range of different contexts: in national and international discourse but also by educational researchers, policy makers of all levels, and teachers. Dyson and Desforges (2002, quoted in Sebba, 2004) distinguish between strategies to broaden and deepen capacity. The former is concerned with doing more of the same, but better; the latter with enabling the system to do new things. They identify three themes around which capacity can be built:

• Development opportunities for researchers (refresh, retrain, update).

• Infrastructure development (dedicated centres, IT, networks).

• Practitioner and policy makers’ capacity to use and produce evidence.

Educational researchers, or the people doing research on education (see Cook, 2003, for a discussion on who is actually doing the research in education), need to be given enough training, financial support, and access to expertise to be able to conduct quality research. Additionally, there needs to be support from within the academic community for research that is policy relevant so that researchers who engage in this kind of research are not penalised in promotion ranking relative to colleagues who are pursuing a more traditional set of topics and publishing options. For those researchers aiming to provide policy-relevant work, specialised training is needed. At the moment, one area of clear weakness is that far too often researchers neglect to assess resource implications – when, for good reasons, this will be the second question the policy maker asks (after effectiveness). This requires knowledge of a range of techniques in addition to methodologies (e.g., understanding opportunity costs).



Researchers need to be able to disseminate the results in a manner that can be understood by the general public, including the policy maker. Research results that remain within the realm of academia will not be able to be understood or accessed when needed, greatly limiting their impact. This is a lesson that has already been learned (at least to some extent) by researchers in other fields such as medicine and agriculture. In the multi-lingual international context, the dissemination of research results in a readily accessible and easily understood manner increases the likelihood that researchers can build off of each other’s work, instead of squandering limited funds repeating the same basic research in various countries (we acknowledge that research must also be sensitive to cultural context to be relevant).

Policy makers may wish to reflect on why evidence-based policy is important and to understand the research process required to produce that evidence, allowing for the creation of realistic projects and deadlines for reporting on the part of the researchers. They might also be encouraged to think carefully about the importance of quality evidence and the need to trust the source of information (i.e., use a “gate-keeper” to filter all available information such that only the best-available evidence is used for decision-making). On regional and national levels, this would facilitate the funding and solicitation of research that seeks to address questions pertinent to policy. It would also allow for more interface between the policy and research communities and (at least theoretically) more reasonable expectations from both sides. On local levels, support for evidence-based policy research from school boards and teachers is an important step in allowing funding to be allocated towards research with the intention of improving practice. It also allows for the possibility that funding might be made available to build capacity of the providers to help ensure faithful implementation of reforms. Alternatively, without accepting any obligation to fund research, boards might demand to see the external evidence on which a proposed innovation is based before approving a change in policy or practice.

Teachers need to be supported so that they have the time and energy to implement education reform in the manner it was intended as well as understand the research that fuels such reforms. This requires giving them access to research that is written for the non-scientist, as well as some background in research and how to interpret results. Otherwise how realistic is it to implement top-down policy change based on research in a system of professionals without giving them the capacity to understand and evaluate the research for themselves? The strategy of exposure to research methods and using research as part of the teacher training programme is one very concrete way to build capacity. In every national context and every kind of system, education reform is, ultimately, only as good as what actually takes place in the classrooms. Reforms unpopular with teachers or perceived as unnecessary (or worse, ill-conceived) will be at best half-heartedly implemented, at worst actively resisted. Cordingley (2000; 2004) offers a summary of factors influencing the use of research by teachers, including perceived relevance, evidence of learning outcomes, and clear links from the research to classroom practice. Garnering support from the people on the ground is one of the most strategic approaches to encouraging active implementation of evidence-based policy.

As mentioned above, a very good way to achieve this is to expect teachers not only to understand research, but take the lead in initiating it at local level to give them a sense of ownership. The capacity-building required for this (in research methods, in creating networks for experience sharing, in building support from management for the time and energy required, providing in-service training for ongoing development) is extensive and implies a re-thinking of existing structures and expectations. It also, rather fundamentally,



implies a degree of teacher and school autonomy such that teachers could act on the basis of research findings. The United Kingdom’s Teaching and Learning Research Programme described in Part Three of this book specifically includes a significant capacity-building component.

Part Two: Mediating the Research/Policy Interface: The Role of Brokerage Agencies

Bridging the gap between internally and externally heterogeneous groups of researchers, policy makers, and educators is no easy task. One of the most common, and possibly most dangerous, ways to do this is to by-pass the communities themselves and turn instead to the Internet as a source of “research” on any given policy concern. The problem is, of course, that there is no system of quality control, and, if contradictory information is presented, there is no formalised process for consolidating or challenging evidence presented as research. The ability to assess the quality of evidence available – whether pulled from the Internet, presented by researchers, or offered by parents and teachers – is a fundamental prerequisite for informed policy-making.

One strategy to bridge this gap has been through think tanks and other intermediate agencies. These are most common in the Anglo-Saxon countries, but are becoming more common in continental Europe. They occupy a particular space in the divide in that they are not quite researchers, but not quite policy makers either. They have traditionally been quite aware of the power of public opinion and seek to harness it to lobby for particular causes and arguments. They are an important “bridging” institution in that they can be extremely effective and highly professional, but of course are not neutral, instead usually marshalling research evidence that would reinforce their particular policy priority.

A number of different initiatives aimed at bridging the divide between policy makers and researchers as well as assessing the quality of evidence available have been developed as general examples of “brokering”. Brokering can be informal, e.g., the exchange amongst colleagues of research evidence and information related to a policy issue at hand. It can also be more formal, e.g., the creation of ties between national research institutions and their closest policy counterparts. In the past ten years this process has developed to the extent that formal brokerage agencies have been developed to officially facilitate both the process of information sharing and ensuring a certain level of quality control. Brokerage agencies vary in type and can be designed to be in-house and aid a particular Ministry to increase effective communication regarding the research and policy interface, evaluate proposed changes and policy recommendations, and assess the implementation of these programmes (e.g., Norway). However most brokerage agencies have a broader agenda and seek to collaborate with as wide a community of researchers and policy makers as possible, so as to broaden the relevance of their work and findings.

Part Two of this publication provides a series of studies of existing and newly formed brokerage agencies. In lifespan, they range from EPPI-Centre (eppi.ioe.ac.uk) in the United Kingdom (formed in 1993) to the Knowledge Chamber of the Netherlands, which was created in 2006 as a result of the OECD/CERI workshop series that is also the basis for this publication. The Danish contribution describes the process behind the creation of a brokerage agency and the political and administrative aims guiding this process.

The brokerage agencies are also distinguished in their goals and means, with New Zealand providing an example of a brokerage programme embedded within the Ministry



that provides hands-on guidance to those wishing to conduct a synthesis of available evidence (www.minedu.govt.nz/goto/bestevidencesynthesis). In contrast the Canadian Council on Learning (www.ccl-cca.ca/), although also federally funded, is not embedded with the various provincial Ministries and, in addition to various other roles, is prepared to conduct reviews and syntheses based on policy priorities and within a very short timespan. Similarly the What Works Clearinghouse (www.whatworks.ed.gov) (United States) works in collaboration with a number of other institutes and subcontractors, to provide information and databases of research syntheses of replicable high-quality interventions, with a particular focus on the methodology of randomised controlled trials. Interestingly, it also conducts consumer surveys and questionnaires in order to ensure that the service it provides is meeting the demands of the users (researchers, practitioners, policy makers, etc).

As part of the continuing effort to draw parallels between education and other subject areas and learn from relevant experience, Part Two also provides an example from the field of social care in the form of the Social Care Institute for Excellence (www.scie.org.uk) (United Kingdom). From this contribution we can observe both the similarities of experience, goals, and efforts of a brokerage agency in this domain, as well as some shared challenges.

All of these centres have the goal of encouraging dialogue between policy makers, researchers and educators with the aim of providing tools and capacity-building within these communities to evaluate what works and what does not work in education. An important first step in this process is the creation of a database of quality education research on particular topics that are of interest to policy makers, as well as providing clear goals and criteria for conducting and evaluating educational research. These criteria serve as a baseline for conducting reviews of research, reviews which can then be used to provide systematic evidence as to the effectiveness of particular policy objectives or classroom practices. A key component to these brokerage agencies is the transparent exchange of findings: all reviews are available on the various brokerage agency websites, and all methodologies used by the review process are defined in detail. Many of the centres require reviewers to commit to updating their work on a regular and pre-defined basis, so as to include new evidence and maintain a state-of-the-art synthesis on each particular topic. And all of the centres have a commitment to disseminating research results to as wide an audience as possible, in order to work on affecting top-down and bottom-up change to the system.

The agencies aim to address one of the key issues identified in the OECD’s 2003 report on New Challenges for Educational Research, and elsewhere (e.g., Raudenbusch 2005): the accumulation of knowledge. Educational research is conspicuously weak in its ability to continuously develop and refine a body of knowledge which is quasi-universally acknowledged as well-founded. The research community, through its induction and training procedures, has the crucial role in this, but brokerage agencies can also have a major part in designating the most recent authoritative additions to the knowledge pile.

Such brokerage agencies can and do play a key role in bringing together the disparate communities and bridging the gaps in the EIPR process. They have provided resources and tools for researchers, policy makers, and educators to openly engage in the discussion of what works in education, and allowed for capacity-building in each of those domains. They are potentially a vital mechanism in aligning supply and demand and there are valuable formative lessons to be learnt from their experiences to date. As Part Two of this



publication demonstrates, standard challenges that have yet to be resolved by these agencies include how best to:

• incorporate all stakeholders into the process;

• address the tension between the time required for solid research and the necessity of quick results for policy-making;

• disseminate findings to all stakeholders, including media, parents, and students;

• ensure sustainability and stability of funding.2

Part Three: Evidence-based Policy Research in Practice: Examples from the Field

The examples from the field are drawn from a number of different countries (Canada, Finland, Singapore, United Kingdom) as well as a variety of different models of how best to put the use of evidence into practice. Canada’s contribution looks at the launch of the National Children’s Agenda, which focused on the long-term vision of fostering good Canadians by aiming for lifelong learners, productive workers, nurturing parents, and engaged citizens. In order to best achieve this, a number of decisions were made regarding key developmental outcomes for children and the need for a body of evidence from multiple data sources and analytic methods. This national data system has flourished in part because it was intended to provide reliable and stable flows of data and as such was assured over time and protected from short-term budget cuts and thinking.

In contrast Life as Learning (Finland) was set up as a discrete project of the Academy of Finland and ran from 2002-06. As a national research programme, it aimed to encourage the development of a research culture and support interdisciplinary and international research projects. A number of interesting developments have already come out of this process and have been well-received in Finland – however the time-bound nature of the exercise and funding difficulties discussed in the paper speak to a challenge shared by many research communities.

The contribution from Singapore is unique in that it is a non-OECD country with an unparalleled set of funding and research opportunities, including a strong database with data on the entire student population. This reflects its major commitment to a profile as a knowledge-based high-skill economy. In addition, this breadth and depth of information allows for comprehensive investigation into a variety of education issues and also permits a particular emphasis on arguments presented to policy makers. In this sense this example is unique not only in the methods, funding, and scope of the research, but also in the relationships involved between the major stakeholders.

Lastly the United Kingdom’s Teaching and Learning Research Programme (TLRP) coordinates over 500 researchers in 60 project teams. An immensely complex project with assured medium-term funding (2000-2011), it aspires to improve the quality of education research in the United Kingdom as well as contributing new knowledge. An explicit emphasis is how best to disseminate activities, results, and events of the

2 OECD/CERI proposes to investigate brokerage agencies more thoroughly in its 2007-08 Programme of Work, addressing the following key questions: What are the different roles played by brokerage agencies in different countries? And how effective are they at this role, and how have they attempted to address the challenges outlined above?



programme, and as such there are active ties to policy makers, practitioners, community members and media in addition to the researchers.

These examples touch on the key issues of the preceding sections, allowing us to look at how the various interactions play out in a real world situation. For example, all of the contributions allow us a close look at the quality of the relationships between the primary stakeholders (e.g., policy makers, researchers, practitioners, etc), and particularly the two points as raised in the preceding section on roles and relationships:

• How well they function in terms of information flows, trust levels.

• What could be done to improve the quality in each case.

In addition, the examples all provide the answer that that particular programme has chosen to the question posed in Part One: What Counts as Evidence? Both Canada and Singapore provide cases of extensive and thorough data collection, using all means available (in the case of Canada, the focus is on socio-demographics rather than education, whereas Singapore uses its unique funding and database opportunities to compile an unparalleled evidence-base). Finland and the United Kingdom, on the other hand, mix models of methods and types of service delivery to best serve the question and mandate of the organisation.

All of the programmes must also concern themselves with the dissemination of their findings and the best way to bridge the gap between the evidence they can provide and the needs of the policy maker, and in this respect all of the programmes benefit from their close ties to government. The UK’s Teaching and Learning Research Project in some senses acts as its own brokerage agency, with dissemination and communication of results playing a central role in project planning and development. Life as Learning (Finland) faces perhaps the most traditional challenge of a research project, in that it must strive to maintain a high profile and disseminate its results. In contrast both the Canadian and Singaporean examples are much more closely tied to government and their respective Ministries and so the brokering in some sense has already been done (i.e., the research is high priority and centrally incorporated in policy-making).

This part thus serves as both a set of concrete examples interesting in their own right and as a way to observe the various theoretical issues discussed in this volume in practice. Similarly to Part Two, it includes an example from outside the education field, in this case a programme governed by the Department of Human Resources and Social Development (Canada).

Part Four: The Politicians’ Perspective

The evidence-based policy research debate has to large extent been focused on the major players of policy maker and researcher. In Part Four of this publication we hear the voice of the politicians, charged with making decisions and evaluating the best way forward for the education system in their jurisdiction. In this part we hear from Johnny Nilsson, the former Secretary of State for Education in Sweden, who speaks of the experience of the past and some of the limitations and concerns of the politician in a period without the current emphasis on evidence-based policy. We hear also of the serious weight that has been placed on this process in the Netherlands, and how and why this work was so important to the Minister of Education, Science, and Culture, Maria van der Hoeven. The Netherlands has a number of initiatives that have followed from the OECD/CERI work and are looking to the future.



In addition, Jane Davidson of Wales uses her experience as the longest serving Minister in the United Kingdom to discuss the role of evidence in policy-making and the struggle she has engaged in to bring it to the forefront of the policy-making process. Finally we have a unique case study from Poland, which has recently brought together current and former Ministers and decision-makers in an on-going effort to raise the priority and profile of education research in policy-making in that country. Jerzy Wisniewski, former senior civil servant and CERI Governing Board member, contributes the challenges and aspirations of a country that is just beginning to focus on these issues.

Concluding note

The debate on what counts as evidence will certainly recur. In another dozen years we may be noting the same weaknesses in educational research, and the same flaws in the communication between research and policy on education. Or fresh obstacles may have emerged, from some of the trends observed in this chapter: the massive diffusion of information of varying quality, without established intermediaries; the pressure on policy makers to reach decisions, whether or not good evidence is to hand; or further modifications in the relationship between different parts of the decision-making process such as a tightening grip of finance ministries on education spending. But some progress will also have been made, in all probability; we can guess that rigorous research techniques will become more widely understood and applied; that practitioners and perhaps also policy makers will broaden their evidence base; and that the potential of brokering will have been explored in many countries. But that is to write the history of the future (Attali, 2007); for now it is only to be hoped that lessons from the past can be learned.

References

Angrist, J. (2004), “American Education Research Changes Tack”, Oxford Review of Economic Policy, Vol. 20(2), pp. 198-212.

Attali, J. (2007), Une brève histoire de l’avenir, Fayard, Paris.

Berliner, D.C. (2002), “Education Research: The Hardest Science of All”, Educational Researcher, Vol. 31(8), pp. 18-20.

Bogdan, R.C. and S.K. Biklen (2002), Qualitative Research for Education: An Introduction to Theories and Methods, 4/E, Allyn and Bacon, Needham Heights, MA.

Boruch, R., D. DeMoya and B. Synder (2002), “The Importance of Randomized Field Trials in Education and Related Areas”, in F. Mosteller and R. Boruch (eds.), Evidence Matters: Randomized Trials in Education Research, The Brookings Institution, Washington DC.



Chatterji, M. (2004), “Evidence on ‘What Works’: An Argument for Extended-Term Mixed-Method (ETMM) Evaluation Designs”, Educational Researcher, Vol. 33:9, pp. 3-13.

Cook, T. (2003), “Why Have Educational Evaluators Chosen not to Do Randomized Experiments?”, Annals of American Academy of Political and Social Science, Vol. 589, pp. 114-149.

Cook, T. (2005), Consultant’s Report, Meeting on Evidence-based Policy, Stockholm Sweden, OECD, Paris.

Cordingley, P. (2000), Teacher Perspectives on the Accessibility and Usability of Research Outputs, paper presented at the British Educational Research Association annual meeting.

Cordingley, P. (2004), “Teachers Using Evidence: Using What we Know about Teaching and Learning to Reconceptualize Evidence-based Practice”, in G. Thomas and R. Pring (eds.), Evidence-based Practice in Education, Open University Press, New York, pp. 77-90.

Dyson, A. and C. Desforges (2002), “Building Research Capacity”, National Education Research Forum, see www.nerf-uk.org

Feuer, M., L. Towne and R. Shavelson (2002), “Scientific Culture and Educational Research”, Educational Researcher, Vol. 31(8), pp. 4-14.

Gorard, S., and C. Taylor (2004), Combining Methods in Educational and Social Research, Open University Press, England.

House of Commons Science and Technology Committee (2006), Scientific Advice, Risk and Evidence Based Policy Making, Document HC 900-1, The Stationary Office Limited, London.

Levin, B. (2004), “Making Research Matter More”, Educational Policy Analysis Archives, Vol. 12(56), pp. 1-20.

Nutley, S. and J. Webb (2000), “Evidence and the Policy Process”, in H.T.O. Davies, S.M. Nutley and P.C. Smith (eds.), What Works? Evidence-based Policy and Practice in Public Services, The Policy Press, England, pp. 13-42.

OECD/CERI (2003), New Challenges for Educational Research, OECD, Paris.

OECD/CERI (2004), Innovation in the Knowledge Economy: Implications for Education and Learning, OECD, Paris.

Raudenbusch, S.W. (2005), “Learning from Attempts to Improve Schooling: The Contribution of Methodological Diversity”, Educational Researcher, June/July, pp. 25-31.

Sebba, J. (2004), Developing an Evidence-based Approach to Policy and Practice in Education, UK Higher Education Academy, Discussion Paper 5.

Shavelson, R.J. and L. Towne (eds.) (2002), Scientific Research in Education, National Academy Press, Washington DC.

Slavin, R. (2004), “Translating Research into Practice on a Large-scale: Lessons from Success for All (US)”, Keynote presentation at DfES research conference, “Research in Education, What Works?”, 19 November, London, England.



St. Pierre, E.A. (2002), “Science Rejects Postmodernism”, Educational Researcher, Vol. 31(8), pp. 25-27.

Stern, N. (2006), Review Report on the Economics of Climate Change, HM Treasury, London.

What Works Clearinghouse.org (2005), www.ed.gov/nclb/methods/whatworks/ww/13.

BIOGRAPHY – 177


Biography

Adrienne Alton-Lee is the Chief Education Adviser for the New Zealand Ministry of Education’s Iterative Best Evidence Synthesis (BES) Programme. Her role is to strengthen the evidence-base informing policy and practice in education and to provide medium term strategic advice to government. Dr. Alton-Lee is a Fellow of the International Academy of Education. She was formerly a teacher, classroom researcher, Professor and an Associate Editor of Teaching and Teacher Education. She has published in leading educational journals including the Harvard Educational Review, the Elementary School Journal, the International Journal of Inclusive Education and the American Educational Research Journal.

René Bugge Bertramsen is the Deputy General Director for the Danish University and Property Agency within the Danish Ministry of Science, Technology and Innovation. Since 1999 he has been involved in reforms aiming at enhancing the quality of the Danish educational R&D system (such as the establishment of the Danish Pedagogical University – DPU – and the R&D centre Learning Lab Denmark). Mr. Bertramsen was responsible for the University Act of 2003 which gave Danish universities a new governance system, i.e. boards with external majority and employed rectors, deans and department heads. In 2006-2007 he was responsible for a merger process where government research institutes were integrated with the universities and a number of single-faculty universities were merged with larger multi-faculty universities, including the merger of DPU with multi-faculty University of Aarhus.

Robert Boruch, Professor, University of Pennsylvania (USA). Dr. Boruch is current co-chair of the Steering Group of the International Campbell Collaboration, and principal investigator for the Institute of Education Sciences What Works Clearinghouse, which is designed to be a central and trusted source of information on evidence about what works in education. Dr. Boruch is an elected Fellow of the American Academy of Arts and Sciences, the American Statistical Association, and the Academy for Experimental Criminology. He has received awards for his work on evaluation policy, randomised trials, and on privacy of individuals and confidentiality in social research. Dr. Boruch’s academic background is in psychology, statistics, and mechanical engineering, with degrees from Iowa State University and Stevens Institute of Technology.

Satya Brink is currently Director, National Learning Policy Research, Human Resources and Social Development Canada. She and her team are responsible for developing evidence in support of policy development for lifelong learning for the Government of Canada. This work includes analysis on outcomes for each age group and type of education as well as the impacts of earlier learning on subsequent learning. In her previous post, she was responsible for research on human development based on two major Canadian longitudinal surveys. During this time she and her team produced a major body of evidence based on the National Longitudinal Survey of Children and Youth which influenced major new initiatives of the Canadian government in support of children and their families.

178 – BIOGRAPHY


Tracey Burns is a research and policy analyst for the Centre for Educational Research and Innovation, OECD, Paris. Previous to this she worked on social determinants of health across the life-span with Charles Ungerleider & Associates in Vancouver, Canada. As a Post-Doctoral Fellow at the University of British Columbia, Dr. Burns led a hospital-based research team investigating newborn infants’ responses to language. Tracey Burns holds a BA from McGill University, Canada and PhD from Northeastern University, USA. She is the recipient of various awards and honours, including the UBC Post-Doctoral Fellowship, a student-nominated university teaching award, and the American Psychological Association Dissertation Research Award.

Thomas D. Cook is the Joan and Serepta Harrison Chair in Ethics and Justice and Professor of Sociology, Psychology, Education and Social Policy at Northwestern University, where he is also a Faculty Fellow at the Institute for Policy Research. He has a BA from Oxford University and a Ph.D. from Stanford University. He is interested in causal methods for the social sciences and in the joint effects of neighborhoods, schools, peers and families on how young people develop socially and cognitively. He is a Fellow of the American Academy of Arts and Sciences and the Margaret Mead Fellow of the American Academy of Political and Social Science. He has been awarded the Myrdal Prize for Science by the American Evaluation Association, the Donald Campbell Prize for Innovative Methodology by the Policy Sciences Organisation, and a Distinguished Research Scholar Prize of the American Psychological Association. He is the author or editor of 10 books and over 150 chapters and articles.

Jane Davidson is the Assembly Member for Pontypridd and former Deputy Presiding Officer for the National Assembly (Wales, United Kingdom). Since October 2000 she has been the National Assembly Education and Life-Long Learning Minister responsible for all aspects of education, training and lifelong learning. Educated at Malvern Girls’ College, Birmingham University and the University of Wales, Jane has taught English, Drama and Physical Education. She is also an experienced youth worker and former Cardiff City Councillor. She was a member of the Arts Council for Wales and its Lottery Board, and Head of Social Affairs at the Welsh Local Government Association before her election to the Assembly. Jane has had a keen interest in education and youth work and is enjoying the challenges of the Education and Life-Long Learning portfolio.

Stephen Gorard holds the Anniversary Chair in Educational Studies at the University of York (United Kingdom), and directs the Centre for Research into Equity and Impact in Education. He is currently leading an Economic and Social Research Council (ESRC)-funded project promoting the use and understanding of randomised controlled trials in public policy (http://trials-pp.co.uk/), and was the originator of the ESRC’s Research Capacity-building Network. He has published widely about the research process in social science, but his substantive work focuses on issues of equity, especially in educational opportunities and outcomes, and on the effectiveness of educational systems. Recent books include “Teacher supply: the key issues”, “Adult learning in the digital age”, “Overcoming the barriers to higher education”, and “Schools, markets and choice policies”.

David Gough is Professor of Evidence Informed Policy and Practice and Director of the Social Science Research Unit (SSRU) and its Evidence for Policy and Practice Information and Coordinating (EPPI) Centre, Institute of Education, University of London, United Kingdom. Previously he worked at the University of Glasgow and Japan Women’s University. He directs the Methods for Research Synthesis node of the ESRC National Centre for Research Methods Node and research projects for the Department of

BIOGRAPHY – 179


Education and Skills, the Teacher Training and Development Agency, the Social Care Institute of Excellence, and the Department for Work and Pensions. Dr. Gough is editor of the journal Child Abuse Review and associate editor of the journal Evidence and Policy.

Rebecca Herman, a principal research scientist at American Institute for Research (USA), specialises in setting standards for the quality of educational research and reviewing research based on those standards. As the project director for the What Works Clearinghouse, she is responsible for the US Department of Education’s flagship project to identify effective educational programmes and practices. Dr. Herman was project director of the Educators’ Guide to Schoolwide Reform. She provided congressional testimony and many invited presentations on this and related work. Dr. Herman holds an M.A. and Ph.D. in sociology from Johns Hopkins University.

Maria J.A. van der Hoeven is the Minister of Economic Affairs (Netherlands). Maria J.A. van der Hoeven was born in 1949. She was trained as a primary teacher and taught at schools of home economics and junior secondary commercial education. Thereafter she was head of the Adult Commercial Vocational Training Centre in Maastricht and of the Limburg Technology Centre. From 1991 to 2002 Ms. Van der Hoeven was a member of the House of Representatives for the Christian Democratic Alliance (CDA). She has held a variety of social and cultural posts. Ms. van der Hoeven served as Minister of Education, Culture and Science from 2002 until February 2007. She was appointed as Minister of Economic Affairs in early 2007.

David Hogan is currently Professor and Dean of the Centre for Pedagogy and Practice at the National Institute of Education, Nanyang Technological University in Singapore. Between 2004 and 2006 he was Vice Dean for Research at CRPP. Prior to that he was Professor of Education at the University of Tasmania in Australia, and before that he held appointments as Assistant and Associate Professor at the University of Pennsylvania in Philadelphia. He completed his PhD in the history of education at the University of Illinois in 1979. His current research interests focus on the intersections between research, policy and practice, pedagogical theory, curriculum theory and design, the design of knowledge management of innovation systems in schools, multi-level and longitudinal modeling of student outcomes, citizenship and education, and education and social theory.

Bill Kilgallon, OBE, has been the Chief Executive of the UK’s Social Care Institute of Excellence since 2003. Prior to that he was Chief Executive of St Anne’s Community Services from 1978 to 2002, an organisation he founded in 1971, which works with single homeless people and people with learning disabilities, mental health problems and alcohol and drug problems across Yorkshire and the North East. He was Chair of the Leeds Teaching Hospitals NHS Trust, the largest NHS Trust in the country from 1998-2002 and Chair of the Leeds Community & Mental Health Services NHS Trust from 1992-1998. Bill Kilgallon served as a member of Leeds City Council from 1979-1992 where he chaired the Social Services, Housing and Environment Committees. He has led independent inquiries, including one into alleged abuse in a local authority children’s service and one into the management of an NHS hospital for people with learning disabilities.

Hannele Niemi is Professor of Education (1998-) and Vice-Rector for academic affairs at the University of Helsinki, Finland (2003-). She has been Professor of Education in Oulu, Turku and Tampere Universities (1987-1998). She has been a member of the Standing Committee of Social Sciences of ESF, the Council for Society and Culture in the Academy of Finland, and the Scientific Council of the University of Helsinki. She is a Steering Committee member of the British national research programme on teaching and

180 – BIOGRAPHY


learning (TLRP). She was Director of the Finnish national research programme “Life as Learning” 2002-2006. Dr. Niemi has been Chair or a researcher in many national and international evaluation projects for development of educational research and teacher education. Her main research interest areas are teachers’ professional development, moral education and technology-based learning environments.

Johnny Nilsson is the Former Secretary of State for Education in Sweden.

Andrew Pollard is Director of the Economic and Social Research Council’s Teaching and Learning Research Programme (www.tlrp.org), the UK’s largest coordinated initiative for educational research. As a teacher, his career started in Yorkshire primary schools and he has worked in teacher education or research at Oxford and Bristol Polytechnics and the Universities of the West of England, Bristol, Cambridge and London. He is presently based at the Institute of Education London. Andrew Pollard has published widely, including work on longitudinal ethnography and analysis of social factors in teaching and learning, learner perspectives, and resources for teacher education and school practitioners. He is at present working on an analysis of learning experiences through secondary education.

Rien Rouw is senior policy advisor at the Dutch Ministry of Education, Culture and Science (Department for General Strategic and Economic Advice). He is secretary of the Knowledge Chamber.

Tom Schuller is Head of the Centre for Educational Research and Innovation (CERI), OECD, Paris. Formerly Dean of the Faculty of Continuing Education and Professor of Lifelong Learning at Birkbeck, University of London, his latest books are The Benefits of Learning: The Impact of Education on Health, Family Life and Social Capital (RoutledgeFalmer, 2004) and International Perspectives on Lifelong Learning (edited with David Istance and Hans Schuetze, Open University Press, 2002).

Hans Stegeman is senior policy advisor at the Dutch Ministry of Education, Culture and Science (Department for International Policy). He is member of the OECD’s Education Policy Committee.

Charles Ungerleider is Director of Research and Knowledge Mobilisation for the Canadian Council on Learning. From 1998 until 2001, Dr. Ungerleider served as Deputy Minister of Education for the Province of British Columbia, Canada. Prior to this he was Associate Dean for teacher education (1993-1998) at the University of British Columbia. Dr. Ungerleider has studied and written about educational policy and governance, student assessment, inter-group relations, and the impact of media on Canadian society. His most recent book Failing Our Kids: How we are ruining our public schools provides a critical analysis of the state of public schooling in Canada, the key part schooling plays in fostering Canadian values, and how public schools are treated by parents, professionals, and politicians.

Jerzy Wiśniewski is a consultant in education, and public administration and an expert of the Center for Social and Economic Research (Poland). From 2003-2006 he served as head of Strategy and Structural Funds of the Ministry of Education. He was also Director General of the Polish Ministry of National Education at the time of launching the reform of the education system, as well as the head of the International Department of the Ministry of Education and project manager in the Foundation for Public Administration Development. He was a member of the CERI/OECD Governing Board as well as the OECD team reviewing the educational system in Lithuania, advised the Ukrainian Ministry of Education on the reform of the system, and led the team reviewing the VET system in Croatia (with the European Training Foundation).


Also available in the CERI collection

Understanding the Brain: The Birth of a Learning Science 330 pages • June 2007 • ISBN: 978-92-64-02912-5 Demand-Sensitive Schooling? Evidence and Issues 146 pages • November 2006 • ISBN: 978-92-64-02840-4 Think Scenarios, Rethink Education 200 pages • April 2006 • ISBN: 978-92-64-02363-1 Personalising Education 128 pages • February 2006 • ISBN: 978-92-64-03659-8 Students with Disabilities, Learning Difficulties and Disadvantages – Statistics and Indicators 152 pages • October 2005 • ISBN: 978-92-64-00980-9 E-learning in Tertiary Education: Where do We Stand? 290 pages • June 2005 • ISBN: 978-92-64-00920-5 Formative Assessment – Improving Learning in Secondary Classrooms 280 pages • February 2005 • ISBN: 978-92-64-00739-3 Quality and Recognition in Higher Education: The Cross-border Challenge 205 pages • October 2004 • ISBN: 978-92-64-01508-6 Internationalisation and Trade in Higher Education – Opportunities and Challenges 250 pages • June 2004 • ISBN: 978-92-64-01504-3 Innovation in the Knowledge Economy – Implications for Education and Learning Knowledge Management series 96 pages • May 2004 • ISBN: 978-92-64-10560-3

www.oecdbookshop.org

TABLE OF CONTENTS – 5


Table of Contents

Executive Summary............................................................................................................................... 9 PART ONE: SETTING THE STAGE: THE EVIDENCE AGENDA AND METHODOLOGICAL ISSUES

Chapter 1. The Evidence Agenda....................................................................................................... 15 by Tracey Burns and Tom Schuller

Part One: Setting the Stage: The Evidence Agenda and Methodological Issues ............................... 15 Part Two: Mediating the Research/Policy Interface: The Role of Brokerage Agencies .................... 26 Part Three: Evidence-based Policy Research in Practice: Examples from the Field ......................... 28 Part Four: The Politicians’ Perspective.............................................................................................. 29 Concluding note ................................................................................................................................. 30 References .......................................................................................................................................... 30

Chapter 2. What Counts and What Should Count as Evidence...................................................... 33 by Thomas Cook and Stephen Gorard

Introduction ........................................................................................................................................ 33 Thomas Cook’s propositions.............................................................................................................. 34 Stephen Gorard’s propositions ........................................................................................................... 40 Agreements and disagreements.......................................................................................................... 43 References .......................................................................................................................................... 46

PART TWO: MEDIATING THE RESEARCH/POLICY INTERFACE: THE ROLE OF BROKERAGE AGENCIES

Chapter 3. What Works Clearinghouse, United States.................................................................... 53 by Robert Boruch and Rebecca Herman

The What Works Clearinghouse and embodiments of science.......................................................... 54 Assumptions and prospects ................................................................................................................ 55 Operating principles ........................................................................................................................... 55 Contemporary history......................................................................................................................... 56 The WWC’S products ........................................................................................................................ 56 The intended consumers and their use of WWC products ................................................................. 58 The WWC topics and workflow......................................................................................................... 58 Concluding remarks ........................................................................................................................... 60 References .......................................................................................................................................... 60

6 – TABLE OF CONTENTS


Chapter 4. The Evidence for Policy and Practice Information and Co-ordinating (EPPI) Centre, United Kingdom ..................................................................................................................... 63 by David Gough

Aims and function .............................................................................................................................. 63 Methods.............................................................................................................................................. 64 Issues .................................................................................................................................................. 68 References .......................................................................................................................................... 69

Chapter 5. The Iterative Best Evidence Synthesis Programme, New Zealand .............................. 71 by Adrienne Alton-Lee

The Iterative BES approach to knowledge brokerage........................................................................ 72 Fit-for-purpose synthesis methodology.............................................................................................. 72 BES development guidelines ............................................................................................................. 72 Rationale for a collaborative approach across policy, research and practice ..................................... 73 Iterative processes of stakeholder engagement in BES development ................................................ 74 Strategy for use .................................................................................................................................. 74 Brokerage from a policy agency: constraints and opportunities where there is an evidence gap ...... 75 References .......................................................................................................................................... 78

Chapter 6. The Canadian Council on Learning, Canada ................................................................ 81 by Charles Ungerleider

The establishment of the Canadian Council on Learning .................................................................. 81 Organisation and illustrative activities............................................................................................... 82 Opportunities and challenges ............................................................................................................. 85

Chapter 7. The Knowledge Clearinghouse, Denmark...................................................................... 87 by René Bugge Bertramsen

Introduction ........................................................................................................................................ 87 The institutional framework of educational R&D in Denmark.......................................................... 88 New expectations and demands ......................................................................................................... 89 New solutions..................................................................................................................................... 91

Chapter 8. The Knowledge Chamber, Netherlands ......................................................................... 93 by Hans Stegeman and Rien Rouw

Introduction ........................................................................................................................................ 93 The Ministry desires a new way to deal with knowledge .................................................................. 94 Mobilising top-ranking officials to minimise overkill, compartmentalisation and process-fetishism... 95 Modernising government ................................................................................................................... 95 The essence: structural consultation on knowledge ........................................................................... 96 Generating validated knowledge........................................................................................................ 97 Organising creativity.......................................................................................................................... 98

TABLE OF CONTENTS – 7


Chapter 9. The Social Care Institute for Excellence, United Kingdom.......................................... 99 by Bill Kilgallon

Background ........................................................................................................................................ 99 Stakeholders in social care ............................................................................................................... 100 SCIE’s remit..................................................................................................................................... 101 Establishing a knowledge base......................................................................................................... 101 Achieving change............................................................................................................................. 102 Examples of brokerage..................................................................................................................... 103 Conclusion........................................................................................................................................ 104 References ........................................................................................................................................ 105

PART THREE: EVIDENCE-BASED POLICY RESEARCH IN PRACTICE: EXAMPLES FROM THE FIELD

Chapter 10. A Large-scale Policy Research Programme: A Canadian Experience .................... 109 by Satya Brink

A major culture change .................................................................................................................... 109 Policy-driven research demands a long-term view based on desirable outcomes............................ 109 A better understanding of the relation between evidence and policy............................................... 111 Public investment in national data ................................................................................................... 111 A policy-driven consolidated policy research programme............................................................... 112 The construction of the body of evidence ........................................................................................ 113 Policy innovations driven by evidence............................................................................................. 113 Concrete results on behalf of Canadian children.............................................................................. 114 Tests for quality of evidence ............................................................................................................ 115 References ........................................................................................................................................ 116

Chapter 11. Life as Learning – A Finnish National Research Programme.................................. 117 by Hannele Niemi

Life as Learning – The Finnish case of a national research programme.......................................... 117 Co-operation and dissemination throughout the programme ........................................................... 119 Strengths and challenges of the programme..................................................................................... 120 How to add additional value to the programme ............................................................................... 121 The new initiatives – next steps after the programme...................................................................... 122 References ........................................................................................................................................ 123

Chapter 12. The United Kingdom’s Teaching and Learning Research Programme .................. 125 by Andrew Pollard

Aims ................................................................................................................................................. 126 User engagement for relevance and quality ..................................................................................... 127 Knowledge generation by project teams .......................................................................................... 127 Knowledge synthesis through thematic activities ............................................................................ 127 Knowledge transformation for impact ............................................................................................. 127 Capacity-building for professional development ............................................................................. 128 Partnerships for sustainability .......................................................................................................... 129 Conclusion........................................................................................................................................ 130

8 – TABLE OF CONTENTS


Chapter 13. Policy-driven Research and Evidence-based Educational Innovation in Singapore . 131 by David Hogan

Context ............................................................................................................................................. 131 The Singapore core research project ................................................................................................ 133 Core Research Programme............................................................................................................... 133 Specific Focus Projects .................................................................................................................... 136 Evidence-based innovation programme ........................................................................................... 136 Reporting: towards a knowledge management and innovation system............................................ 138 Conclusion........................................................................................................................................ 140 References ........................................................................................................................................ 140

PART FOUR: THE POLITICIANS' PERSPECTIVE

Chapter 14. Research-based Policy-Making: The Need for a Long-term Perspective........................ 145 by Johnny Nilsson

Imbalance between the tempo of policy-making and of research .................................................... 146 The long-term perspective................................................................................................................ 147 Interpretations of research findings are important ........................................................................... 148 References ........................................................................................................................................ 150

Chapter 15. Evidence-based Policy: Yes, but Evidence-based Practice as Well!......................... 151 by Maria J.A. van der Hoeven

Introduction ...................................................................................................................................... 151 Brief outline of the policy context.................................................................................................... 152 More solid knowledge base for national policy ............................................................................... 152 More solid knowledge base for educational practice ....................................................................... 154 In conclusion .................................................................................................................................... 155

Chapter 16. The Importance of Evidence-informed Policy Research in Education A perspective from Wales ................................................................................................................. 157 by Jane Davidson

Introduction ...................................................................................................................................... 157 The Learning Country ...................................................................................................................... 158 Evidence informed policy ................................................................................................................ 158 Areas for further work...................................................................................................................... 164 Working together ............................................................................................................................. 166

Chapter 17. Promoting Evidence-based Policy in Education: The Case of Poland..................... 167 by Jerzy Wisniewski

Background ...................................................................................................................................... 167 Research base ................................................................................................................................... 168 OECD and reform ............................................................................................................................ 169 Effect of EU accession ..................................................................................................................... 171 Agenda-building............................................................................................................................... 172

Biography ........................................................................................................................................... 177

Chapter 1 The Evidence Agenda - OECD.org · 2019-04-24 · Chapter 1 The Evidence Agenda Tracey...

Documents

Transcript of Chapter 1 The Evidence Agenda - OECD.org · 2019-04-24 · Chapter 1 The Evidence Agenda Tracey...