1

Exploring the relationship between system-wide

assessment policies and initial teacher training.

A comparative study of Chile and England.

E. de Padua Njera

Thesis submitted in part-fulfilment of the requirements for the degree of

Master of Philosophy in the University of Cambridge, Faculty of Education. July 2011 Supervised by Carole Waugh Word count: 19,978 words

2

Abstract

The purpose of this research is to explore the relationship between system-wide assessments and initial teacher training comparing experiences from two countries: Chile and England. This comparative study is grounded in the conviction that a good preparation of teachers to deal with system-wide assessments could prevent some of the unintended negative consequences that have been attached to national tests. During this research, 48 official documents and seven websites from assessments agencies and faculties of education were analysed; additionally, seven interviews with members from assessment agencies and academics from faculties of education were held and analysed. Trough the analysis of these documents and interviews it could be observed that the link between system-wide assessments and initial teacher training is weak in both countries; in none of them national tests have a prominent and substantial role during the preparation of new teachers and assessment agencies are not always explicit regarding what is expected from teachers in relation with system-wide assessments. However, the results of this research also highlight some elements that could reinforce a stronger link between system-wide assessments and initial teacher training; among them: agreement between system-wide assessment professionals and academics regarding the importance of considering this type of assessments as part of competences of good teachers and the necessity of introducing system-wide assessment taking into account an underlying learning framework which connects teaching practices and external tests.

This research aims to contribute to the construction of assessment systems that effectively contribute to a better teaching and learning.

Key words: System-wide assessments, classroom-based assessment, initial teacher

training, teachers professional development, curriculum framework. I hereby declare that the sources of which I have availed myself have been stated in the body of the thesis and in the bibliography and that the rest of the work is my own. This thesis does not exceed 20,000 words in length

3

Acknowledgements This thesis could not have been completed without the experience, support and generosity of my supervisor, Carole Waugh. This research was also nurtured by the rich comments made by Sue Swaffield, Allison Fox, Darleen Opfer and Joanne Waterhouse in my previous essays and by interesting conversations with my critical friends from the MPhil. I will also like to thank to Mariano for been an unconditional support and for having the right words at the right time; to Xabier and Gabriel for their patience and collaboration with small but fundamental details. Finally, I thank to my mother for inheriting me her courage to deal with adversity, to Jos for his silent but deep understanding of things and to Roque for raising the sun in the darkest days.

4

Table of contents List of tables and diagrams .................................................................................................................5 List of acronyms and abbreviations................................................................................................6 Introduction ..............................................................................................................................................7 1. Literature review: Framing the relationship between system-wide assessments and the preparation of new teachers .......................................................................................... 11 1.1 Defining system-wide assessments.................................................................................................11 1.2 SWAs and their impact on teachers practice..............................................................................13 1.3 Assessment and initial teachers preparation.............................................................................16 1.4 ITT as a way to bridge the gap between SWAs and teachers practices ..........................18 1.5 The Chilean and English educational context for assessment and teacher training..22 1.6 Framing the research.............................................................................................................................35 2. Methodology...................................................................................................................................... 37 2.1 Epistemological and Ethical issues ..................................................................................................37 2.2 Research design .......................................................................................................................................39 2.3 Data analysis..............................................................................................................................................51 2.4 Managing quality in this research ....................................................................................................52 3. Describing the relationship between SWAs and ITT in Chile and England ........... 55 3.1 How is the relationship between SWAs and ITT in Chile?.....................................................56 3.2 How is the relationship between SWAs and teacher training in England? ....................63 4. Conclusions: Understanding the relationship between SWAs and ITT ................... 71 4.1 General research question: How is the relationship between system-wide assessment policies and the preparation of teachers during their initial teacher training? ..............................................................................................................................................................71 4.2 Specific research question n1: How ITT institutions prepare trainees for system-wide assessments? Is this related to how teachers are trained to deal with CBA?............74 4.3 Specific research question n2: What do departments in charge of system-wide assessments expect from teachers preparation? Is there a similarity with what they expect in relation to CBAs? .........................................................................................................................76 4.4 Specific research question n3: What differences and similarities exist between Chile and England regarding the relation between SWAs and teacher training?...........................78 5. Further reflections on the relation between SWA and ITT........................................... 81 5.1 How the relationship between SWAs and ITT can be enhanced? ......................................81 5.2 Methodological reflections ..................................................................................................................83 5.3. Reflections as a researcher in education......................................................................................85 References............................................................................................................................................... 87 Appendixes ............................................................................................................................................. 94 Appendix 1: Consent form...........................................................................................................................94 Appendix 2: Comparison between different countries in relation with policies for assessment and other characteristics. ...................................................................................................96 Appendix 3: Documents considered in this research......................................................................97 Appendix 4: example of documentary analysis. .............................................................................100 Appendix 5: Description of codes interviews example. ..............................................................103

5

List of tables and diagrams

Tables Table n1: Findings about how trainees learn to assess during ITT...17 Table n2: Evaluations that are part of the English assessment system..31 Table n3: The main features of SWAs in Chile and England...34 Table n4: Interview schedule....43 Table n5: Information gathered from each method45 Table n6: Sample description48 Table n7: Comparison between Chile and England.79 Diagrams Diagram 1: Gap between SWAs and teachers assessment practices and beliefs about assessment.20 Diagram 2: Representation of the research problem.35 Diagram n3: Sampling levels.46

6

List of acronyms and abbreviations ACARA: Australian Curriculum, Assessment and Reporting Authority APP: Assessing Pupils Progress CBA: Classroom-based assessment CEPPE: Center for the Study of Educational Policy and Practice (Centro de Estudios de Polticas y Prcticas en Educacin) DES: Department of Education and Science DfE: Department for Education GCSE: General Certificate of Secondary Education ITT: Initial Teacher Training LGE: General Law for Education (Ley General de Educacin) LOCE: Constitutional Law of Education (Ley Orgnica Contitucional de Educacin) Mineduc: Ministry of Education Chile (Ministerio de Educacin) NAPLAN: National Assessment Program Literacy and Numeracy NCDUS: National Committee for the Development and Use of SIMCE (Comit Nacional para el Desarrollo y Uso de SIMCE) NQT: Newly Qualified Teacher OECD: Organisation for Economic Co-operation and Development Ofqual: Office of Qualifications and Examinations Regulation Ofsted: Office for Standards in Education, Childrens Services and Skills PER: Program for the Evaluation of School Attainment (Programa de Evaluacin del Rendimiento Escolar) PGCE: Postgraduate Certificate of Education QCA: Qualifications and Curriculum Authority QCDA: Qualifications and Curriculum Development Agency QTS: Qualified Teacher Status SIMCE: National Assessment of Education Quality (Sistema de Medicin de la Calidad de la Educacin) SWAs: System-wide assessments TDA: Training and Development Agency for Schools TGAT: Task Group on Assessment and Testing TTA: Teacher Training Agency

7

Introduction Assessment is considered an important lever for change in education. While some types of classroom assessment are shown to have a positive impact on childrens learning (Black & Wiliam, 1998), most countries trying to improve learning outcomes use system-wide assessments (SWAs) as part of their strategies (Barber & Mourshed, 2007; OECD, 2010). With the purpose of improving the quality of education, countries such as the United States, England, Australia and Canada have developed massive external tests that are applied at different levels and with different consequences for schools and their leaders, teachers and students. These tests are known as SWAs and while some of them are designed mainly to support teachers decision-making (as in Canada or Australia) others have external accountability as their main focus (England or the US) (Forster, 2001). In the latter cases, researchers and policy-makers have initiated a discussion about the consequences that SWAs have for teaching. Some argue that SWAs have narrowed the curriculum and constrained teaching; others state that tests have given a clear focus for teachers to enhance students learning. Whether the consequences are positive or negative, teachers must incorporate SWAs into their practice and, as some evidence shows, how this integration is made determines the impact on students learning. According to my professional experience, most of the time this integration is improperly made, preparing pupils to take tests as a decontextualised activity, focusing almost exclusively on teaching them how to answer multiple-choice questions. Underpinning this I firmly believe that there is an implicit (and sometime explicit) resistance from teachers and central governments to considering each other in the development of their own assessment methods; this conviction forms the focus of this research.

8

This belief has been influenced by my own work in the National Assessment System in Chile (SIMCE)1. There I had the opportunity to work with teachers supporting their use of SIMCE results in teaching and learning improvement. When working with teachers I perceived a great demand to know how to use SWAs and several practices and misconceptions hindering the use of these tests as a teaching tool. This perception has been confirmed year after year by the national evaluation of teachers knowledge and practices, where the capacity to use assessment tools is the weakest dimension of teacher performance. Nonetheless, the Chilean government has focused its efforts in augmenting the frequency and number of SIMCE tests without taking account of the capacities that teachers need to develop to benefit from those data. Based on my professional experience and revised research evidence, I believe that SWAs can be a useful tool for teachers to, for example, complement their own diagnostic of pupils or align their practice with national expectations. Based on my experience of developing teaching standards for recently graduated teachers at CEPPE2, I believe that the positive aspects of SWAs have been underestimated by teachers and initial teacher training (ITT) institutions. I also believe that governments simply state that external assessments should be useful for schools, but do not properly support teachers in their application. This situation can be defined as an assessment gap, where SWAs are seen as a decontextualised tool by teachers while governments continue thinking that external assessments can have an impact inside classrooms. Governments in different countries have tried to address this gap by financing programmes for in-service assessment training. The costs are enormous and a significant amount of the training time is dedicated to changing certain beliefs that teachers have about assessment (Gardner et al., 2010; Winterbottom et al., 2008). Given that SWAs are far from disappearing, how teachers are trained during their first years could make a difference to how they use these tests. To investigate how 1 SIMCE is the Chilean assessments system that assesses all schools in grades 4, 8 and 10. It depends from the Ministry of Education (more information at www.simce.cl). 2 CEPPE is Center for the Study of Educational Policy and Practice. This centre was mandated by the Ministry of Education to develop standards for recently graduated teachers (more information at www.ceppe.cl).

9

teachers are prepared for SWAs seems to be a natural step; however, this topic has not been frequently considered in educational research. The relationship between SWAs and ITT is the main topic of this research. To explore this issue, I will compare the Chilean assessment system with another national system having a stronger tradition and more systematic efforts to link external assessments with classroom-based assessments (CBAs), with the conviction that this link could facilitate a link between SWAs and ITT. I have selected Englands assessment system because it has a well-known trajectory in relation to SWAs, because, just as in Chile, despite centrally led efforts to support teachers in this regard, assessment is one of the weakest aspects of in-service teachers (Ofsted, 2010) and because I am currently studying there. Inside both educational systems, I will explore how teachers are supposed to use national assessments and how this is linked to the preparation of trainee teachers in SWAs. The research questions that guide this investigation are: General research question: How is the relationship between system-wide

assessment policies and the preparation of teachers during their initial

teacher training? Specific research questions: 1. How do ITT institutions prepare trainee teachers for SWAs? Is this related to how teachers are trained to deal with CBA? 2. What do departments in charge of SWAs expect from teachers preparation? Is this related to what they expect from CBA? 3. What differences and similarities exist between Chile and England regarding the relationship between SWAs and ITT? It is necessary to state that even when I recognise the SWAs challenges and weaknesses, this research does not focus on them but on how ITT can help teachers to take advantage of this type of assessment.

10

My research adopts a qualitative approach and is based on an exploratory design. Its main methods are documentary analysis and interviews. Regarding ethical issues, it is necessary to declare that, since I worked in the SIMCE, I personally knew some of the interviewees; hence strategies were implemented to guarantee that my own experiences had a limited impact on my approach to interviewees and on the analysis. This thesis is divided in five main sections. The first presents the literature framing the relationship between SWAs and ITT. This section deals with research conducted on the impact of SWAs on teaching and on how teachers are prepared to assess their students; in addition, key contextual information regarding Chile and England is included. The second section describes the methodology employed in this research, including epistemological and ethical issues. The results of this research are contained in section three. In the fourth section, the results are analysed in order to answer this researchs questions. In section five further reflections are presented, including aspects for consideration to enhance the relationship between SWAs and ITT and some methodological reflections.

11

1. Literature review: Framing the relationship between system-

wide assessments and the preparation of new teachers A fundamental part of any research is to revise and analyse background information supporting its conduction. Through the literature review, the researcher set the limits of the investigation and the contribution it can provide to a certain knowledge field (Ary, Jacobs, Razavieh, & Sorensen, 2009). The following section presents the main research findings and contextual information that can help frame the research. This literature review was structured using a deductive approach grounded in this researchs questions.

1.1 Defining system-wide assessments SWAs are increasingly seen as an agent of reform by governments (Barber & Mourshed, 2007; M. Forster, 2001; OECD, 2010). SWAs refer to one type of educational evaluation that is applied and used at the system level; this does not mean that schools or teachers cannot use this kind of information, but SWAs features and requirements are not defined by them. SWAs are designed to investigate and monitor the health of an education system and to improve students learning by providing information to stakeholders at different levels of the system (Forster, 2001, p. 3). SWAs differ from CBAs mainly because the latter are designed, applied, analysed and reported by teachers and their features are mainly defined inside schools. Different types of assessment can have different purposes. Among the main distinctions are formative and summative purposes. This distinction has been discussed by Newton (2007), arguing that the terms belong to qualitatively different categories [and] to attempt to identify characteristics that distinguish them within a single category is to make a category error (p. 156). However, it is common to find articles making that distinction stating the negative

12

consequences of summative assessments on teaching and learning and highlighting the positive aspects of formative assessment (Black & Wiliam, 2005; Boston, 2002; Harlen & Deakin Crick, 2002; James, 2008); in these researches it is also common to associate SWAs with summative purposes. This academic perspective could be at the base of teachers beliefs who consider SWAs to be a less desirable practice. Some authors claim that SWAs can accomplish both purposes. Forster (2001) states that they are used for formative and summative purposes, that is: improvement and accountability. The first purpose can be achieved, for example, by monitoring student performance from different geographic, gender and ethnic backgrounds to ensure that all students enjoy equal access to education; on the basis of these data, additional resources may be allocated to programmes targeting a particular subgroup of students. The second purpose can be achieved when publishing assessment results and establishing consequences for school according to those results. Nevertheless, other researchers argue that it is not possible for one assessment system to effectively accomplish more than one purpose: the fact that a system which is fit for one purpose will not necessarily be fit for all purposes is a fundamental consideration (Newton, 2007, p. 149). This research is focused on SWAs with the conviction that this type of assessment, when designed and used properly and fruitfully by teachers, can contribute to the improvement of teaching and learning. The following section describes the opportunities and challenges that SWAs face to effectively help teachers to support pupils learning.

13

1.2 SWAs and their impact on teachers practice At one extreme, SWAs are considered the most straightforward and noticeable way to push for change in education: Compared with reforms such as targeting instructional time, professional development for teachers, and reducing class sizes, state assessment programs are relatively inexpensive. The assessments also can be mandated (unlike changes in classroom practice), can be rapidly implemented, and have a public visibility (Linn, 1995, p. 5). In other extreme, SWAs are implemented as a complement for other policy strategies with the clear purpose of monitoring learning outcomes and with the conviction that used alone they will not improve learning (for example, see the case of NAPLAN tests in Australia in ACARA (2010)). To find equilibrium between these two extremes is a hard task. In countries such as Chile, the United States and England, SWAs are primarily used as accountability measures for schools. In these cases, while the value of SWAs for such purpose is clear, it is equally clear that they are less useful for the purpose of supporting teachers in recognising pupils needs (Committee on Assessment in Support of Instruction and Learning, 2003). In that sense, simply implementing an assessment system is not a guarantee for educational improvement. Despite some well-documented negative consequences of SWAs, some researchers have acknowledged that the context in which these tests are applied can determine their effect on teaching: there is a good deal of ambiguity about the effects of testing. The same test may lead to different consequences in different circumstances (Firestone, Schorr and Monfils, 2004, p. vii). The following section explores the impact of SWAs on teaching.

14

1.2.1 SWAs as an obstacle for good teaching The negative consequences of SWAs on teaching and learning have been widely researched and documented. These appear especially when sanctions are attached to assessment results (Forster, 2001). In that sense, it is important to highlight that negative consequences are associated with the high stakes attached to assessment outcomes rather than with the tests themselves (Darling-Hammond, 2003). One of the commonest negative consequences of SWAs is that teachers start to emphasise the knowledge and skills being tested, uncovering certain areas of learning not considered in testing and, consequently, narrowing the curriculum (Forster, 2001; Gipps, Brown, & McAlister, 1995; Whetton, 2009); teachers also adopt a teaching style that emphasises the transmission of knowledge (Harlen & Deakin Crick, 2002; McNeil, 2000). Other negative consequences are related to schools selecting pupils to take the tests by manipulating admissions, dropouts, and pupil classifications (Figlio & Getzler, 2002). Negative consequences for pupils have also been reported: testing generates anxiety among students (Reay & Wiliam, 1999); and decreases their motivation levels, impacting negatively on their self-esteem (Harlen & Deakin-Crick, 2002). Finally, students spend too much time learning techniques that help them do nothing more than cope with specific test formats (McNeil, 2000). The consequences mentioned above can be classified inside the phenomenon known as teaching to the test. This phenomenon refers to teachers doing something special to help students do well on a test, often without helping them to better understand the underlying subject matter (Firestone et al., 2004, p. 2). This phenomenon ends in a decontextualized test preparation, because it is a special activity disconnected from regular lessons and focused on the test itself. According to Firestone et al. (2004), preparing pupils to do well in tests is not necessarily bad teaching practice; teachers with a proper pedagogical content knowledge and assessment skills can take advantage of external tests and merge test preparation into good teaching practice; in these cases, teaching to the test is not a decontextualised practice. Most decontextualised teaching to the test is

15

caused by a lack of teacher preparation to use SWA information (Firestone et al., 2004) and by the perception of external tests as an extra workload (Whetton, 2009). 1.2.2 SWAs as a tool for good teaching As shown in the previous section, teachers can transform their practices in ways that block pupils learning. However, in response to certain types of test, some teachers are encouraged to explore more intellectually challenging practices and integrate them into their classrooms throughout the year (Firestone et al., 2004, p. 159). For example, tests with open-ended questions and a focus on high-order thinking skills may encourage more challenging forms of teaching (Firestone et al., 2004). Moreover, tests can make teachers explore curriculum areas that were traditionally left behind and may prompt teachers to consider more challenging content and activities that provide an opportunity for students to develop a better understanding (Gipps et al., 1995). Additionally, the introduction of external criteria to judge the quality of pupils learning allows teachers to base their judgements on objective information, avoiding the influence of social class, gender and racial bias (Gipps et al., 1995). Despite the positive teaching outcomes of SWAs, research suggests that no test can support and improve teachers practice without adequate professional preparation (Firestone et al., 2004; Gardner, Harlen, Hayward, Stobart, & Montgomery, 2010). Hence the balance between pressure and support that, according to Fullan (2001), is fundamental when trying to make changes in education: Successful change projects always include elements of both pressure and support. Pressure without support leads to resistance and alienation; support without pressure leads to drift or waste of resources (p. 60). Teachers professional development can support their accomplishment of SWA demands, but most of these development activities are not designed to help teachers fundamentally change their worldviews or effectively synthesise new ideas into instructional practice (Firestone et al., 2004). It is plausible considering

16

that it can be more effective to prepare teachers using SWA during ITT since, according to Stiggins (2005), it is more expensive and difficult to change entrenched practices and beliefs than those that are not yet established. The following section explores the research on ITT factors related to assessment. 1.3 Assessment and initial teachers preparation ITT should be a focus of improvement when implementing SWAs; according to Darling-Hammond (2003), investments in teaching quality are required before changes in assessment strategies result in more challenging and effective instruction for currently underserved students. The research about how new teachers are trained to assess is still in an embryonic stage. Stiggins (2005) and Spitzer et al. (2010) have developed recommendations for preparing trainee teachers to assess students; however, these researchers do not consider SWAs as a topic. A more developed research field can be found in studies about student teachers perceptions and beliefs about assessment (Gardner et al., 2010; Graham, 2005; James & Pedder, 2006; Winterbottom et al., 2008), and about what and how new teachers learn about assessment (Campbell & Evans, 2000; DeLuca & Klinger, 2010; Graham, 2005). Some important findings when investigating how teachers learn to assess during ITT are summarised in Table n1:

17

Table n1: Findings about how trainees learn to assess during ITT Trainees tend to treat assessment and testing as synonymous (Graham, 2005). Perceptions and beliefs There is a value/practice gap in relation to the importance that trainees attribute to a performance/summative orientation in assessment practices; they value less this type of practices than they implemented them (Winterbottom et al., 2008). The most powerful influence in the way that trainees learn how to assess is their mentor, especially when he or she shows them how to assess using practical settings (DeLuca & Klinger, 2010; Graham, 2005). Butterfield et al. (1999) also highlight the role of mentors, but as a potential source of increasing mistaken preconceptions about assessment. According to DeLuca and Klinger (2005), mentors understanding about assessment may be incomplete, outdated, and biased (p.434). Among the main weaknesses of trainees in relation to assessment is the difficulty to establish learning goals and assess pupils in relation to those goals (Graham, 2005).

What and how they learn to assess?

New teachers tend to fail in understanding concepts such as validity and reliability; however, these concepts have not proved to be an esential part of teachers preparation to assess pupils (Campbell & Evans, 2000). Source: Personal elaboration from Campbell & Evans, 2000; DeLuca & Klinger, 2005; Graham, 2005 and Winterbottom et al., 2008. But how do you prepare new teachers to effectively use SWAs? A possible answer emerges from a different field to assessment itself. Proper use of SWAs requires a deep understanding of the content that has to be taught as well as how to teach it: content and pedagogical content knowledge, as it has been popularised by Schulman (1986). According to Firestone et al. (2004), this knowledge will significantly influence the ways that teachers teach to the test. Teachers who know the aims of the discipline they teach are able to take advantage of SWA information in ways that can significantly contribute to pupils learning. This knowledge is fundamental in a climate where the autonomy of teachers is challenged by government official initiatives and requirements (Cohen, Wyse, Manion, & Morrison, 2010).

18

1.4 ITT as a way to bridge the gap between SWAs and teachers practices The relation between SWAs and ITT is a new topic for research; however, it can be understood by looking at the relation between SWAs and CBAs. This relationship has been documented from two perspectives: (1) some researchers describe a gap between teachers beliefs on assessments oriented to measure pupils achievement and what teachers actually do in their classrooms (James & Pedder, 2006; Winterbottom et al., 2008); (2) others describe a gap between the purposes, uses and frameworks of SWAs and teachers assessment practices (Committee on Assessment in Support of Instruction and Learning, 2003; Fernandes, 2009; Forster & Masters, 2004; Maier, 2009). Regarding teachers beliefs, James and Pedder (2006) found that teachers in their study used assessment practices that were linked more to summative purposes, even when the same teachers thought these practices were less useful in enhancing pupils learning. They believed they were acting in pursuit of ends they clearly do not view as worthwhile or desirable (p. 132). Winterbottom et al. (2008) drew similar conclusions, but applied to trainee teachers. When observing the relationship between SWA purposes and teachers assessment practices, researchers have found that: - Even when SWAs can be designed by governments with formative purposes, their results are used and perceived by teachers from a summative perspective and they do not consider assessment information useful for their practice (Maier, 2009). - SWA information is not always as clear for teachers (Committee on Assessment in Support of Instruction and Learning, 2003). - Assessment results from SWAs are mainly employed for grading, selecting and certifying rather than for feedback to improve learning (Fernandes, 2009). - There are mismatches between what is taught and assessed inside classrooms and what is tested through SWAs, because of the tests inability to assess all parts of the curriculum or because of a lack of teacher preparation in relation

19

to certain topics (Committee on Assessment in Support of Instruction and Learning, 2003; Forster & Masters, 2004). - This gap is also manifested in the complexity of the skills needed to use SWAs and the professional development provided for teachers on the topic; and the data provided by many large-scale assessments and teachers informational needs (Firestone et al., 2004). According to the first approach for conceptualising the assessment gap, it could be inferred that some teachers practices are closer to a summative perspective since teachers tend to apply this approach to their practice more than they would want to. However, according to the second approach, SWAs are not coherent with classroom assessment practices. These findings reinforce a bigger gap between teachers beliefs and how SWAs are designed and implemented. In other words, teachers believe that summative assessments are not desirable, but because of external pressures they are forced to implement them. One can argue that teachers assessment practices could be more aligned to summative assessments such as SWAs but, on the contrary, there is also a gap between teachers assessment practices and SWAs. The following diagram is a schematic way to visualise the assessment gap between SWAs, teachers assessment practices and beliefs.

20

Diagram 1: Gap between SWAs and teachers assessment practices and beliefs about assessment

Source: personal elaboration. As illustrated, teachers do not perceive SWAs as a fundamental tool for their practice. But why should a gap between these elements be considered a problem? SWAs and CBAs should be grounded in the same principle: to support learning. To accomplish this purpose each assessment type uses different strategies (for example, standardised tests in the case of SWAs, and daily feedback through classroom interaction in the case of CBAs), but coherence between them is fundamental for a synergic action. A weak alignment between SWAs and CBAs can increase the negative consequences of external tests (Forster & Masters, 2004). According to Forster and Masters (2004), a coherent assessment system will be one that places the assessment work of classroom teachers and system administrators within the same conceptual framework (p. 64). By this conceptual framework, the authors understand the definition of learning outcomes that are valued across the system and that should guide teaching, assessment, school leaders decision-making, etc. A problem arises when, because of reliability an applicability issues, the alignment between SWAs and this conceptual framework weakens. According to Forster (2001), the professional development of teachers also needs to be considered for a coherent and synergic assessment system

Teachers beliefs about assessment SWA Teachers assessment practices Assessment gap as described by Committee on Assessment in Support of Instruction and Learning, 2003; Fernandes, 2009; Forster & Masters, 2004; Maier, 2009.

Assessment gap as described by James & Pedder, 2006; Winterbottom et al., 2008.

21

(Forster, 2001). Consequently, ITT could play an important role in developing a coherent assessment system. In summary, the relation between SWAs and ITT forms a new research topic. However, it is possible to frame this topic by gathering research evidence both from the study of the relation between SWAs and teaching and the study on how new teachers learn to assess their students. From the revision of this evidence, it is possible to see that the relationship between SWAs and ITT is characterised by a weak consideration of SWAs; this can be explained by a negative perception of SWAs and a tendency to use them more for accountability purposes than for supporting teachers practice and by a gap between SWAs and teachers assessment practices. However, this relationship is fundamental when designing an assessment system that effectively contributes to improving students learning. The next section is intended to contextualise the relationship between SWAs and ITT in the countries where this study is based: Chile and England.

22

1.5 The Chilean and English educational context for assessment and teacher

training

The comparison between two different assessment systems of two countries is an essential aspect of this research. To make this comparison it is fundamental to consider the context in which these systems were developed. This section presents contextual information that aids understanding of the relations between SWAs and ITT in Chile and England.

1.5.1 Contextualising Chilean national assessment system

Educational System Overview

The system is based on a per-pupil public subsidy. The amount of funding that public and subsidised schools receive depends on student attendance. Chilean school administration is shared by the public and private sectors. This administrative framework was implemented during the military regime in the 1980s (Meckes & Carrasco, 2010). The Ministry of Education is responsible for securing student access to schools and the quality of education throughout the system (Chilean Government, 2009). History of National Assessments SIMCE has its origin in the National Test3 in the 1960s. Its main purpose was to contribute to the national curriculum development and resources allocation (Eyzaguirre & Fontaine, 1999; NCDUS, 2003). In 1982 the Program for the Evaluation of School Attainment (PER)4 was established. Neither National Test nor PER published any data about schools or pupils; their results were intended to be 3 Prueba Nacional 4 Programa de Evaluacin del Rendimiento Escolar - PER

23

used by teachers and authorities to improve education (Eyzaguirre & Fontaine, 1999; NCDUS, 2003). In 1988 the Measurement System of Education Quality5 (SIMCE) was established and has been operating since then (Meckes & Carrasco, 2010). According to the education law proclaimed by the military regime (Constitutional Law of Education - LOCE6), this system should be operated from the Ministry of Education and assess students attainment according to the national curriculum. The law stated that results have to be published by school and geographical region (Chilean Government, 1990). However, until 1994 results were delivered directly to schools. In 1995 SIMCE results were published for the first time. Along with the publication of results, SIMCE started to be used for establishing teachers incentives7 and to allocate resources in schools with low results8 (NCDUS, 2003). SIMCE started to accomplish the original purpose of its predecessor (PER): to support a voucher system. This system was established as part of the profound reforms of the military regime; it implies that the state pays schools directly based on student attendance and parents are supposed to choose the best schools according to test results, leaving bad schools without pupils (Carnoy, 1998). Due to the publication of results in a system where schools have to compete for students, SIMCE becomes a high-stakes assessment system. SIMCE results were increasingly the object of schools, policy-makers and mass-medias attention, with consequent criticism of the negative consequences of external tests, causing mistrust among teachers. SIMCE started to be perceived mainly as a tool for external control (Meckes & Carrasco, 2010). In 2003, in answer to criticisms, the Ministry of Education called a national committee to review SIMCE (National Committee for the Development and Use of SIMCE - NCDUS); this committee included representatives from research centres, 5 Sistema de Medicin de la Calidad de la Educacin SIMCE 6 Ley Orgnica Constitucional de la Educacin - LOCE 7 National System for the Evaluation of Performance / Sistema Nacional de Evaluacin de Desempeo SNED 8 900 Schools Program / Programa de las 900 escuelas P900

24

government and opposition party think tanks, school leaders, policy-makers, and the teachers union. The committees recommendations were focused on refining SIMCE for accountability (i.e. increasing testing frequency and incorporating value-added measures) and, at the same time, improving SIMCE characteristics to provide teachers with more comprehensible, useful information (i.e. changing from a norm-based to a criterion-based evaluation through the incorporation of attainment levels) (NCDUS, 2003). The current system Nowadays, SIMCE is no longer determined by the LOCE; instead, a new legal framework was developed: the General Law of Education (LGE)9. This law directs the transfer of SIMCE from the Ministry to an independent and non-departmental institution (Chilean Government, 2009). Despite changes in the legal framework, SIMCE continues to be externally applied every year to fourth graders (in Language, Maths, Science and Social Studies) and every two years for eighth (in English, Maths, Science and Social Studies) and tenth graders (in English and Maths). Additionally, SIMCE complements its census-based tests with sample-based surveys; this is the case for Physical Education. However, these types of test are exceptional: in previous years, sample-based writing and English tests were applied just once (SIMCE, n.d.). SIMCE results are reported to schools through the SIMCE website and printed reports individualised for each school. These reports include extended explanations to interpret results and a workshop for teachers to help them analyse school results. This workshop was linked to a national working day exclusively oriented to analysing SIMCE results but this initiative was discontinued in 2009. One aspect that has been widely recognised in SIMCE is the quality of its dissemination and use of its results; it has also been recognised for its stability, coherence and coordination (Vegas & Petrow, 2008). This feature is confirmed in PISA 2009 analysis, where Chile falls between countries with a high use and 9 Ley General de Educacin-LGE

25

prolific communication of external assessment information by schools (OECD, 2010). Before initiatives about attainment levels and the SIMCE working day were implemented, teachers considered that SIMCE information was not sufficiently clear (OECD, 2004). Among studies made after the implementation of these initiatives, there is a tendency to observe an improved perception of the quality of SIMCE communication strategy (Taut, Corts, Sebastian, & Preiss, 2009). According to OECD (2004), in highly socially segmented educational systems like that of Chile, there is tension between the publication of school-level learning outcomes and their use by teachers, especially in teachers working in lower-income areas, who perceive and interpret SIMCE as being unfair, despite efforts to consider socio-economic backgrounds when reporting results. The limited use of data by teachers can also be explained by their lack of preparation in assessment generally (Manzi et al., 2011), and by the fact that the link between SWAs and CBAs has not been sufficiently explicit in national policies. The recent efforts of the SIMCE and Curriculum Unit to develop a common framework of standards for reporting national test results and guiding school-based assessment (attainment levels and progress maps10) reveal a promising initiative to build a bridge between SWAs and teachers practices. In that sense, SWAs can contribute significantly to school practices by providing valuable information, but because this feedback does not occur spontaneously, special efforts must be made to facilitate the use of this information and build a bridge of meaning between school and teacher practices and what the assessment system evaluates (Meckes & Carrasco, 2010, p. 246). The use of SIMCE data for accountability purposes has increased with the introduction of the Preferential Subsidy in 2008, and will increase further when a new law on school quality assurance is implemented. Both laws introduce a system of public school classification based on SIMCE scores, which determines, for 10 A progress map is a picture of the path students typically follow as they learn. It is structured according to vertical levels that describe the skills, understandings and knowledge in the sequence in which they typically develop (Masters, 2002).

26

example, schools level of autonomy in the use of resources (Meckes & Carrasco, 2010). 1.5.2 Contextualising Chilean initial teacher-training strategy The professionalisation of teachers started after the foundation of the first school for teachers (Escuela Normal) in 1842 (Nuez, 2007). In 1967 the government ordered that prospective teachers should have finished their secondary education before starting ITT; this was the beginning of the end for Escuelas Normales and the start of universities as ITT providers (Cox & Gysling, 1990). In 1981 the educational system was decentralised by the military government and public schools switched their dependence from the state to the municipalities. The same happened to teachers, their contracts becoming the same as that of any private employee. In 1990, with the recovery of democracy, the state tried to reverse some of the labour damage suffered in the past decade by implanting new regulations (Experts Committee for Quality in Education, 2010). According to the current educational law (Ley de Educacin General LGE), in order to become a primary teacher it is necessary to have a professional diploma in line with this level of education. To become a secondary teacher one is required to have a professional diploma for secondary education in the relevant subject (Chilean Government, 2009). In the last five years there has been a wide consensus about the necessity to improve the quality of ITT. This consensus has been nurtured by the application of a national test for recently graduated teachers (Inicia test11); this test has revealed severe deficiencies among new teachers (Brandt, 2010). As part of this consensus, several committees and researchers have highlighted challenges that must be faced if the quality of teaching is to be improved. For example, Latorre (2006) claims that Chilean teachers first practices are grounded 11 Inicia is a Spanish word referring to something that is beginning.

27

more in their experience as pupils and first school experiences than in their studies at an ITT institution. She also states that there is incongruence between how teachers are prepared and the skills and knowledge needed inside a classroom. Another problem highlighted by Brandt (2010) is that most of the trainees begin their ITT without having achieved the competences expected for secondary education, and that teaching is viewed as a professional option by those whose university entry exam results did not allow them to enter other specialities. The OECD (2004) found that training in pedagogy often remains too theoretical with little or no communication between faculties teaching subject content and university education departments in the case of secondary teacher preparation; this lack of communication can be also found between ITT providers and schools where trainees take their practical courses (it is important to mention that there are no official requirements in relation to the length and content of school work for trainees). Despite these concerns, ITT has expanded significantly. The number of student teachers rose by 150 per cent between 2000 and 2009. This situation has implied a flexibilisation of entry requirements and thus a drop in the quality of trainee baseline skills and knowledge (Experts Committee for Quality in Education, 2010). As a means to face the challenges of ITT, the government implemented the Inicia strategy in 2008. This strategy considers the implementation of three elements: (i) a national evaluation for recently graduated teachers (Inicia tests), (ii) the development of standards to establish a minimum common baseline to guide how new teachers should be prepared and (iii) resources for ITT providers who desire to improve the quality of provision (Brandt, 2010; Mineduc, n.d.). The effects of this strategy have not been evaluated yet.

28

1.5.3 Contextualising the English national assessment system General overview of the educational system in England The English educational system is organised at national and local levels. The Department for Education (DfE) is responsible for planning and monitoring the education services at a national level. The DfE must ensure the provision of education for all children. At the local level, the responsibility for the administration of publicly funded English schools relies on the Local Authorities (LEAs) (Eurydice, 2010). Along with the DfE there are non-ministerial government departments in charge of different aspects of education. There are two departments in charge of national assessments: 1. The Qualification and Curriculum Development Agency (QCDA, formerly Qualification and Curriculum Authority - QCA) is in charge of developing the national curriculum and the assessment aligned to it (QCDA, n.d.). According to the DfE (2010), the role of this department is currently under revision. 2. The Office of Qualifications and Examinations Regulation (Ofqual) regulates the quality of qualifications, examinations and assessments (Ofqual, n.d.). QCDA acts as an awarding organisation for national curriculum tests although it outsources their development, generally to one or more external awarding bodies, and Ofqual regulates them (Isaacs, 2010). Additionally, the Office for Standards in Education, Childrens Services and Skills (Ofsted) is responsible for inspecting educational institutions; when evaluating the quality of provision in a school, Ofsted considers pupils results in national tests (Ofsted, n.d.).

29

History of National Assessments The introduction of a national curriculum in 1988 is considered a milestone in educational policies in England (Wyse & Torrance, 2009). Before, there was no control over primary or secondary curriculums, nor was there a national assessment plan, only a system of qualifications for students between 14 and 16 years old (Isaacs, 2010). The 1988 Education Reform Act set the bases for a national assessment system at the end of each key stage in relation to the attainment targets. Test formats, the use of the results and other details were left to be decided by the Secretary of State for Education; all these aspects have been a matter of political debate until now (Whetton, 2009). The Task Group on Assessment and Testing (TGAT) defined the first guidelines for the development of a national assessment strategy. According to this group, national assessment should accomplish several purposes: formative, summative, evaluative, informative, and for teachers professional development (DES, 1988). In order to meet these purposes, attainment levels were designed. The assumption was that, by linking assessment with learning attainment levels, a system based on professional judgement and diagnostic assessment could be preserved over accountability purposes (Whetton, 2009). The first attempt to design a national assessment was made for Key Stage 1. The main concerns were that teachers and children would devote too many hours to this task. However, this assessment strategy had some positive outcomes related to the effects on the curriculum and on teachers knowledge and skills in assessment (i.e. focus on previously neglected areas of the curriculum such as mathematics and science investigation) (Gipps, Brown, McCallum, & McAlister, 1995). The establishment of Key Stage 3 tests in 1993 was more controversial since they provoked a boycott from teachers. This took the form of legal action in the High Court against a teachers union to prevent the boycott of the tests. The Court considered that teachers cannot be required to undertake the extra-unpaid work

30

of marking the tests and the union won the case. This situation changed the future development of the English assessment system (Whetton, 2009). From this point, the assessment system focused on diminishing the workload of teachers, determining a system that was based more on externally marked tests (Dearing, 1994). From that time, the equilibrium between validity, reliability and manageability was gradually broken to the detriment of validity: tests should not overburden teachers and should be able to fulfil accountability aims (Whetton, 2009). The application of Key Stage 2 tests was less controversial as they were designed considering reliability and manageability issues from the beginning. They were externally marked and based on written tasks (Whetton, 2009). In October 2008 the Key Stage 3 assessments were abolished after an incident that delays the delivery of assessment results to schools; however, the tests are still available for schools, although there are no arrangements for external marking (Higginson, 2010). The accountability function will be accomplished by the General Certificate of Secondary Education (GCSE) at school level and by sample assessments at the national level (Isaacs, 2010). National tests become the main measure of government policies. This situation initiated a target culture where ministers and schools were increasingly worried about the publication of national assessment results (Whetton, 2009). The current system Nowadays, the English national assessment system is based on a mixture of different types of assessment. Contrary to what is observed in Chile, in England teachers must take an active role in some centrally mandated evaluations (Ofqual, n.d.). Following, the Table n2 describes this mixed assessment system:

31

Table n2: Evaluations that are part of the English assessment system. Centrally led assessments: Teacher-led assessments: End of Key Stage 2 tests End of Key Stage 2 tests cover English and mathematics. They are designed to measure pupils knowledge and understanding of the specific subjects or elements covered in that key stage. The tests are externally set and marked. The DfE makes the results of these tests for each school available to the public. Pupils individual results must be reported by schools to parents and Key Stage 3 teachers (Ofqual, n.d.). National sampling Science is sampled at the end of Key Stage 2 using externally marked tests undertaken by Year 6 children in a representative sample of schools. Pupil results in this sample of schools are not used in their external evaluation, for example, by Ofsted inspectors (Ofqual, n.d.). General Certificate of Secondary Education (GCSE) At the end of compulsory education (age 16), the majority of pupils are assessed through external qualifications. These qualifications are based on the National Curriculum and are developed by independent organisations, (awarding bodies), which are regulated by the Ofqual (Eurydice, 2010). GCSEs are available in over forty-five subjects, . GCSEs are developed to cover two tiers of attainment (three for Maths) and the teacher must decide which one is most appropriate for each student. GCSEs use coursework and controlled assessments to grade students (QCA, 2005). Entry-level qualifications are aimed at pupils who, according to teachers judgement, are not ready for GCSEs at the end of Key Stage 4 (Eurydice, 2010).

Practitioner/teacher assessment judgements Judgements are made by teachers about pupils learning in relation to national attainment levels. These judgements must be set in a profile summary for each child and sent to local authorities, the DfE and to parents. The submission of teacher assessment is a statutory requirement for Key Stage 2 and Key Stage 3 schools (Ofqual, n.d.). When making teacher assessment judgements, schools are increasingly using Assessing pupils progress (APP) materials, which is a key element of the Assessment for Learning Strategy. APP aims to: (1) enable teachers to make judgements about their pupils attainment in relation to national standards; (2) provide diagnostic information about the strengths and weaknesses of pupils; (3) enable teachers to track pupils progress over time (DCSF, 2008). Teacher-led tests and tasks at Key Stage 1 Towards the end of Key Stage 1, teachers must provide additional evidence about pupils learning using a mixture of tests and tasks developed by the Qualifications and Curriculum Development Agency (QCDA) in the areas of reading, writing and mathematics. To ensure consistency, local authorities moderate teachers decisions both internally and externally (Ofqual, n.d.).

Source: personal elaboration from Ofqual (n.d); QCA, 2005; DCSF (2008) and Eurydice (2010). These evaluations are being used for a range of purposes along a continuum that ranges from support to pressure. In the support pole, national assessments are used by teachers to guide their decisions. In the pressure pole, national assessments are used to hold the government and schools accountable for pupils

32

learning. Despite the latter purpose, learning outcomes seem to have remained static since 2000 (Wyse & Torrance, 2009). Nowadays, there is a new government and changes are being considered. The government has ordered an independent review of the effectiveness of the current Key Stage 2 tests in order to improve the current system so that parents have the information they need and schools can be properly accountable without feeling that they must drill children for tests (DfE, 2010, p. 41). 1.5.4 Contextualising English initial teacher-training strategy In 1974 professional training for all school teachers became mandatory; since then, graduates from different disciplines have been required to take a one-year postgraduate course of professional training: the Postgraduate Certificate in Education (PGCE)(Eurydice, 2010). The Education Act 1994 created the Teacher Training Agency (TTA), a non-departmental public body with the purpose of raising standards in schools by attracting able people to teaching and improving the quality of teacher training. A fundamental element of the TTAs strategy was to promote employment-based training and flexible routes to becoming a teacher. The Act also empowered schools, accredited by the TTA, to provide courses in ITT. In September 2005 the TTA became the Training and Development Agency for Schools (TDA)(Eurydice, 2010). The introduction of these agencies revealed greater government control over the training of teachers, introducing a national curriculum for teachers and the prescription of standards for newly qualified teachers (NQTs) in the late 1990s. These standards are considered among the most important interventions in the definition of a new teaching profession (Devlin et al., 2011, Eurydice, 2010). The current coalition government supports a stronger role of schools in teacher training (DfE, 2010); however, according to Devlin et al. (2011), this intention is ideologically driven rather than evidence-based since Ofsteds inspection findings

33

indicate a better provision of ITT delivered through higher-education-led partnerships than through school-based training. Certain standards must be met by any professional who wants to work as a teacher; these standards are known as the Qualified Teacher Status (QTS) and they indicate what trainee teachers must know, understand and be able to do in order to start an induction period enabling them to work as teachers. The QTS standards set requirements for teachers in three dimensions (TDA, 2008): Attributes regarding their relationship with children and young people, their attitudes for national policies and frameworks, their capacity for communicating and working with others and their personal professional development. Knowledge and understanding about: teaching and learning; assessment and monitoring; subjects and curriculum; literacy, numeracy and ICT; achievement and diversity; and health and well-being. Skills for planning; teaching; assessing; monitoring and giving feedback; reviewing teaching and learning; creating a learning environment and for teamwork and collaboration. TDA standards also set requirements for ITT providers, including: entry, training and management and quality assurance (TDA, 2008). There are more than 30 routes to become a teacher. The PGCE is an example and it is available at universities and colleges throughout the UK and also through flexible distance learning or by completing a programme of school-centred ITT (TDA, 2008). Entry requirements are the same for all these routes: to have at least a grade C in English and Maths GCSE (for primary teaching a grade C in Science is also required) and a previous academic degree (TDA, 2008). If a higher-education institution is chosen, the candidate must take a course lasting roughly 1 year including 18 weeks in a school for trainees wanting to teach at primary level, or 24 weeks for those wanting to teach at secondary level (on a full-

34

time basis). At the end of the course, trainees must pass the QTS skills tests in Numeracy, Literacy and Information and Communication Technology before they can be recommended for the award of QTS. It is then assumed that this person meets the QTS standards and will be awarded qualified teacher status (QTS) to become a newly qualified teacher (NQT) ready to undertake the induction year. QTS standards enable teachers to teach at state-maintained schools across England and Wales (TDA, 2008). The described national contexts for SWAs and ITT in Chile and England allow the findings of this research to be set in a wider perspective. As can be seen, these countries share certain characteristics such as, for example, a census assessment system with school results that are published or the presence of teaching standards and evaluations framing what teachers are supposed to know and do. Table n3 summarises the main characteristics of SWAs in each country. Table n3: The main features of SWAs in Chile and England Chile England Assessment system characteristics National Assessment System Yes Yes Publication of results Yes Yes Results per pupil No Yes (for schools and parents) External assessments Yes Yes Teacher assessments No Yes Consequences attached to results Yes Yes Official teacher support to analyse external data Yes (report to school, SIMCE workshop) Yes (Raise online) Census tests Yes Yes Sample tests Yes (not systematically) Yes Participation of teachers Marking Marking Teachers assessments Initial teacher training characteristics Teaching standards Yes Yes Multiple routes into teaching No Yes Assessment of recently graduated teachers Yes Yes Assessment of in-service teachers Yes Yes Source: personal elaboration.

35

1.6 Framing the research When looking at the relationship between SWAs and ITT, there is no research evidence that helps to describe ITT role in that relation. Is ITT a factor that helps to bridge the gap between system-wide and classroom assessment or, on the contrary, is ITT helping to deepen this gap? These are the main questions that guide this research. These questions are placed within a conceptual framework that has been nurtured by the arguments and evidence presented in the previous literature review. The following conceptual framework synthesises the main findings described in the literature review and helps to understand the concepts and processes that were considered in the design and analysis stages of this research. Diagram 2: Representation of the research problem

Source: personal elaboration.

36

Diagram 2 illustrates the underlying problem that this research faces. According to this diagram, the gap between SWAs and teacher practices determines that SWAs are not seen as useful tool by teachers and it has been excavated by certain teachers beliefs, weak knowledge and skills in relation to SWAs and excessive workload that hinder teachers tackling SWAs demands. In the counterpart, SWAs have been linked to accountability strategies with a consequent debilitation of their link to the conceptual framework since reliability is reinforced to the detriment of validity; additionally, information from SWAs is not always clear. This gap has different consequences on teaching and learning. A fundamental aspect to bridging this gap is the existence of a conceptual framework (i.e. a national curriculum) and teachers professional development, ITT being the first step to consider. These considerations underline the research questions presented in page 9 of this thesis. To answer those questions, a methodology has been developed. This methodology is described in the next chapter of this thesis.

37

2. Methodology

This section explains the methods and procedures of this research and makes explicit the underpinning epistemological and ethical issues that set the limits of this study. I begin this section by exploring the epistemological and ethical issues on which my study is based. Then, an explanation of the research design is provided along with a description of the methods and procedures used, and the participants and documents considered as sample. I then describe how the data was analysed; the section ends with some cautions and limitations that must be considered when evaluating the validity and possibilities of this researchs findings.

2.1 Epistemological and Ethical issues A research can take different approaches to its objects of study; understanding those approaches is crucial to defining the design of the investigation, how its results should be interpreted and in establishing the limitations of the findings (Flick, 2009). This study is based on a constructionist ontological approach, since the social world is conceived as being constructed through peoples perceptions and reinforced by their interactions (Denscombe, 2010a, p. 119). In this particular case, how SWAs and ITT are connected depends more on how the social context and the interpretations of people involved can shape this relation than on what is explicitly stated in official documents. Regarding its epistemological approach, this research adopts the tradition of interpretivism, where knowledge about the social world is produced, rather than discovered; in that sense, my own influence as a researcher and the personal interpretations of interviewees have a central role in the description of the relation of SWAs and ITT (Denscombe, 2010a). Consequently, as this research is concerned with subjective meanings, a qualitative approach is more suitable than a quantitative one, because qualitative research is

38

a means for exploring and understanding the meaning individuals or groups ascribe to a social or human problem (Creswell, 2008, p. 4). In order to guarantee ethical considerations in this study, the Revised Ethical Guidelines for Educational Research from the British Educational Research

Association (BERA, 2004) were used as a main framework. According to this framework, two main ethical dilemmas need to be considered here. The first dilemma is the protection of the identity of the participants; this is a critical aspect in the case of interviewees from the assessment system in Chile as, contrary to the situation in England, there is only one institution in charge of national evaluations and it is therefore easy to identify interviewees. To solve this problem, the description of the interviewees is made in very general terms, without stating their specific role inside the organisations. Consequently, and as agreed with interviewees, all names were changed, in order to avoid any possibility of identification. The second ethical dilemma is that I was directly involved in the development of some of the Chilean documents analysed in this study. Moreover, I knew some of the interviewees as colleagues; hence it was important for me to establish trustworthiness in their accounts and to find ways of limiting my previous knowledge about them and their work. To deal with this problem, my approach with interviewees was very formal and I clearly explained to them that I would not assume the position of a peer, but a researcher open to gathering the most information possible from each interview. In addition, they had the opportunity to see the final analysis of their interviews to check for any misinterpretations. The generation of codes applied consistently to all documents and interviews also helped to diminish the influence of my preconceptions. Additionally, one of the interviews was re-codified by another researcher (critical friend). Participants permission to record and to confirm their voluntary participations was consented by signing a consent form (see Appendix 1).

39

2.2 Research design This study is grounded in an exploratory research design (Denscombe, 2010a) since knowledge about the intended topic is scarce and no theory has been developed about the relation between SWAs and ITT. Specifically, this research is framed as international comparative research and uses methods aligned with a qualitative approach. An international comparative research is a study that describes a specific dimension (culture, systems, institutions, etc.) using the same research tools to compare systematically the manifestations of phenomena in more than one country (Hantrais, 2008). Comparing different countries allows researchers, educators and policy-makers to move towards a more explicit understanding of their own conceptualisations about different aspects of education and, in doing so, acknowledge that there are alternative ways to deal with the challenges of teaching and learning (Knipping, 2003). My research compares Chile a country beginning its path to stronger accountability strategies with England a country where relationships and tensions between external assessments and teachers capacities have a longer history. Some considerations are needed when conducting comparative studies (Flick, 2009): Selection of appropriate dimensions allowing comparisons between cases: these dimensions were selected according to this researchs questions and consider the link of SWAs with teaching in general, ITT and CBAs. Consideration of conditions that should be kept constant in order to facilitate the comparison according to previous dimensions. Basically, this condition was the existence of a national assessment system, externally applied to schools and with some consequences attached (as the publication of results).

40

Special attention to the complexity (context and inherent structure) of the cases that are being compared: this consideration is addressed through a deep investigation about contextual factors shaping SWAs and ITT in each country. In order to validate the quality of contextual information, interviews were held with leading academics from the field of education in each country. These interviews were structured following the information gathered in Chapter 1 of this thesis (highlighting suggested changes in situ); after interviews, some information was eliminated and some added. When analysing the findings of this research it is fundamental to consider that qualitative methods in comparative research are at an early stage of development (Hantrais, 2008). This highlights the exploratory character of this study and it also draws attention to its contribution to innovative research methods in social sciences. 2.2.1 General description of research procedures This research started with the literature review and analysis of background information of SWAs in different countries with the aim of finding a case that could be interesting to compare with Chile. After selecting the cases that would be studied, background information was analysed in order to define the dimensions of this research and also its units of analysis and participants. Having decided on the cases for the study, research methods were selected according to the criteria of obtaining the most information possible considering the restricted time and resources for this study. Documentary analysis was chosen as the method allowing the widest and most complete view of how different educational systems design their national assessments and which elements are considered when training new teachers. The inclusion of interviews was considered as a complement to documentary analysis under the conviction that official documents do not always reflect the people involved in their elaboration and implementation; in this sense, interviews can triangulate the information

41

gathered from the documentary analysis. After the selection of documents and the description of contextual background information, special interviews were held with leading academics to validate this part of the research. Documentary analysis was completed during March, April and May 2011 and the interviews were conducted between April and May 2011. The fact that interviews were conducted after documentary analysis was started, allowed inclusion of aspects that were implicit or omitted in most of the official documents. Finally, the interviews were analysed and that analysis complemented the information obtained through the documentary analysis. 2.2.2 Methods and procedures This section outlines the research methods used as part of the research strategy. It gives a description of the methods used: documentary analysis and interviews. Documentary analysis A document is an artifact which has as its central feature an inscribed text (Scott 1990, p. 5). Documents may be found in different formats (as notes, case reports, contracts, drafts, diaries, statistics, letters or expert opinions) and all shape reality in a particular way. In that sense, documents function as communicative devices and not as mere information containers (Flick, 2009). A main advantage of the analysis of documents over other methods is that they often permit going beyond the perspectives of the members in the field (Flick, 2009, p. 261); in other words, documents are a privileged way to gain access to fundamental aspects inside an institution. This research has adopted a problem-oriented approach for documentary analysis (Bell, 1999), since predetermined questions will be applied to documents to analyse them.

42

Interviews When the researcher is interested in peoples opinions, feelings, emotions and experiences, interviews may be one of the most appropriate instruments for research (Denscombe, 2010b). As this research is interested in how different institutions conceptualise the relation between SWAs and ITT, interviewing members of relevant institutions offer a more direct approach to subjective meanings behind official communications. There are many definitions for interviews (Drever, 2003); in this research the interview is understood as a social encounter where knowledge is co-constructed between interviewer and interviewee (Walford, 2001). In the same way as there are many definitions for interviews, there are also many types of interview. For this research, the interviews combined characteristics of both exploratory and semi-structured interviews. The former aims to develop hypotheses rather than to collect facts (Oppenheim, 2000) since it is important to gain access to what interviewees think about the possible links between SWAs and ITT; the latter allows flexibility for the interviewer to adapt the interview to contextual and research conditions without losing the focus on the main dimensions being studied (Denscombe, 2010b). The interview schedule for this research was developed after defining its research questions. The questions were developed aiming to cover three dimensions: 1. Teachers assessment competencies 2. Nationwide assessment policies and relationship with teaching 3. ITT and SWA policies The interview schedule was piloted with a professional of the Chilean national assessment system in order to assure the timing of the interview and that the questions would effectively lead to exploration of the desired topics. After this pilot, the interview was adjusted mainly by merging questions; also, an initial

43

question was added in order to frame and introduce the topic of the interview. Table n4 contains the final interview schedule: Table 4: Interview schedule Dimension Questions Introduction Experience in the English/Chilean assessment systems. Dimension 1: Teachers assessment competencies

1. Teaching standards state that recently graduated teachers should: Know how to use local and national statistical information to evaluate the effectiveness of their teaching, to monitor the progress of those they teach and to raise levels of attainment. Do you think that most of the teachers in your country demonstrate a performance according to this standard? 2. What are the main obstacles for teachers to overcome in reaching that standard? Dimension 2: SWAs policies and teaching 1. What is your general opinion about system-wide learning assessment policies in England/Chile? 2. What do you think has been the impact of SWAs on teaching and on how students learn? 3. What is the relation between SWAs and CBAst? What should be the role of the government and external agencies/faculty of education within that relation? 4. Nowadays, are there some official initiatives to reinforce the relation between external and internal assessments? Dimension 3: ITT and SWAs 1. Do you think that teachers should be trained to use SWAs assessments during ITT? Why? If yes: 2. Do you think that teachers are properly prepared to use SWAs during ITT? 3. What does your institution do to prepare teachers to use SWAs assessments during ITT? If no: 2. How should teachers be prepared to use SWAs? Source: personal elaboration.

44

It is important to state that the same schedule was not applied to all interviewees. Some changes were implemented during the interview in order to maintain its natural flow (for example, in some cases, questions about teachers assessment competencies were asked after questions about the impact of SWAs on teaching, especially in the case of interviewees from assessment agencies for whom this topic naturally emerged after speaking of their relationship with SWAs). Some changes were planned: the first question about teaching standards was adapted for Chilean interviewees referring to the Framework for Good Teaching and the standards related to the expected knowledge of educational policies and of different ways to assess pupils. Due to resource and time constraints, interviews in Chile were made using Skype software, facilitating a face-to-face meeting through the computer (Flick, 2009). Interviews in England were made face-to-face in interviewees work places. Each interview lasted between 40 and 60 minutes. All interviews were recorded after the explicit consent of the interviewee and transcribed by the researcher using specialised software. Finally, it is necessary to mention that one of the English interviews could not be completely recorded; researchers notes of the interview were expanded and validated by the interviewee and those notes were then analysed as the other interviews. Finally, each selected method will help to answer the research questions with different perspectives and levels of informational detail, as shown in Table n5:

45

Table n5: Information gathered from each method Research questions Documentary analysis Interviews How is the relationship between system-wide assessment policies and the preparation of teachers during their initial teacher training? How do ITT institutions prepare trainee teachers for SWAs? Is this related to how teachers are trained to deal with CBA?

Analysis of course handbooks and other material shows how SWAs are considered in the work with trainees. By looking at how SWAs are considered, a relation between national assessments and ITT can be inferred.

Interviewees give their personal opinions about how trainees should be prepared for SWAs and also how this type of assessment should or should not be linked to CBAs. What do departments in charge of SWAs expect from teachers preparation? Is this related to what they expect from CBA?

Analysis of official documents from the departments in charge of SWAs shows what is expected from teachers, and this can be linked with knowledge and capacities that could be considered in ITT. This analysis also shows the relation between SWAs and CBAs that is officially promoted.

Interviewees give their personal opinions about the demands that SWAs represent for teachers. These demands can be delineated, for example, by referring to the consequences of SWAs and possible links between SWAs and CBAs, etc. What differences and similarities exist between Chile and England regarding the relationship between SWAs and ITT?

Documents show the official (governmental or university) position of each country in relation to SWAs and how teachers should be trained for them. Interviewees present how the official position is implemented and the different nuances in relation to this topic.

Source: personal elaboration.

46

2.2.3 Participants and documents selection The selection of documents and participants in this research was made considering that sampling should be purposive with an eye on the dimensions that are relevant for the comparison (Flick, 2009, p. 135). The sample of this research is non-probabilistic since it does not involve random selection of participants. Specifically, this researchs sample combines the characteristics of purposive samples, where subjects are selected because of some characteristic and according to research purposes; and convenience samples, where subjects are selected because of their convenient accessibility and proximity to the researcher (Ary et al., 2009). In order to do this, a successive selection of predefined sampling levels was designed. The levels are illustrated in Diagram n3: Diagram n3: Sampling levels

Source: personal elaboration.

47

As this diagram shows, the first step to building the sample is the selection of countries for