School-based Assess Qld Sys

8/13/2019 School-based Assess Qld Sys

1/24

School-based assessmentThe Queensland system


2/24

Published February 2010

The State of Queensland (Queensland Studies Authority) 2010

It cannot be over-emphasised that the mode of assessment dictates the nature

of the educational experience and the quality of the relationship between

teacher and pupils. Assessment is not something separate a tool by

which education may be evaluated; it acts upon the educational system soas to shape it in accordance with what the assessment demands. You cannot

have, at one and the same time, education for personal growth and a totally

impersonal system of assessment. Assessment should be a bond between

teachers and taught, not something which threatens and antagonises.

To humanise assessment, then, we have to make of schooling a more co-

operative enterprise between teachers and pupils, and an opportunity to

develop the whole range of human competencies, leading up to informative

profiles. This should be the pattern of things for the immediate future; it is

the way to shed the dreary, and often unjust, grading techniques of traditional

education.

Hemming (1980, p. 11314)


3/24


4/24

2 | Queensland Studies Authority

Before the 1970s, Queenslands students wereassessed at the end of their school studies by asingle highstakes examination that consistedprimarily of a penandpaper test. The examinationwas set and marked by authorities external toschools, and certicates were issued by centralauthorities. Because both teachers and studentswere focused on learning what was required to dowell in the nal examination, rather than pursuingan educational program tailored to meet studentneeds, these exams had an enormous inuence onthe curriculum and the teaching methods used inschools.

At this time, universities had the strongest inuenceon how these examinations were set a featurethat Queensland had in common with otherjurisdictions around the world. The main goal oftertiary institutions was to assess the capabilitiesof the small number of students who were destinedfor university study. However during the 1960s, asstudent retention rates started to increase, therearose an inherent conict between the goals of theuniversity in setting public examinations and thoseof the recently established Board of SecondarySchool Studies in developing the senior curriculum.

Universities were focused on academic excellenceand identifying students most able to undertakeuniversity studies. The Board, largely made up ofproviders of secondary education, was interested incatering for the aptitudes and abilities of all students.

At the end of 1967, the conicting goals of secondaryand tertiary education institutions were thrust intothe public eye when 68 per cent of Queenslandstudents sitting the Senior Public Examination inPhysics failed to achieve a Pgrade, which was, initself, short of the standard required for universityentrance. This was the second consecutive year

that the Physics papers had attracted widespreadcriticism for being unrealistic and extremely difcult.It was clear that the nal examinations bore littleresemblance to the curriculum that had been thefocus of the teachers and their students in the naltwo years of schooling.

Along with teachers and students, journalists,politicians and members of the public questionedhow a public examination could equally serve its dualpurposes as the culmination of school studies forstudents not intending to go to university, and asa mechanism for selecting students for university

study.

The catalyst for change

Was there a better alternative? This was the questionfor educationalists and policymakers in Australiaand overseas, aware that the increasing numberof students undertaking their senior schoolingrequired a curriculum that prepared them fordifferent occupations or roles in a rapidly changingworld. Many were keen to break the hold that endofschooling public examinations had over teachingand learning. But only in Queensland could the localconditions bring about this kind of sea change ineducational philosophy and practice.

The events of the late 1960s set in motion a seriesof changes that led to the abolition of QueenslandsSenior Public Examination and the start ofexternally moderated schoolbased assessmentin 1972.

Almost 40 years later, Queensland continues tobenet from a unique system of assessment that hascaught the eye of many international experts whoregard it as remarkable or even revolutionary.

The legacy of change

The Queensland system of externally moderatedschoolbased assessment is the end point in acohesive and cumulative approach to teaching,learning and assessment from the Preparatory Year toYear 12 (P12). Our integrated approach:

establishes a better preparation for learning inthe early years

strengthens learning in the middle years ofschooling

builds Year 10 as the transition to the seniorphase of learning

provides young people in the senior phase withexible options for certication and tertiaryentrance (Queensland Studies Authority 2009,p. 1).

Within this approach, assessment is dened asthe purposeful, systemic and ongoing collection ofinformation about student learning.

Introduction


5/24

School-based assessment The Queensland system | 3

Although our system of assessment is at its mostintricate in the senior years when students arecompleting their school studies, there are importantfeatures that are common to Queenslands approachto assessment across the P12 years. These include:

Teaching, learning, assessment and reportingis aligned so that what is taught informs what is

assessed, and what is assessed forms the basis ofwhat is reported.

Teachers design continuous schoolbasedassessment programs and make judgments aboutstandards achieved by their students, includingsummative judgments for reporting purposes.

Clear and specic content and achievementstandards are included in syllabuses, EssentialLearnings and guidelines.

Consistency of teachers judgments is promotedwhen teachers engage in professional dialogue

to discuss and analyse the connections betweenstandards and student work.

Teachers work in partnership with the QueenslandStudies Authority (QSA) to develop highqualityassessment tasks, a process supported bythe development of exemplar assessmentinstruments and annotated samples of studentresponses.

Each of these features exists to varying degreesacross P12, with many of the principles andpractices of the continuously improving systemfor the senior years gradually being applied to suit

the circumstances of the early and middle years ofschooling.

The focus of this document is on explaining theintricacies of the system of externally moderatedschoolbased assessment in place in the senioryears of schooling in Queensland. The benets ofthis mature yet evolving system are laid bare, andthe various steps involved in the moderation journeyand the processes that contribute to certicationand tertiary entrance are described in detail. Theintellectual credibility of the system also receivesattention through a brief discussion of the evidence

for its reliability and validity.

The document concludes by demonstrating howthe principles of the system in the senior yearsare starting to be applied to the middle years ofschooling an important part of Queenslandsdeveloping P12 approach to teaching, learning andassessment.

The evolution of externally moderatedschool-based assessment

191072: Students achievements are assessed at theend of the course of study by a single examination.

1972: Queenslands system of externally moderatedschoolbased assessment is introduced based on

recommendations for schooling in the Radford Report(1970). For the rst time, assessment instrumentsdevised by teachers, and the judgments they makeabout how well the students have learnt, become themajor component of the nal results. It is the end ofthe oneoff external examination and the beginning ofhighstakes conversations between teachers.

1974: The Australian Scholastic Aptitude Test (ASAT) isintroduced. Tertiary Entrance scores are derived fromschool assessments scaled against the test.

1980: The Review of SchoolBased Assessment(ROSBA) results in the Queensland system moving

from a normbased model to a criteriabased model ofassessing student achievement.

1990: The Review of Tertiary Entrance in Queensland,chaired by Professor Nancy Viviani, recommendsindependent research be conducted to determine thecomparability of assessment outcomes across schoolsin Queensland.

1992: The Queensland Core Skills Test replaces theASAT to provide group results for scaling purposes inorder to calculate Overall Positions and Field Positionsfor tertiary entrance. The scaling process now usesan achievement test of the 49 Common CurriculumElements derived from senior syllabuses.

1994: An independent study by Masters and McBrydeprovides strong evidence of comparability in theQueensland senior secondary system.

To improve the reliability of the system, studentfolios, selected randomly by the QSA each year, areindependently reviewed and the results analysed asa measure of the comparability within the system.Random sampling has facilitated additional andongoing research into the review panel moderationprocess.

2006: Cumming, WyattSmith, Elkins and Neville arguethat the Queensland system has gained international

recognition by placing highstakes judgments in thehands of teachers who, if appropriately supported,are well placed to deliver valid and reliable judgmentsabout student achievement.

2007: The random sampling processes are reviewedand rened, leading to improved efciency andincreased consistency in schools judgments forawarding levels of achievement.


6/24


7/24


pedagogy is practised, the teachers do not teachand then hand over the assessment that counts toexternal experts to assess what the students havelearnt. Authentic pedagogy occurs when the act ofteaching involves placing highstakes judgments inthe hands of teachers. Since the abolition of externalexams, Queenslands teachers have been able to

broaden the ways in which students can demonstratewhat they have learnt rather than try and predict whatexaminers might be looking for. They have cateredto the diverse learning needs of students by offeringa variety of assessment experiences. And all of thisassessment counts.

Teachers as professionals

The Queensland system is based on condence andtrust in the professional judgments of teachers.Their judgments may be open to external scrutiny

and moderation through review panels (seeThe moderation process on page 7) and thisis an important safeguard in the system butteachers have developed a sense of ownershipand accountability independent of any externalcompliance measures.

Since the early 1970s, thousands of teachers havecontributed to the Queensland system. They havebeen involved in processes that promote sharingbetween each other, among schools and acrosssystems at a level that cannot be replicated insystems that rely on external examinations. Evidence

from surveys indicates that teachers prize above allelse the professionalism, networking and sharing ofresources, practices and ideas that is evident in theQueensland system.

Other key benets of the Queensland system:

Students are able to show the full extent of theirknowledge and abilities

Students have the opportunity to demonstratewhat they know and can do over a period oftime, in situations that are similar to those theywill encounter in further education, training andemployment.

Students receive frequent and detailed feedback

Continuous assessment provides moreopportunities for teachers to give timely andcomprehensive feedback to students about howthey might improve their performance. Formativeassessment at particular points indicates thestandards achieved by students and providesmeaningful reports to parents/carers on studentsachievements. It helps students achieve thehighest standards they can within their owncapabilities. From this, students learn that in

any assessment instrument they should bemade aware of the task, the conditions and the

criteria and standards against which they will beassessed. They experience a total engagementwith the assessment process and learn to thinkcreatively about problems.

Teachers may choose from a range of assessmenttechniques

Teachers can use a range of assessmenttechniques, including group work, oralpresentations, practical responses, assignmentsand supervised examinations, to cater for variedlearning styles.

Teachers constantly improve their assessmentabilities

Teachers improve their knowledge of assessmentand their assessment practices, thus improvingtheir own teaching.

Teachers receive specialist training andprofessional development

Teacher panellists receive training by the QSAin how to make comparable judgments onstudent achievement. They also receive valuableprofessional development by discussingassessment issues with their peers as members ofthe QSAs moderation panels.

Teachers can tailor curriculum and assessment tolocal needs

Teachers can write work programs (i.e. anoutline of how the school intends to implementa syllabus) that reect the schools clientele interms of interests and issues, and that make bestuse of school and local facilities.

Teachers receive external advice

Teachers in schools are provided with adviceabout:

the extent to which assessment instrumentsprovide opportunities for students to meet thesyllabus standards

how well school judgments of the qualities ofstudent work match the syllabus standards

the schools level of achievement decisions.


8/24


Assessment in the senior years ofschooling

The role of externalmoderation

Moderation is a set of processes designed to ensurethat results recorded for Authority subjects matchthe requirements of the syllabus (Authority subjectsare those subjects based on QSA syllabuses that,when taken collectively by students, qualify themfor university entrance on exit from Year 12). Theaim of moderation is to ensure comparability thatis, students who take the same subject in differentschools, and who attain the same standard throughassessment programs on a common syllabus, will beawarded the same level of achievement. This doesnot imply that two students who receive the samelevel of achievement have had the same collectionof experiences or have achieved equally in any oneaspect of the course. Rather, it means that they have,on balance, reached the same broad standard.

The Queensland system for Years 11 and 12 isfounded on a partnership between schools and theQSA. Central to the Queensland system, and one ofits unique features, is the involvement of teachersand schools in all facets of moderation. It is thisengagement of teachers that allows the system towork effectively.

All Queensland teachers of senior students havesome involvement in the system, whether it be as amember of a syllabus writing committee, a teacherassessing a students achievements, a teacherdeveloping a schools work program or as a memberof a review panel.

The role of standardsIn Queensland, assessment is standardsbased. Thismeans that judgments about the quality of studentachievement are made with reference to predenedstandards that describe how well students haveachieved the objectives in syllabuses. Predenedstandards ensure that:

students and teachers know what is expected foreach level of achievement and can work togetherto achieve the best result for the student

comparability from school to school can beachieved

teachers can discuss standards with parents/carers when reporting a students achievements.

Each syllabus has general objectives that state whatmust be taught and assessed. The general objectivesare grouped by dimensions, the most importantaspects of the subject. A standards matrix describesthe standards for each dimension, expressed as AEgrades.

Teachers use the standards matrix rst at the level ofthe individual assessment instrument and second for

decisions about overall achievement across a rangeof assessment instruments towards the end of thecourse. These decisions are onbalance judgmentsabout how the qualities of the students work matchthe standards descriptors overall in each dimension.On completion of a senior secondary course of study,teachers award one of ve levels of achievement:

Very High Achievement (VHA)

High Achievement (HA)

Sound Achievement (SA)

Limited Achievement (LA)

Very Limited Achievement (VLA).


9/24


The moderation process

Our system has been evolving for nearly 40 years,with rigorous qualityassurance processes in placeto ensure public condence in the system. In orderto ensure that the levels of achievement in Authoritysubjects match the requirements of syllabuses, theQSA oversees seven phases that together form themoderation process.

In this section, the journey of moderation isexplained from the perspective of a school, SunshineState High School (Sunshine SHS), and the seniorsubject, Modern History.

Moderation roles

Teachersassess student progress throughout Years 11and 12. At the end of Year 12, teachers decide the exitlevels of achievement to be awarded to their students.Teachers judgments about the standards achieved bytheir students are moderated by the QSA, using trained

expert panels of teachers from schools.District review panels are appointed and trained by theQSA for each subject in each of the QSAs 13 districts.Each district panel comprises practising teachers and achair, with one member for every two schools offeringthe subject in the district.

State review panelsare established for each subject.Each state panel comprises practising teachers anda chair, with membership based on the number ofdistricts with schools offering the subject.


10/24


QSA Schools

Diagram 1: The moderation process

Phase 1: Syllabusdevelopment

Learning Area Reference Committeesdevelop syllabuses for QSA approval

Schools write work programs

Phase 2: Work programapprovalDistrict review panels ensure thatwork programs meet syllabusrequirements

Phase 3: MonitoringDistrict review panels provideadvice to schools about courseimplementation based on samplesof Year 11 work

Phase 4: VerificationDistrict review panels verifyschools judgments of studentachievements

Phase 5: Comparability

State review panels ensurejudgments are comparable acrossthe state

Phase 7:RandomsamplingQSA qualityassures theprocess byrandomly samplingstudent folios

QSA approves work programSchools teach and assessstudents according to theirapproved work programs

Schools apply panel advice

Schools respond to paneladvice

Disagreement?

State panel arbitrates

Phase 6: Confirmation

QSA checks school results andconfirms levels of achievement

StudentoutcomesExit levels ofachievement

awarded


11/24


Phase 1: Syllabus development

Syllabuses are developed by the QSA in collaborationwith Learning Area Reference Committees (formerlySyllabus Advisory Committees) made up of practisingteachers, academics and curriculum experts, andsupported by writing teams of subject experts.Sunshine SHSs contribution to the development of

a new Modern History syllabus may have been toprovide feedback to draft versions of the syllabus thatwere posted on the QSAs website, or one of theirteachers who was a subject expert may have beenpart of the committee that wrote the syllabus.

Before Sunshine SHS begins to teach senior ModernHistory, the QSA provides syllabus implementationworkshops that outline the key aspects of the ModernHistory Senior Syllabus.

This professional development will include outliningthe key aspects of the syllabus:

general objectivesfor a foursemester course ofstudy that state what students should know andbe able to do by the end of Year 12

subject matterto be taught to students, includingcore and mandatory requirements

assessment adviceand requirementsfor achieving the general objectives anddemonstrating exit standards

standards descriptorsthat are used to makejudgments about how well students haveachieved the general objectives by the end ofYear 12.

These workshops occur the year before a syllabus isto be implemented. After attending such a workshopabout the Modern History Senior Syllabus, the historyteachers of Sunshine SHS are now prepared to developtheir schools work program for the following year.

Principles for syllabus development

QSA syllabuses draw on wellresearched designprinciples and elements for equity and excellence,and set the parameters for the intended curriculum.The principles that underpin QSA syllabuses are:

Informed prescription the syllabus concisely sets

outs expected coverage, or what is essential, inclear, accessible professional language. Informedprescription emphasises centralised standardsand syllabus mandates, and so supports theaccountability of the system.

Informedprofessionalism the syllabus setsthe conditions for teachers to exercise theirprofessional judgment to shape their curriculum byproviding:

clarity about the focus of teaching, learning andassessment, balanced with exibility to makedecisions about how best to promote studentlearning

scope for school sectors and schools to

inform practice across diverse school settingsthroughout Queensland, and exibilityand autonomy to plan their curriculum andassessment in ways that best suit theirindividual contexts

reference to useful resources that supportplanning for teaching, learning and assessment,and the needs of specic groups such asIndigenous students, students with specialneeds, and migrant, rural and socioeconomicallymarginalised students.


12/24


Phase 2: Work program approval

Parallel to the professional development of allteachers implementing the Modern History SeniorSyllabus is the training of Modern History reviewpanels across the state. Panel training is animportant step in ensuring that panellists understandthe elements required for a work program to be

approved.

Using the syllabus, Sunshine SHS now writes awork program that shows how the school intendsto implement the syllabus. A work program mustinclude:

a courseorganisationconsistent with the syllabusand suited to the particular needs of students ina school in this instance Sunshine SHS wouldconsider its students, location and availableresources, and write a program appropriate tothese conditions

intended student learningindicating learningexperiences selected by Sunshine SHS

an assessmentprogramwith the required numberof assessment instruments and range and breadthof assessment techniques.

A sample unit of work and examples of assessmentinstruments to be administered by the school may berequired to demonstrate intended student learning.

A work program provides:

guidance to teachers about how the course willbe delivered and assessed based on the schools

interpretation of the syllabus guidance to students and their parents about

the subject matter to be studied and howachievement of the syllabus objectives will beassessed

the basis for QSA approval for the purposes ofcertifying students results for the subject.

The review panel provides recommendations to theQSA about the suitability of the work program forapproval. These recommendations are communicatedto schools and may include aspects of their program

that require further consideration.If a schools work program is not approved onrst submission, additional support is provided.Conversations between district review panel chairsand schools also occur and are an effective beginningto a relationship that continues throughout themoderation process.

Whos involved in moderation in schools?

The school moderator(the school principal ornominee) is responsible for the total assessmentprogram and moderation processes within theschool.

Subject moderators(subject teachers, subjectcoordinators or heads of department) are directly

responsible for preparing and implementing workprograms, using assessment criteria consistentwith syllabus standards descriptors, organisinginternal subject moderation processes wherethese are required (usually if the school hasseveral classes in the same subject), and preparingexternal moderation submissions.

Teachersdesign and deliver learning experiencesand assessment opportunities for their studentsbased on their schools QSAapproved workprogram. Teachers assess student work, determinelevels of achievement according to standardsdescriptors outlined in the subject syllabus, andrecord achievement on a student prole.

Phase 3: Monitoring

Monitoring is the process by which review panelsconsider a schools implementation of a course andassessment decisions in subjects. It is carried outafter approximately half of the course of study hasbeen completed.

At this point, Sunshine SHS selects ve folios ofstudent work for senior Modern History (and all otherAuthority subjects offered at the school). Typically,

one folio to illustrate each proposed interim levelof achievement is submitted. At the monitoringmeeting, a review panel of teachers trained by theQSA will give advice to Sunshine SHS and all otherschools about:

implementation of the Modern History SeniorSyllabus

effectiveness of assessment instruments inoffering students opportunities to demonstratethe syllabus standards

evidence in sample folios of the schools capacity

to match qualities of student work with theappropriate standards descriptors

assessment decisions about interim levels ofachievement.

After Sunshine SHS has received advice from QSAmonitoring ofcers, the district review panel chair isavailable to advise on developing quality assessmentinstruments, implementing the work program andapplying standards. This process also occurs for allother Authority subjects.


13/24


Phase 5: Comparability

Comparability is the process by which state panelslook for evidence that the judgment of standardsacross districts in the state are comparable. Thepanels look at a sample of folios across all levels ofachievement from each district to ensure that thesame standards are being implemented.

This allows the school and the community to becondent that, for example, a Very High Achievement(VHA) awarded in Modern History at Sunshine SHSis comparable to a VHA awarded by other schoolsoffering Modern History.

Phase 6: Confirmation

Prior to exit, schools may consult and negotiate withreview panel chairs about awarding exit levels ofachievement based on additional evidence of studentachievement collected in assessment completedbetween verication and exit.

The QSA then conrms the levels of achievementproposed by schools. This involves:

checking that exit proposals and the relativedistribution of cohorts are consistent with paneladvice

reconciling school data with the exit proposal

negotiating with schools, where necessary, aboutthe nal distribution of levels of achievement.

Phase 7: Random sampling

Random sampling is a qualityassurance proceduredesigned to provide information about thecomparability of nal exit level of achievementdecisions. This process occurs after a course iscompleted.

Each year the QSA takes a stratied random sampleof nalised Year 12 folios from schools acrossthe state and sends them for review to panelsfrom different districts. Where there are issues orconcerns, QSA staff contact the school and advisethe principal of appropriate action to be taken. Areport on the random sampling process is published

annually on the QSA website.

Phase 4: Verification

Verication is the process by which review panelsadvise schools about Year 12 student achievementrelative to syllabus standards descriptors.

At the end of the course of study, Sunshine SHSprepares and sends their verication submissionto be reviewed by the district review panel. It is atthis point in the process that Sunshine SHS willreceive advice about their ability to award interimlevels of achievement relative to syllabus standardsdescriptors. For verication in October, SunshineSHS selected a representative sample of ninefolios of student work for Modern History (and allother Authority subjects offered to their students).Typically, up to ve folios at midrange for each level ofachievement and four folios at thresholds, includingthe topplaced student, will be sent for review.

A panel of teachers trained by the QSA will provide

advice to Sunshine SHS and all other schools about: interim level of achievement decisions

the appropriateness of schools judgments aboutstudent responses to assessment instrumentsrelative to the syllabus standards descriptors

the effectiveness of assessment in allowingopportunities for students to demonstrate thecriteria across the range of standards

coverage of the course as demonstrated by theverication submission.

One of the key features of the Queensland

moderation system is that schools use the adviceoffered by panels to make decisions about levelsof achievement and relative achievement for theirstudents whose work was not sampled at verication.Schools and panel chairs consult and negotiate aboutthe appropriateness of the awarded standards andlevels of achievement.

There is an appeals process arbitrated by the statereview panel if a school and a panel cannot reachagreement. In order to support consensus betweenschools and panels, the QSA provides training forboth review panel chairs and schools in consultation

and negotiation skills. We also continually reviewpolicies and procedures for moderation, andprovide yearly professional development to schooladministrators to achieve a deeper understanding ofthe system.

The coordination of statewide monitoring andverication relies on a commitment from schools torelease staff for panels across all jurisdictions anddistricts. An immense logistical operation takes placein providing samples to each of over 4000 panellists,who then meet in district central locations to reviewschools submissions of student work.


14/24


Fifteen years of data now shows that the QSA canexpect a rate of agreement in the assignment oflevels of achievement that is better than 85%. Thisrepresents a high level of consistency of teacherjudgments in terms of the ve broad levels ofachievement. It is testament to the competenceof teachers in making judgments about the

achievements of their students and the adviceprovided to teachers through the local review panelprocess.

Although less empirical research has been devotedto the validity of the Queensland system, the natureof schoolbased assessment ensures validity inthe process. Validity means that the assessmentinstruments actually assess what they were designedto assess. In 1995 Matters, Pitman and OBrienexamined validity from a functional perspectivethrough discussions with practising teachers. A fewyears later, in 1998, they established strong evidence

of authenticity of assessment through links madebetween syllabus objectives and assessment criteriaand the abundance of information used in makingdecisions regarding student achievement. Accordingto Cumming, WyattSmith, Elkins and Neville (2006),the model of externally moderated schoolbasedassessment is based on the assumption that teachersare in the best position to make valid and reliablejudgments about the achievements of students.

The effect on students is to promote lifelong learning.As suggested by Sadler (1998, 2009), by disclosingcriteria and standards, and ensuring that the

decisionmaking underpinning teacher judgmentsis visible to students and understood by students,the locus of control for learning can move moreeffectively from the teacher to the student and canempower students in terms of their own learning.

A reliable and validprocess

Two important dimensions by which any highstakesassessment system should be judged are reliabilityand validity. Some people mistakenly believe that

because the Queensland system is not basedon a common external examination at the end ofschooling, the results of students are not comparablebetween schools. The research shows that theQueensland system is both reliable and valid.

A 1994 study by Masters and McBryde into thereliability of the Queensland system provided strongevidence of comparability in the assignment of levelsof achievement. The study looked at a sample of 546student assessment folios in English, Mathematics,Chemistry and Modern History, which were ratedindependently by different sets of two markers

under different sets of conditions. The results of thestudy revealed a very high level of comparability andintermarker agreement of 94 per cent (Masters &McBryde 1994, p. 32).

The researchers compared these results with anumber of studies undertaken in other Australianstates at the time, examining the comparability ofresults from external examination markers. In relationto the results of the Queensland study, Masters andMcBryde (1994, p. vivii) concluded:

These levels of agreement are signicantly higher

than levels of intermarker reliability typicallyreported from independent assessments ofstudent work including assessment of externalexaminations.

As a result of the Masters and McBryde study, theQSA has undertaken its own longitudinal studyinvolving the process of random sampling thatcommenced in 1994 and has been repeated eachyear since. This involves independently reviewing aset of student folios, randomly selected by the QSAeach year, and analysing the results as a measure ofthe comparability within the system. The numbers

vary each year depending on the number of subjectsunder review. In 2008, for example, 2250 folioswere independently examined from 327 schoolsubmissions involving a total of 183 schools across18 different subjects. In some years over 3000 folioshave been examined.


15/24


Wiliam advocates a system that is:

Distributed, so that evidence is not collectedentirely at the end of the course of study. TheQueensland system does this.

Synoptic, so that learning has to accumulate. Itdoes in the Queensland system.

Extensive, so that all important aspectsare covered, breadth as well as depth. TheQueensland system is based on results that arefullest, because they accumulate over two years,as well as latest.

Manageable, so that costs are proportionateto benets. The dollar cost of the Queenslandsystem is signicantly less than budgets for endofyear external examinations, but the benetsfor professional development of our teachers faroutweigh budget savings.

Trusted, so that stakeholders have faith in the

outcomes. The Queensland system is accepted byparents, teachers, students, the government andthe wider community.

Continuous improvement

Many countries around the world want to learn fromour system. It is by no means perfect, however,and we continue to review our processes with aview to continuous improvement. We are currentlyundertaking a major review of our moderationprocesses as national curriculum issues impact uponthe Queensland system. We have also extendeda form of the moderation process to the qualityassurance of Authorityregistered subjects acategory of subjects that do not directly contributeto university entrance. Teachers are now alsoundertaking moderation meetings using the resultsof Queensland Comparable Assessment Tasksin Years 4, 6 and 9 as a basis for professionaldiscussions (see Queensland ComparableAssessment Tasks (QCATs) on page 16). However, weneed to do more to conrm the reliability and validity

of the Queensland system by conducting furtherresearch, and continually reviewing and improvingthe moderation process.

The Queensland community can be condent that oursystem meets the criteria for an ideal assessmentmodel, as dened by international assessment expertDylan Wiliam (2008).


16/24


17/24


contributing a maximum of 6 credits. Partialcompletion of a Core course of study may alsocontribute some credit. Students must also meet theQCE literacy and numeracy requirements.

Most students are awarded a QCE at the end ofYear 12. All students who nish Year 12 receive atranscript of their learning in the form of a Senior

Statement, issued in December each year.

Importantly, students who do not meet the QCErequirements at the end of Year 12 can continue towork towards their certicate, regardless of their age.However, credits expire after 9 years. The QSA awardsQCEs in July and December of each year.

TES pathway to furtherstudy

Those students at Sunshine SHS who are eligible willreceive a Tertiary Entrance Statement (TES) at the endof Year 12. The TES gives students a prole of ranks:

a main rank, called an Overall Position (OP),indicates where students are placed in relationto other Year 12 students who completed therequired number of semester units of Authoritysubjects. The OP places students in one of 25bands from 1 (highest) to 25 (lowest).

ve subsidiary ranks called Field Positions (FPs),which rank students on their achievements in up

to ve skill areas in Authority subjects. Each FPhas 10 bands, from 1 (highest) to 10 (lowest).The number of elds reported depends on thecombination of subjects studied. The elds are:

extended written expression, involvingcomplex analysis and synthesis of ideas

short written communication, involvingreading, comprehension and expression inEnglish or a foreign language

basic numeracy, involving simple calculationsand graphical and tabular interpretation

solving complex problems, involvingmathematical symbols and abstractions

substantial practical performance, involvingphysical or creative arts or expressive skills.

Queensland Core Skills (QCS) Test

The QCS Test complements Queenslands system ofexternally moderated schoolbased assessment. Itcontributes information for the calculation of OverallPositions (OPs) and Field Positions (FPs), which areused to rank students for tertiary entrance.

A students individual QCS Test result is not used onits own in the calculation of their OP. Instead, groupresults from the test are used as part of statisticalscaling processes.

The QCS Test is a standardised statewide test set bythe QSA that provides:

an individual result on a vepoint AE scale,reported on the students Senior Statementreceived at the end of Year 12

a group result for the calculation of the tertiary

entrance rank.

QSA calculates the group result by comparingstudents distribution of results on the QCS Test withthe distribution of results from externally moderatedschoolbased assessment.

The QCS Test is not an intelligence test or an aptitudetest. Rather, it is an achievement test grounded in theQueensland senior curriculum and is accessible to allYear 12 students, regardless of their subject selection.

The QCS Test is held over two consecutive days in earlySeptember, late in Term 3 of Year 12, and consists offour papers:

writing task 600 words of continuous prose inresponse to stimulus material (2 hours duration)

shortresponse paper responding in differentways, such as writing a sentence or paragraph,drawing a diagram, performing a calculation,sketching (2 hours duration)

two multiplechoice papers (90 minutes durationeach).

Eligible students receive an OP and up to ve FPs.These ranks are used by universities and colleges oftechnical and further education to select studentsfor tertiary courses. If prerequisite requirements aresatised, then applicants with OP1 are selected beforeOP2 and so on. FPs are used to produce a ner rankingfor selection if the number of applicants with the same

OP exceeds the number of places for a course.The calculation of OPs and FPs uses studentsachievements in 20 semester units of Authoritysubjects, including at least three subjects for foursemesters, each scaled against group results in theQueensland Core Skills (QCS) Test.


18/24


The framework supports this through:

Essential Learnings, which help teachers to aligncurriculum with assessment through connectionsbetween Ways of working and Knowledge andunderstanding, and the assessable elements

Standardsfor Years 19, which reect the twodimensions of Ways of working and Knowledgeand understanding

an onlineAssessmentBankthat providesmodels of assessment instruments that align theEssential Learnings and Standards to supportconsistency of teacher judgments

Queensland Comparable Assessment Tasks(QCATs) in English, Mathematics and Sciencein Years 4, 6 and 9. QCATs provide models ofassessment tasks aligned to the EssentialLearnings and Standards and help promoteconsistency of teacher judgments.

Queensland ComparableAssessment Tasks (QCATs)

The QCATs, a key component of the QCAR Framework,have clear links with Queenslands tradition ofmoderated schoolbased assessment in the senioryears of schooling. These annually administered,statewide assessment tasks are designed to gatherdata that is most useful for teachers in schools notto measure the performance of schools or students.Aimed at building the culture of schoolbasedassessment in the middle years of schooling, they area source of professional development for teachersnot normally involved in the highstakes moderationprocesses that are at the heart of the senior system.

QCATs are designed to provide evidence of whatstudents know, understand and can do in relationto a selection of Essential Learnings for English,Mathematics and Science in Years 4, 6 and 9.Achievement in the QCATs is reported by schools tostudents and parents/carers using ve grades fromAE. QCATs gather important information for teacherson how their students are progressing and alsoinform teachers and administrators in schools aboutthe effectiveness of curriculum programs or the areasof the curriculum designed to cover problem solvingand higherorder thinking.

While moderated schoolbased assessment hasbeen a feature of Queenslands education system inYears 11 and 12 for many years, there is no similartradition in the early and middle years of schooling.In recent years, however, the Queensland systemin Years 19 has been characterised by a focuson aligning curriculum with assessment both forstudent learning and the way student learning isreported. This has been delivered via the QueenslandCurriculum, Assessment and Reporting (QCAR)Framework (QSA 2008).

A crucial feature of this framework is the way itprovides practical models of how principles ofeffective assessment can be applied, throughan online Assessment Bank and the QueenslandComparable Assessment Tasks (QCATs).

The objective is to improve the quality of learningfor students by supporting the development ofteachers assessment capabilities. In this way, wecan establish a network of teachers from Prep toYear 12 skilled in the advanced assessment practicesthat have become commonplace in the senioryears. Assessment within the QCAR Frameworkvalues the professional judgment of teachers and

demonstrates how assessment that measures notjust what students know, but how students work withknowledge, assists teachers in improving learning.

Supporting teachersassessment capability

It has been argued that:

[t]eachers need the freedom to make denitiveevidencebased judgements on their students work

according to established standards and a qualityframework that guarantees the dependability ofteacherled assessments

Klenowski (2008, p. 150).

The QCAR Framework puts teachers professionalknowledge and practice at the heart of aligning whatis taught, how it is taught, how student learning isassessed and how learning is reported.

Building the culture of assessment in themiddle years of schooling


19/24


The Assessment Bank is an online collection ofassessment instruments and resources that arelinked to the Essential Learnings and Standards, anddemonstrates the alignment of assessment, curriculumand reporting guidelines.

Teachers have access to a range of assessmentinstruments and resources in all key learning areas(KLAs) and year levels across Years 19. By engagingwith and contributing to the Assessment Bank, teachershave the opportunity to enhance their understanding ofthe alignment of curriculum, assessment and reporting.

Assessment packages include:

a student booklet the assessment as presented tostudents

a guide to making judgments states what is valuedin the assessment and gives descriptions of theexpected qualities of demonstrated student learning

teacher guidelines taskspecic information aboutthe Essential Learnings being assessed, preparation,implementation and feedback

an indicative A response an example of anAstandard model response

assessment-related resources(where applicable) audio and/or visual stimulus required to completethe assessment

sample responses(where available) studentresponses annotated to clarify how the samplesmatch the taskspecic descriptors in the guide tomaking judgments.

Resources are presented as professional resources or

classroom resources. Professional resources provideinformation and links to articles of professionalinterest to teachers, e.g. readings, presentations, QSA

publications and professional development materials.Classroom resources are designed for teachers to use oradapt for teaching and learning, e.g. planning, teaching,assessing and reporting resources.

All assessment instruments and resources undergoqualityassurance processes, including internal andexternal panelling, editing and, where possible, eldtrialling.

Each complete assessment package models veprocesses to be considered when developing qualityunits of work and everyday assessment instruments:

identifying curriculum selecting the EssentialLearnings, school priorities and the context forlearning

sequencinglearning planning learning experiencesand teaching strategies to respond to the needs andinterests of learners

developingassessment planning a varietyof assessments to collect comprehensive andmeaningful evidence of learning

makingjudgments considering how judgmentswill be made about the quality of learning, using theevidence in student responses

usingfeedback considering how and when toprovide feedback to students.

The Assessment Bank also includes an online forumfor informal teacher collaboration and discussionabout assessment. This forum provides opportunitiesfor professional discussion, which helps to improveconsistency of teacher judgments and build a shared

understanding of standards. Schools can also contributeto the development of the sample responses by providingstudent work samples.

Assessment Bank


20/24


student booklet an authentic, performancebased assessment with one or more products(one booklet per student)

teacher guidelines to inform teachers aboutadministrative procedures necessary to completethe assessment under comparable conditions

guide to making judgments assessable

elements, taskspecic descriptors and a facilityto record the overall grade achieved (included inthe student booklet and teacher guidelines).

From the rst day of the period for administeringthe QCATs, sample responses are available on theweb via the Assessment Bank. They demonstratethe characteristics of typical student responses foreach assessable element at each of the ve grades(AE). The student work samples are sourced fromeld trials. Annotations for each student work sampleexplain how it demonstrates the qualities in taskspecic descriptors.

The teachers at Sunshine SHS make time to preparetheir students for the assessment tasks. But they areaware that they should not rehearse the actual orsimilar task.

QCATs are not pointintime tests. Sunshine SHShas the entire administration period of three tofour months in which to administer, grade andmoderate the three 90minute tasks with itsstudents. Its classroom teachers are in control of theadministration process. They make decisions about:

the preparation of students

administration times one, two or more sessions

when to administer the QCATs within theadministration period

student achievement

moderation of student work

evaluation of student work and school programs

feedback given to the student

use of data collected.

Sunshine SHS elects to administer the English,Mathematics and Science QCATs early in the

administration period. The studentfree day in Term 4is used to meet with teachers from other local schoolsto moderate the students responses.

Once the teachers from Sunshine SHS and theother schools agree on the students grades, theysubmit their data via a database maintained by theDepartment of Education and Training (nonstateschools use a data collection tool available on theQSA website).

After about 10 weeks, Sunshine SHS is notied thatit has been chosen to submit representative samples

of student work to the QSA. The school must identifyve sample responses that are representative

Each QCAT undergoes rigorous qualityassuranceprocesses, including scrutiny by numerous panelsthat include practising teachers, academics andrepresentatives of various stakeholder groups.Results from school trials and feedback from studentsare also incorporated in the development process.

How do the QCATs work?

QCATs are authentic, performancebased assessmenttasks. They are written to be engaging for studentsand incorporate the notion of students solvingmeaningful problems. Schools have the option ofadministering either:

centrallydevised QCATs, developed by the QSA,with common requirements and parameters, andmarked according to a common guide to makingjudgments

schooldevised QCATs, developed by schools inaccordance with a design brief.

Sunshine SHS is one of the vast majority of schoolswhich choose to administer centrallydevised QCATs.Its decision to do so is the focus of the followingdiscussion.

The three main purposes of the QCATs are to:

model an example of a quality assessmentinstrument aligned closely to the curriculum

support teachers in making consistent judgmentsabout the quality of student work, and to increase

the consistency of teacher judgments provide information to teachers and students

about what students know and can do, as well aswhat is working well and what needs attention inschool programs.

Sunshine SHS is rst able to engage with the QCATswhen the QSA makes available a design brief for theYear 9 tasks on its website at least six months beforeimplementation. At the same time, primary schoolscan also access design briefs for the Years 4 and 6QCATs. A design brief:

species the Essential Learnings to be assessed

describes the form of the task and the purpose ofeach component

explains design elements used to maximise thevalidity and reliability of:

centrallydevised QCATs

judgments made about student learning fromthe evidence.

Having had substantial time to plan and makearrangements for the administration of the QCATs,Sunshine SHS receives an assessment package at thebeginning of the administration period comprised ofthe following:


21/24


of midrange AE student responses. Theserepresentative samples of student work are thenused to determine the level of statewide consistencyof teacher judgments.

The information collected from the QCATs isconsidered to be of most value for use within theschool. It is low stakes because it is not used

for measuring or comparing schools or teachers.Individual student information from QCATs is used togive teachers, students and parents/carers feedbackon student learning.

Only aggregate data is forwarded by schools to theQSA. This information is not suitable for comparingthe performance of students in one school againstthe performance of students in other schools.Sunshine SHS is only able to access its ownindividualised school report.

Sunshine SHS accesses a report from the QSA

website on the implementation of the QCATs. It isbased on analysis of all data collected. This reportprovides teachers with insights into the way studentstypically responded to the QCATs. It is valuableinformation for informing the teaching and learningprocess.

Consistency of teachersjudgments

The most effective way to build consistency of teacherjudgment about the quality of student work is for

teachers to look at, discuss and analyse student worktogether in a partnership or team situation.

The QCATs and the Assessment Bank provideresources to support teachers discussions. Amajor benet of the QCATs is that they provideopportunities for teachers to engage in focuseddiscussions, as part of a moderation process, duringwhich they can consider the different taskspecicdescriptors, and how closely the evidence in thestudent work matches them.

When teachers apply the Standards through this

process, they increase their awareness about thevariety of ways in which students may respond to anassessment task. In this way, teachers gain valuableinsights into what students need to learn for eachlevel of achievement. They also move towards acommon understanding of the application of theStandards.

Through professional dialogue, teachers developshared understandings that enhance classroompractice and support alignment of curriculum andassessment. These professional discussions are vitalfor developing an understanding of the connections

between Standards and student work.

What have we learnt from theQCATs?

A key objective of the QCATs has been to strengthenteachers capacity to make consistent judgmentsof student work. Moderation processes within andacross schools are essential for achieving that aim.

Results from the 2009 Years 4 and 6 QCATs suggestthat when teachers are provided with teacherguidelines for administration and a common guideto making judgments about the quality of student

responses, reasonable consistency of teacherjudgments is possible.

During the random sampling process, classroomteachers who are trained as markers independentlygrade each QCAT and then meet to check forcomparability. If for any student booklet a discrepancyhas occurred, they are asked to reach consensus.The level of agreement among the trained markersat random sampling was signicantly higher thanbetween the trained markers and teachers in schools.

With further professional development and

experience of implementing the QCATs, we can becertain that comparability of teacher judgmentswill only increase. This is a lesson that has beenunderstood for many years in the senior system.

Moderation a way of working:A resource for schools

The QSA has provided all Queensland schools with aDVD that models different ways of working to supportquality moderation processes in the early and middleyears of schooling. Moderation a way of workingis a

professional development resource that explains: the benets of moderation

how to make judgments about the quality ofstudent work

the critical factors in the process of moderation

strategies for reaching consensus about studentachievement.

The DVD uses the QCATs as an example, but theprocesses and principles are applicable for anyhighquality assessment activity.


22/24


A system for today and the future

As Australia moves towards the implementationof a national curriculum, Queenslands system ofschoolbased assessment is more relevant than

ever before. Queensland has nearly 40 yearsexperience in developing and implementing asystem of moderation in which teachers and schoolsare accountable for the assessment and reportingof student achievement in Years 11 and 12. A keyfeature of our system is that it does not run the riskof narrowing the curriculum through teachers feelingunder pressure to teach to the test.

There is evidence to show that the Queensland systemstands up to external scrutiny (Masters & McBryde1994). Its intermarker reliability is high and thecrucial role of social moderation in improving teachers

assessment capabilities in an era of accountabilitycannot be ignored. It is a sophisticated system forthe certication of students completing their schooleducation, for making fair and equitable decisionsabout tertiary entrance and for providing richinformation accumulated over two years of study, notjust at the end of the nal year of schooling.

Other education systems can learn from theQueensland experience, not just in the senior yearsbut also in the earlier years of schooling. Across

P12, our system respects and relies upon the roleof teachers as professionals. Teachers have thefreedom to develop a curriculum suited to localneeds and use a wide variety of techniques to assessstudent performance. They are also provided withopportunities to improve their assessment abilitiesthrough specialist training, professional developmentand external advice.

The involvement of classroom teachers in bothcurriculum development and assessment is thefundamental condition for creating authenticpedagogy. The Queensland system is most likely to:

deliver assessment instruments that motivatestudents to learn

provide maximum opportunities for students todemonstrate what they have learnt

improve consistency of teacher judgment aboutthe quality of student work.

It is a system that has evolved over four decadesand is sufciently exible and robust to meet thechallenges of education in the 21st century.


23/24


Bibliography

Board of Secondary School Studies, 198587, Review of School-based Assessment (ROSBA): Discussion papers,Board of Secondary School Studies, Brisbane.

Clarke, E 1987,Assessment in Queensland Secondary Schools: Two decades of change 19641983, QueenslandDepartment of Education, Brisbane.

Cumming, J, WyattSmith, C, Elkins, J & Neville, M 2006, Teacher Judgment: Building an evidentiary base forquality literacy and numeracy education, Centre for Applied Language, Literacy and Communication Studiesand Centre for Learning Research, Brisbane.

Hemming, J 1980, The Betrayal of Youth, Marion Boyars, London.

Klenowski, V 2008, A call to honour: Teacher professionalism in the context of standards referenced assessmentreform, commissioned paper in A Luke, K Weir & A Woods, Development of a Set of Principles to Guide a P12Syllabus Framework: A report to the Queensland Studies Authority, QSA, Brisbane.

Klenowski, V & WyattSmith, C 2008, Standardsdriven reform Years 110: Moderation an optional extra?, paperpresented at the Australian Association for Research in Education Conference, Brisbane, December.

Masters, G & McBryde, B 1994,An Investigation of the Comparability of Teachers Assessments of Student Folios,Tertiary Entrance Procedures Authority, Brisbane.

Matters, G, Pitman, J & OBrien, J 1998, Validity and reliability in educational assessment and testing: A matter ofjudgment, Queensland Journal of Educational Research, 14(2), pp. 5788.

Qualications and Curriculum Authority 2009, Report on Trial of Models of Moderation within Assessing PupilsProgress 2007/8, report to project team, London.

Queensland Studies Authority 2008, Building Student Success: A guide to the Queensland Curriculum,Assessment and Reporting Framework, QSA, Brisbane.

2009, Learning P12: Informed prescription, Informed professionalism, QSA, Brisbane.

Radford, W 1970, Public Examinations for Queensland Secondary School Students, Queensland Department ofEducation, Brisbane.

Sadler, R 1998, Letting students into the secret: Further steps in making criteria and standards work to improvelearning, paper presented at the Annual Conference for State Review Panels and District Review Panel Chairs,July.

2009, Thinking differently about assessment: Why feedback is not enough, paper presented at theInternational Association for Educational Assessment 35th Annual Conference, Brisbane, 15 September.

Viviani, N 1990, The Review of Tertiary Entrance in Queensland, report submitted to the Minister for Education bythe Tertiary Entrance Reviewer, Queensland Department of Education, Brisbane.

Wiliam, D 2008, What do you know when you know the test results? The meanings of educational assessments,keynote address at the Annual Conference of the International Association for Educational Assessment,

Cambridge, September.


24/24

Queensland Studies Authority295 Ann Street Brisbane

PO Box 307 Spring Hill

QLD 4004 Australia

T +61 7 3864 0299

F +61 7 3221 2553

www.qsa.qld.edu.au

School-based Assess Qld Sys

Documents

Transcript of School-based Assess Qld Sys