Offic

e o

f th

e D

VC

(S

&E

) Development of a University

Standards Framework

(USF) and Online Evaluation

and Reporting Tool at the

University of Tasmania

Dr Cassandra Saunders and Dr Sara

Booth

Student Evaluation, Review and Reporting Unit (SERRU)

http://www.utas.edu.au/student-evaluation-review-and-reporting-unit/

Overview

Origin of standards and performance indicators in Australian HE

What’s happening at the National level

Development of a University Standards Framework and Online

Evaluation and Benchmarking tool at UTAS

Origin of Standards and Performance

Indicators in Higher Education

The overall HE student experience plays a critical role in student

success

Student experience can be defined as;

‘All experiences of facets of the university experienced by an individual

student’ (Baird and Gordon, 2009)

Difficult to identify ‘the student experience’ as a single construct

Includes; pre-enrolment engagement, experience of first-year and

subsequent years of study, graduate studies and graduate outcomes

(Chalmers, 2008)

2008-2009 highlighted the development of performance indicators

(PIs) in the Australian HE sector to measure the quality of the

student experience

Bradley Review (2008) was instrumental in setting the context for

change in accountability and transparency

Transforming Australia’s HE System (May 2009):

Indicators should include measures of success for equity groups and measures of quality

in learning and teaching

Setting and Monitoring Academic Standards for Australian HE: A

Discussion (Coates, May 2009):

Elucidated the description, and the HE sector’s understanding, of standards of academic

achievement

International Trends in Establishing the Standards of Academic

Achievement in HE (Harris, August 2009):

Literature review on academic standards in the international scene

Learning and Teaching Academic Standards (LTAS) Project (October

2009):

Facilitated the formulation of academic standards in 6 discipline areas

An Indicator Framework for HE Performance Funding (Dec 2009):

4 indicators that underpin institutional targets against sector-wide measures

Nationwide Consultation Process

Multidimensional standards frameworks established in 2008-2010;

Charles Sturt University: → The next steps: Defining standards in learning and teaching, research and professional

engagement (Burnett, Chambers and Gorman, 2008)

→ Monitoring and promoting progress towards achieving CSU standards (Chambers,2008)

Curtin University: → Curriculum, Staff, Learning Environment, Students, Ethics, Equity and Social Justice

Macquarie University: → Teaching, Learning Environment and Curriculum

RMIT: → Academic Standards Framework

University of Canberra: → Teaching Quality, Research Quality, Administration & Management Quality

University of South Australia: → Institutional climate & Systems, Diversity & Inclusivity, Engagement & Learning

Outcomes and Assessment

University of Western Sydney: → Course Design, Support, Delivery and Impact

Establishment of Standards at the

Institutional Level

TEQSA’s HE Standards Framework

Provider Standards-Provider Registration, Provider Category, Provider Course Accreditation Standards

Qualification Standards

Teaching and Learning Standards

Research Standards

Information Standards

Dom

ain

s

Thre

shold

Sta

ndard

A standard may be defined as; ‘A specification or other precise criteria designed to be used consistently as a rule,

guideline or definition of a level of performance or achievement’ (AUQA 2007)

TEQSA has established a HE Standards Panel (HESP) responsible for;

Development and monitoring of HE Standards Framework

Reviewing and broadening the threshold standards

Alignment of Processes at the National and

Institutional Levels

National Level

Higher Education

Standards Panel National instruments

to measure threshold

Standards

HE Standards Framework

Data and Data Systems

Data Governance

Validation

Process

Institutional Level

Quality and Standards

Committee Institutional Instruments

to measure threshold

standards

University Standards

Framework

Data and Business

Intelligence Systems

L&T Dashboard

Validation

Process

Validation Process:

Development of a University Standards

Framework (USF) at UTAS

Conceptual Design

Standards Framework Conceptual Design

Selection of Standards and

Measures

Standards Statement Design

Performance Indicator Design

Measures Design

Collection and Analysis of Data

Data collection process – software tool

Integrity and accuracy of data sample

Evidence/rating equivalence

Data analysis process

Data storage process

Benchmarking of Data

Benchmarking themes and processes

Benchmarking data - areas of good practice;

areas for improvement

Validation Checks

Validation checks on data and analysis

Peer review

Adjustment

Adjust standards for context and national

drivers

Adjust software tool

Adapted from Radloff et al. 2012

UTAS USF Project was initiated and endorsed by the Quality

Committee in May 2011

Aim:

To develop a comprehensive standards framework that could be used for;

regulatory, self-improvement and self-enhancement purposes

Aligned to HE Standards Framework

Influenced by;

Macquarie University’s Teaching Standards Framework (TSF) (Sachs et al. 2012)

ALTC Inter-University Moderation Project (Krause et al. 2011 - ongoing)

Higher Degree by Research (HDR) Best Practice Framework (Luca et al. 2011)

Conceptual Design of UTAS USF

Conceptual Design of UTAS USF

Student Evaluation, Review and Reporting Unit (SERRU)

Research

Research Training

Learning

Teaching

Curriculum

Student Experience (previously Student Support)

6 K

ey D

imen

sio

ns

Each dimension is composed of a set of standards, indicators and

performance measures that were developed based on their capacity

to measure and improve practice

A comprehensive consultation and engagement plan was

implemented with key stakeholders

Quality Committee recommended testing of the USF;

Student Experience dimension - Faculty of Health Science (FHS)

Teaching dimension – Cradle Coast Campus (CCC)

Conversations during the testing phase were framed by 2 questions;

What are our standards?

How do we know we have achieved those standards?

Selection of Standards and Measures

Student Evaluation, Review and Reporting Unit (SERRU)

USF Online Evaluation and

Benchmarking Tool

Student Evaluation, Review and Reporting Unit (SERRU)

Evaluate In

stitu

tion

al

Facu

lty

Co

urs

e

Sch

oo

l

Continual improvement and better outcomes across UTAS

Report

Benchmark

Online Evaluation and Benchmarking

Tool: User Levels

Student Evaluation, Review and Reporting Unit (SERRU)

Project Manager (oversees review process and

reports on actions)

Coordinators (coordinate information collection, collate data and report on areas of

review they are responsible for)

Information Collectors

(collect information against indicators of practice, establish

actions for improvement

System Administrator (installation and set-up)

Uploading Evidence to the Online Tool

Standard

Indicator

FHS Standards Working Group (SWG) was established to oversee

the monitoring of Student Experience standards, performance

indicators and measures

Student Experience dimension was revised on feedback from SWG

prior to data collection

Self-review of Student Experience policies and processes using the

framework

Individuals within each School were designated responsibility for collecting and

collating evidence against each performance indicator

Data provided was uploaded into the online tool and a report

generated and validated by relevant stakeholders

Data Collection and Analysis of the

Student Experience Dimension

Student Evaluation, Review and Reporting Unit (SERRU)

CCC Standards Working Group (SWG) was established to oversee

the monitoring of Teaching standards, performance indicators and

measures

Teaching dimension was revised on feedback from SWG prior to data

collection

Online survey was administered to all CCC teaching staff using the

Evaluation and Benchmarking Tool (n=98)

24% response rate

Data from survey instrument was analysed and a report generated

using the Evaluation and Benchmarking Tool.

Data Collection and Analysis of the

Teaching Dimension

Student Evaluation, Review and Reporting Unit (SERRU)

Online tool can be used for benchmarking purposes both internally

and externally;

More investigative, research-informed process

Clear and consistent mechanism for HE institutions to benchmark a diverse

range of data, both within the institution and cross-institutionally

FHS School responses for each PI were benchmarked to demonstrate this

capability;

Identified areas;

of best practice across the Faculty

where some Schools are performing better than others

Provided an opportunity to share best practices across Schools

Lead to enhancement of Student Experience practices across the Faculty

Benchmarking of Data

Reports discussed with, and validated by, key stakeholders

and Senior Executive

Testing of the USF and online tool highlighted;

An emphasis on practices

Areas where performance measures and/or PI did not clearly reflect

the standard

Gaps in performance measures

Areas where information collectors did not understand the PI or

measure

14 recommendations for changes to the framework

Framework has since been revised and specific performance

measures have been removed

Validation Checks and Adjustment

Development and testing of the UTAS USF has resulted in;

1. A framework that is adaptable to the diversity of the University’s organisational structure

2. A framework that links institutional, strategic, financial and managerial responsibilities as

well as processes, policies and practices across the institution

3. An online benchmarking and reporting tool and survey instrument that collects and stores

evidence in a secure MySQL environment

4. Guidelines for evaluating standards using the online reporting tool

5. A standards validation process

6. A comprehensive list of evidence that takes into account differences in context

7. A comprehensive list of policies that relate to each Dimension

8. Support on how to use the framework through SERRU

USF must be a fluid and flexible model

Continuous adjustment will;

enhance its meaningfulness and functionality at an institutional level

ensure alignment with national drivers

Conclusion

References

Australian Learning and Teaching Council (2009) Learning and Teaching Academic Standards Project.

Australian Universities Quality Agency (AUQA) (2007) AUQA Audits – 2007 Onwards: Overview of Audit Cycle 2, AUQA,

Melbourne, VIC.

Baird, J., & Gordon, G. (2009) Beyond the rhetoric: A framework for evaluating improvements to the student experience.

Tertiary Education and Management, 15(3), pp. 193-207.

Bradley, D., Noonan, P., Nugent, H., & Scales, B. (2008) Review of Australian Higher Education: Final Report. Department of

Education, Employment and Workplace Relations: Canberra, ACT.

Burnett, P., Chambers, R. & Gormon, L. (2008) The Next Steps: Defining Standards in Learning and Teaching, Research and

Professional Engagement at CSU, Senate Discussion Paper, Charles Sturt University

Chalmers, D. (2008) Indicators of university teaching and learning quality, Australian Learning and Teaching Council: Sydney,

NSW.

Chambers, R. (2008) Monitoring and Promoting Progress towards Achieving CSU Standards, Senate Discussion Paper,

Charles Sturt University.

Coates, H.B., Woodhouse, D., James, R. & Sadler, R. (2009) Setting and monitoring academic standards for Australian higher

education: A discussion paper, Australian Universities Quality Agency (AUQA).

Department of Education, Employment and Workplace Relations (DEEWR) (2009) An indicator framework for higher education

performance funding, Commonwealth of Australia: Canberra, ACT.

References

Department of Education, Employment and Workplace Relations (DEEWR) (2009) Transforming Australia’s higher education

system, Commonwealth of Australia: Canberra, ACT.

Harris, K-L. (2009) International trends in establishing the standards of academic achievement in higher education: An

independent report and analysis, Centre for the Study of Higher Education, University of Melbourne: Melbourne, VIC.

Krause, K-L., Scott, G. and colleagues (2011-ongoing) A sector-wide model for assuring final year subject and program

achievement of standards through inter-university moderation, ALTC Strategic Priority Project, Griffith University (Lead

Institution).

Luca, J. (2011) HDR training excellence in Australia: A best practice framework. ALTC Project, Edith Cowan University.

Radloff, A., Coates, H., James, R. & Krause, K-L. (2012) University Experience Survey: Report on the development of the

University Experience Survey, Commonwealth of Australia: Canberra, ACT.

Sachs, J. and colleagues (2012) Teaching Standards Framework: Online Development and Pilot Study Project, DISSRTE

Project, Macquarie University.