NTLT 2013 - Paul Maramos - How we are measured impacts upon how we teach
26
How we are measured impacts upon how we teach Paul Marambos Southern Institute of Technology [email protected]
-
Upload
ntlt-conference -
Category
Education
-
view
127 -
download
0
description
How we are measured impacts upon how we teach
Transcript of NTLT 2013 - Paul Maramos - How we are measured impacts upon how we teach
How we are measured impacts upon how we teachPaul MarambosSouthern Institute of [email protected]
Research Topic
Will EPIs contribute to the achievement of key objectives for the tertiary sector?
Why measure? Nearly 700,000 New Zealanders
participated in formal tertiary education in 2009, including industry training.
The Government spends around $4 billion a year in operating expenditure
Background Widespread increase in measurement
within organisations. Management accounting – drive to link
research to practice Case study approach
Content Key tertiary objectives for New Zealand Education Performance Indicators (EPIs) Research undertaken Results Suggested changes to the EPIs
Key tertiary objectives for New Zealand
1. provide New Zealanders of all backgrounds with opportunities to gain world-class skills and knowledge
2. raise the skills and knowledge of the current and future workforce to meet labour market demand and social needs
3. produce high-quality research to build New Zealand's knowledge base, respond to the needs of the economy and address environmental and social changes
4. enable Maori to enjoy education success as Maori.
Government expectations of Universities
The Government expects universities to: enable a wide range of students to
successfully complete degree and post-graduate qualifications
undertake internationally recognised original research
create and share new knowledge that contributes to New Zealand’s economic and social development and environmental management.
Government expectations of Polytechnics
The Government expects polytechnics to: enable a wide range of students to complete
industry-relevant certificate, diploma and applied degree qualifications
enable local access to appropriate tertiary education
support students with low literacy, language, and numeracy skills to improve these skills and progress to higher levels of learning
work with industry to ensure that vocational learning meets industry needs.
Education Performance Indicators (EPIs):
1. Qualification Completion Rate
2. Course Completion Rate3. Retention4. Progression from levels
1/2/3 to levels 4/5/6
Show EPI’s and league tables
for Polytechnics
and Universities
Linkages between objectives and measures …..
“The Government …...... expect to see better course and qualification completion and progression rates for students as a result of higher quality teaching and learning, and more effective and culturally responsive pastoral care.”
Guideline from management accounting research …. Care needs to be taken both to observe the measures actually in use, and also areas where measures are absent. It is a truism that what is measured tends to drive out what is not measured, and so omissions may be as influential as measures in use. (Ferreira & Otley, 2006)
Guideline from management accounting research ….
Was guideline from management accounting research followed….
There is insufficient explanation provided as to how the TEC’s EPIs were derived from its strategic objectives, eg by utilising strategy maps in order to link objectives to measures.
Guideline from management accounting research ….
it is as much the process of developing and using the CPM system, as it is the resultant performance measures that yield motivational benefits. To drive motivation the CPM system should be developed and used in a way that enhances the employees’ participation, psychological empowerment, and goal commitment (Bourne, Franco-Santos, & Lucianetti, 2012).
Was guideline from management accounting research followed….
There is little evidence to suggest that a consultative process took place with key stakeholders during their development.
Guideline from management accounting research ….
The evidence suggests that the information included in CPM systems is likely to increase the understanding of individuals regarding what is expected from them at work (Bourne, Franco-Santos, & Lucianetti, 2012).
Was guideline from management accounting research followed….
How educators have interpreted and responded to the implementation of the EPI’s is a potential area for further research
Guideline from education research ….
The lack of a demonstrated causal logic between the measures used in many of the university league tables for first-level degree programmes and student learning outcomes …… encourages the gaming behaviour and strategic mis-investments reported in the US (Dill & Soo, 2005).
Was guideline from education research followed….
An area that needs further research is looking at whether using EPIs to create league tables for universities and polytechnics has encouraged “gaming behaviour” amongst tertiary organisations in New Zealand.
Guideline from education research ….
The use of the results of performance measurement in league tables assumes that all those being measured start from the same point; however there are usually more sources of difference between same-service providers than there are similarities (Adcroft & Willis, 2005).
EPIs: Qualification completionOne completion:
University 1 One completion: University 2
Was guideline from education research followed….
Incongruent behaviourTEOs who select only the most academically able students, as shown by their secondary school achievements, effectively deny educational opportunities to those whose socio-economic demographic meant that they may never have had a chance to obtain similar secondary school achievements. (McClelland, 2012)
Recommendations
1. Outline how vision, key success factors and strategic objectives for the tertiary sector are linked.
2. Outline how the TEC’s EPIs were derived from its strategic objectives
3. Consider whether alternative or additional EPIs could be used in order to ensure that strategic objectives are met..
4. If EPIs are revised make use of a consultative process that ensures all key stakeholders are involved.
Recommendations (cont.)5. Consider what levers universities and
polytechnics can use with regard to improving the scores of their EPIs
6. Determine what expectations the EPIs communicate to tertiary organisations and their teaching staff, and whether these expectations are in line with the strategic plan for the tertiary sector.
7. Measure individual programmes, this will provide more meaningful information that enables effective benchmarking.
8. Consider providing information on employment rates of graduates for individual programmes.
