Building the evidence of system performance What are we learning about the tertiary education...
-
Upload
alyson-howeth -
Category
Documents
-
view
213 -
download
1
Transcript of Building the evidence of system performance What are we learning about the tertiary education...
![Page 1: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/1.jpg)
Building the evidence of system performance
What are we learning about the tertiary education system?
ATEM NZ Conference
10 July 2007
![Page 2: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/2.jpg)
Page 2 New Zealand Ministry of Education
Lies, damned lies and statistics
▲ Government collects large amounts of data about education – and tertiary education in particular. From it, we can work out many facts.
▲ But facts without context are not much more than trivia. And data is useful only if it helps us to answer ‘important’ questions.
▲ The really important questions:
▲ how well do our students do in completing their studies?
▲ how well do they do after they leave study and in what ways?
▲ how much does our system add value?
▲ It is all about gathering evidence
![Page 3: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/3.jpg)
Page 3 New Zealand Ministry of Education
This presentation
▲ Looks at the Ministry of Education’s tertiary education analytical programme
▲ Where it came from ▲ Current priorities▲ Some examples
▲ What do you want? And what is missing?
![Page 4: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/4.jpg)
Page 4 New Zealand Ministry of Education
Building evidence of performance in tertiary education
▲ Government, institutions and students all have a stake.
▲ Need to improve study outcomes for many.
▲ Need to get good value for money.
▲ Policy shift from emphasis on participation to outcomes.
▲ Funding changes.
▲ Historically, lack of evidence has hampered our understanding.
![Page 5: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/5.jpg)
Page 5 New Zealand Ministry of Education
Where we came from
▲ Unit record data has been collected since early 90s on parts of the tertiary education system
▲ From 2000, the collection was enhanced – with the SDR ▲ So there was a great and rich resource ▲ But little use was made of it, except internal analyses to underpin
some policy projects▲ With the 2002 reforms, the ministry needed to take a system view of
performance – and saw a greater need for public accountability information
▲ So we set about creating an evidence base from the available data ▲ to underpin policy▲ to provide accountability to public▲ and to support sector planning
![Page 6: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/6.jpg)
Page 6 New Zealand Ministry of Education
Where we came from
▲ Where to start ….
▲ First, we analysed information and evidence needs▲ Conclusions
▲ a focus on performance ▲ and on outcomes▲ a system focus
▲ But to assess the performance of a system, we need to look at value created
▲ And that means primarily looking outside the system itself▲ what was a person’s status before▲ and what happens after leaving▲ plus something about how things work inside
![Page 7: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/7.jpg)
Page 7 New Zealand Ministry of Education
Where we came from
▲ While we have great data, one issue was its usability▲ A key issue was the lack of links across time and across
providers▲ NSNs would eventually resolve that issue, but waiting
would take years▲ So in the meantime, statistical matching enabled us to do
completion rates
![Page 8: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/8.jpg)
Page 8 New Zealand Ministry of Education
Where we came from
▲ But looking at SDR data – even with matching – didn’t give us outcomes data
▲ Especially given the need to assess performance by what happens after study
▲ The integrated dataset on Student Loan Scheme borrowers allows us to establish precise linkages between an individual’s education profile and post-study income – a good proxy for outcomes
▲ Although coverage is limited – it applies only to those who use the loan scheme – this was also a breakthrough
▲ But issues remain – data on industry training and targeted programmes is held separately – and how do we measure pre-entry status
▲ The NSN is now at a stage where we can start to address these issues
![Page 9: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/9.jpg)
Page 9 New Zealand Ministry of Education
We set ourselves some goals
▲Our mission ▲ leadership in analysis ….▲ production and dissemination ….▲ timely, authoritative and relevant
• the tertiary education system • its interaction with the wider education sector • its contribution to New Zealand’s longer term social and economic
outcomes.
▲Our role▲ we look at a system that is made up of components▲ the TEC looks at components within a system▲ but we apply common or complementary analytical approaches to
common data
![Page 10: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/10.jpg)
Page 10 New Zealand Ministry of Education
What is happening now
▲ We do summary reports▲ like Profile and Trends and TES monitoring▲ they are comprehensive▲ and summarise analyses
▲ And more detailed reports on aspects of the system▲ Mainly our approach is quantitative not qualitative▲ We focus on four themes
▲ Learners▲ Post-study outcomes▲ Supporting the system▲ Research and knowledge creation
▲ Education Counts has over 60 reports on many aspects of the system▲ Examples …
![Page 11: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/11.jpg)
Page 11 New Zealand Ministry of Education
What is happening now
![Page 12: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/12.jpg)
Page 12 New Zealand Ministry of Education
What is happening now
More examples▲ Who doesn’t pay back their loans?▲ What determines the research performance of staff? ▲ How long do people spend in tertiary education?▲ A changing population and the NZ tertiary sector▲ Participation in tertiary education – a birth cohort approach▲ He tini manu reo▲ Maori student performance in degrees
![Page 13: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/13.jpg)
Page 13 New Zealand Ministry of Education
So what do we find?
▲ Some examples▲ Post-study outcomes▲ Passing courses and completing qualifications▲ Research performance
![Page 14: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/14.jpg)
Page 14 New Zealand Ministry of Education
Outcomes – estimated earnings by level and gender
0
10
20
30
40
Leve
l 1-3
cer
tific
ates
Leve
l 4 c
ertif
icat
es
Leve
l 5-7
dip
lom
as
Leve
l 7 b
ache
lors
Leve
l 8-1
0ho
nour
s/m
aste
rs/d
octo
rate
s
Leve
l 1-3
cer
tific
ates
Leve
l 4 c
ertif
icat
es
Leve
l 5-7
dip
lom
as
Leve
l 7 b
ache
lors
Leve
l 8-1
0ho
nour
s/m
aste
rs/d
octo
rate
s
3 y ears after study 5 y ears after study
Estim
ated
ear
ning
s $
(000
)
Female Male
![Page 15: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/15.jpg)
Page 15 New Zealand Ministry of Education
Outcomes – estimated earnings 5 years post-study by level and ethnic group
0
10
20
30
40
Lev el 1-3 certificates Lev el 4 certificates Lev el 5-7 diplomas Lev el 7 bachelors Lev el 8-10
honours/masters/doctorates
Estim
ated
ear
nings
$ (0
00)
Asian European Māori Other Pasifika
![Page 16: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/16.jpg)
Page 16 New Zealand Ministry of Education
So what do we find? Completions
▲ 50%-55% completion rate for bachelors degree.▲ 80% pass rate for bachelors courses.▲ Rates are higher at postgraduate level and lower at sub-
degree level.▲ 25% pass all their courses without gaining a qualification.▲ Full-time students have much higher rates, even allowing
for the extra time part-timers need to complete.▲ NZ compares below average internationally, but
reasonably well when adjusted for its high level of part-time study.
![Page 17: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/17.jpg)
Page 17 New Zealand Ministry of Education
So what do we find? Completions
Completion rates for 1995 starters over time
0%
10%
20%
30%
40%
50%
60%
70%
1 2 3 4 5 6 7 8 9 10 11
Years in study
Bachelors PhDs
![Page 18: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/18.jpg)
Page 18 New Zealand Ministry of Education
Bibliometric analysis of university research
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Relat
ive im
pact
Biological sciences Medicine & public health Health
![Page 19: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/19.jpg)
Page 19 New Zealand Ministry of Education
Bibliometrics vs PBRF scoresBiological sciences
0
1
2
3
4
5
6
0 10 20 30 40 50 60 70 80
Citations per FTE
PB
RF
ave
rage
qua
lity
scor
e
AUT
LN VUW MY
WK
CY OT
AK
![Page 20: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/20.jpg)
Page 20 New Zealand Ministry of Education
Where are we going?
▲ Breakthroughs in data infrastructure lead to analytical breakthroughs▲ Links between agencies’ datasets▲ Broadening of integrated data
▲ Some projects▲ More on research and on the PBRF
• How has the PBRF affected attrition in research degrees?• Statistical analyses of shifts
▲ Industry training▲ Transitions – how does school performance impact on tertiary
participation and on tertiary achievement
▲ Improving statistics – quality, timeliness and usability
![Page 21: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/21.jpg)
Page 21 New Zealand Ministry of Education
The challenge for the future
▲ As we break new ground, we shift the frontier▲ We are changing the way we look at and talk about
tertiary education▲ And the new Tertiary Education Strategy – what demands
is that making?▲ Hard to get at information▲ For instance ….
▲ Improving research connections and linkages …▲ … to create economic opportunities▲ Increasing language, literacy and numeracy levels for the
workforce
![Page 22: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/22.jpg)
Page 22 New Zealand Ministry of Education
And how can you help
▲ What do you need to know?▲ What is missing?
![Page 23: Building the evidence of system performance What are we learning about the tertiary education system? ATEM NZ Conference 10 July 2007.](https://reader031.fdocuments.in/reader031/viewer/2022013100/551b058c5503462e578b546a/html5/thumbnails/23.jpg)
Page 23 New Zealand Ministry of Education
▲ Our task – build the evidence base▲ Our achievement
▲ lots more material▲ better statistics▲ better analysis
▲ But a way to go▲ Our challenge
▲ data infrastructure▲ improve the relevance of the analyses▲ the hard questions
http://educationcounts.edcentre.govt.nz/