Building the evidence of system performance
What are we learning about the tertiary education system?
ATEM NZ Conference
10 July 2007
Page 2 New Zealand Ministry of Education
Lies, damned lies and statistics
▲ Government collects large amounts of data about education – and tertiary education in particular. From it, we can work out many facts.
▲ But facts without context are not much more than trivia. And data is useful only if it helps us to answer ‘important’ questions.
▲ The really important questions:
▲ how well do our students do in completing their studies?
▲ how well do they do after they leave study and in what ways?
▲ how much does our system add value?
▲ It is all about gathering evidence
Page 3 New Zealand Ministry of Education
This presentation
▲ Looks at the Ministry of Education’s tertiary education analytical programme
▲ Where it came from ▲ Current priorities▲ Some examples
▲ What do you want? And what is missing?
Page 4 New Zealand Ministry of Education
Building evidence of performance in tertiary education
▲ Government, institutions and students all have a stake.
▲ Need to improve study outcomes for many.
▲ Need to get good value for money.
▲ Policy shift from emphasis on participation to outcomes.
▲ Funding changes.
▲ Historically, lack of evidence has hampered our understanding.
Page 5 New Zealand Ministry of Education
Where we came from
▲ Unit record data has been collected since early 90s on parts of the tertiary education system
▲ From 2000, the collection was enhanced – with the SDR ▲ So there was a great and rich resource ▲ But little use was made of it, except internal analyses to underpin
some policy projects▲ With the 2002 reforms, the ministry needed to take a system view of
performance – and saw a greater need for public accountability information
▲ So we set about creating an evidence base from the available data ▲ to underpin policy▲ to provide accountability to public▲ and to support sector planning
Page 6 New Zealand Ministry of Education
Where we came from
▲ Where to start ….
▲ First, we analysed information and evidence needs▲ Conclusions
▲ a focus on performance ▲ and on outcomes▲ a system focus
▲ But to assess the performance of a system, we need to look at value created
▲ And that means primarily looking outside the system itself▲ what was a person’s status before▲ and what happens after leaving▲ plus something about how things work inside
Page 7 New Zealand Ministry of Education
Where we came from
▲ While we have great data, one issue was its usability▲ A key issue was the lack of links across time and across
providers▲ NSNs would eventually resolve that issue, but waiting
would take years▲ So in the meantime, statistical matching enabled us to do
completion rates
Page 8 New Zealand Ministry of Education
Where we came from
▲ But looking at SDR data – even with matching – didn’t give us outcomes data
▲ Especially given the need to assess performance by what happens after study
▲ The integrated dataset on Student Loan Scheme borrowers allows us to establish precise linkages between an individual’s education profile and post-study income – a good proxy for outcomes
▲ Although coverage is limited – it applies only to those who use the loan scheme – this was also a breakthrough
▲ But issues remain – data on industry training and targeted programmes is held separately – and how do we measure pre-entry status
▲ The NSN is now at a stage where we can start to address these issues
Page 9 New Zealand Ministry of Education
We set ourselves some goals
▲Our mission ▲ leadership in analysis ….▲ production and dissemination ….▲ timely, authoritative and relevant
• the tertiary education system • its interaction with the wider education sector • its contribution to New Zealand’s longer term social and economic
outcomes.
▲Our role▲ we look at a system that is made up of components▲ the TEC looks at components within a system▲ but we apply common or complementary analytical approaches to
common data
Page 10 New Zealand Ministry of Education
What is happening now
▲ We do summary reports▲ like Profile and Trends and TES monitoring▲ they are comprehensive▲ and summarise analyses
▲ And more detailed reports on aspects of the system▲ Mainly our approach is quantitative not qualitative▲ We focus on four themes
▲ Learners▲ Post-study outcomes▲ Supporting the system▲ Research and knowledge creation
▲ Education Counts has over 60 reports on many aspects of the system▲ Examples …
Page 11 New Zealand Ministry of Education
What is happening now
Page 12 New Zealand Ministry of Education
What is happening now
More examples▲ Who doesn’t pay back their loans?▲ What determines the research performance of staff? ▲ How long do people spend in tertiary education?▲ A changing population and the NZ tertiary sector▲ Participation in tertiary education – a birth cohort approach▲ He tini manu reo▲ Maori student performance in degrees
Page 13 New Zealand Ministry of Education
So what do we find?
▲ Some examples▲ Post-study outcomes▲ Passing courses and completing qualifications▲ Research performance
Page 14 New Zealand Ministry of Education
Outcomes – estimated earnings by level and gender
0
10
20
30
40
Leve
l 1-3
cer
tific
ates
Leve
l 4 c
ertif
icat
es
Leve
l 5-7
dip
lom
as
Leve
l 7 b
ache
lors
Leve
l 8-1
0ho
nour
s/m
aste
rs/d
octo
rate
s
Leve
l 1-3
cer
tific
ates
Leve
l 4 c
ertif
icat
es
Leve
l 5-7
dip
lom
as
Leve
l 7 b
ache
lors
Leve
l 8-1
0ho
nour
s/m
aste
rs/d
octo
rate
s
3 y ears after study 5 y ears after study
Estim
ated
ear
ning
s $
(000
)
Female Male
Page 15 New Zealand Ministry of Education
Outcomes – estimated earnings 5 years post-study by level and ethnic group
0
10
20
30
40
Lev el 1-3 certificates Lev el 4 certificates Lev el 5-7 diplomas Lev el 7 bachelors Lev el 8-10
honours/masters/doctorates
Estim
ated
ear
nings
$ (0
00)
Asian European Māori Other Pasifika
Page 16 New Zealand Ministry of Education
So what do we find? Completions
▲ 50%-55% completion rate for bachelors degree.▲ 80% pass rate for bachelors courses.▲ Rates are higher at postgraduate level and lower at sub-
degree level.▲ 25% pass all their courses without gaining a qualification.▲ Full-time students have much higher rates, even allowing
for the extra time part-timers need to complete.▲ NZ compares below average internationally, but
reasonably well when adjusted for its high level of part-time study.
Page 17 New Zealand Ministry of Education
So what do we find? Completions
Completion rates for 1995 starters over time
0%
10%
20%
30%
40%
50%
60%
70%
1 2 3 4 5 6 7 8 9 10 11
Years in study
Bachelors PhDs
Page 18 New Zealand Ministry of Education
Bibliometric analysis of university research
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Relat
ive im
pact
Biological sciences Medicine & public health Health
Page 19 New Zealand Ministry of Education
Bibliometrics vs PBRF scoresBiological sciences
0
1
2
3
4
5
6
0 10 20 30 40 50 60 70 80
Citations per FTE
PB
RF
ave
rage
qua
lity
scor
e
AUT
LN VUW MY
WK
CY OT
AK
Page 20 New Zealand Ministry of Education
Where are we going?
▲ Breakthroughs in data infrastructure lead to analytical breakthroughs▲ Links between agencies’ datasets▲ Broadening of integrated data
▲ Some projects▲ More on research and on the PBRF
• How has the PBRF affected attrition in research degrees?• Statistical analyses of shifts
▲ Industry training▲ Transitions – how does school performance impact on tertiary
participation and on tertiary achievement
▲ Improving statistics – quality, timeliness and usability
Page 21 New Zealand Ministry of Education
The challenge for the future
▲ As we break new ground, we shift the frontier▲ We are changing the way we look at and talk about
tertiary education▲ And the new Tertiary Education Strategy – what demands
is that making?▲ Hard to get at information▲ For instance ….
▲ Improving research connections and linkages …▲ … to create economic opportunities▲ Increasing language, literacy and numeracy levels for the
workforce
Page 22 New Zealand Ministry of Education
And how can you help
▲ What do you need to know?▲ What is missing?
Page 23 New Zealand Ministry of Education
▲ Our task – build the evidence base▲ Our achievement
▲ lots more material▲ better statistics▲ better analysis
▲ But a way to go▲ Our challenge
▲ data infrastructure▲ improve the relevance of the analyses▲ the hard questions
http://educationcounts.edcentre.govt.nz/
Top Related