The NDIS Evaluation Two Years On...The NDIS Evaluation Two Years On Kostas Mavromaras (Chief...
Transcript of The NDIS Evaluation Two Years On...The NDIS Evaluation Two Years On Kostas Mavromaras (Chief...
The NDIS Evaluation Two Years On
Kostas Mavromaras
(Chief Investigator of the NDIS trial Evaluation)
NGO Forum
Council of Social Service of NSW (NCOSS) Sydney 26 August 2015
The evaluation is funded by the Australian Government Department of Social Services (DSS) and is being carried out by an independent research team, led by the National Institute of Labour Studies (NILS) at Flinders University.
Background and acknowledgements
Flinders University was commissioned in 2013 to carry out an independent evaluation of the impact of the NDIS in the trial sites (excluding WA)
The evaluation started in May 2013 and will be completed in June 2017
The work is funded exclusively by the Australian Government Department of Social Services
The Team
• Led by the National Institute of Labour Studies (NILS)
and supported by a wide range of expertise:
• Five national State Experts in disability (including Sydney
and LaTrobe Universities)
• Three International Experts in disability, disability insurance,
and evaluation (Michigan, Maryland and Nottingham
Universities)
• Two Survey Companies: the Social Research Centre and
Iview/Ipsos
• Flinders University (Adelaide): Disability and Community
Inclusion Unit (DCIU); and Flinders Northern Territory
A Roadmap of the talk
• A description of the NDIS Evaluation and its
objectives
• The components of the NDIS Evaluation
• Putting the evidence together
• Lessons and highlights
Two main purposes of the talk
• We do not wish to influence the policy (this
would contaminate our results/conclusions)
• But Wave 2 starts soon and we do want to
make the Evaluation known to get the widest
possible representation of all voices
• We also wish to be continually informed
about the relevant thinking
• So, your ideas and comments please! (But we
will not be responding with opinions or results.)
Some of the Evaluation questions
• Is the NDIS bringing change?
• Where is it doing it?
• What is working well and what is not?
• Lessons for the broader scheme?
• Several different aspects…
• Several different viewpoints…
• 6 trial sites: VIC, NSW, TAS, SA, ACT, NT
Some of the Evaluation questions (cont.)
• Not measuring costs and funding
• Not about cost-shifting or similar issues
• Not about the NDIS financial viability or feasibility
• We want to know what all concerned (including people with disability, their families and carers, the providers of supports and their workers) think about the introduction of the NDIS
• Is it making their lives better? Is it closing the gap?
Ask what is happening (Because of the NDIS):
• Is the wellbeing and participation of people with disability and their families and carers changing?
• Is the demand for provisions changing?
• Will there be the providers to deliver?
• Will providers have the workforces to deliver?
• Is the NDIS serving the least vocal people with disability?
• Will early interventions become more prevalent?
The methodology
• We collect evidence from NDIS participants in all trial sites (the “trial” group)
• We collect evidence from NDIS potential participants not in the trial sites (the “comparison” group)
• We collect evidence twice (2014/15 & 2015/16) from the same participants (approx. one year apart) and measure how things have changed
• For similar people in the trial and comparison groups, the difference in the 2014/15-to-2015/16 change experienced by the two groups can be attributed to the NDIS
• This is called the Difference-in-Difference method
The methodology
• The D-in-D method measures the change experienced by two groups between a starting point in time and a later point in time
• It measures this change for two as similar as possible groups, but only the one group is exposed to the NDIS and the other is not
The methodology (employment as an example)
• There are many reasons why employment may change for people with disability, including what is introduced by the NDIS
• Suppose we find higher wellbeing and lower employment participation between wave 1 and wave 2 data collections
• The start of the D-in-D method measures the change experienced by two groups between a starting point in time and a later point in time
• It measures this change for two as similar as possible groups, one group having joined the NDIS and the other has not
The methodology (cont.)
• Multidisciplinary
• The role of matching
• The delights of phasing-in
• The advantages of the D-in-D method
• The role of multivariate analysis
• The delights of messy data
How do we handle this complexity?
• By collecting lots of complementary evidence big in scale and wide in scope and using a rigorous evaluation method
• Broad representative evidence from new large national surveys
• Deep and granular evidence from new in-depth qualitative studies
• Support from several national administrative data sets
• Wide expertise in disability studies, sociology, social work, evaluation, survey design, economics and statistics
• The combined evidence is analysed and evaluated by a team of national and international experts to produce a number of reports for the DSS
The main components of the NDIS Evaluation (2013-2017)
• Stakeholder Engagement Strategy
• Ethics, accessibility, privacy, age-appropriate design
• Survey of people with disability, their families and carers
(2014/15 and 2015/16)
• Survey of disability support providers, their workforce
and self employed (2014/15 and 2015/16 for all trial
sites, 2014/15 only for QLD)
• Comprehensive set of qualitative studies (2014 to 2016)
What do we do with this evidence?
• Merging of qualitative and quantitative evidence
• Statistical and econometric analysis of the evidence
• Evaluation of the impact of the NDIS on people with disability, their Families and Carers
• Assessment of the impact of the NDIS on the relevant providers and their workforces
• Synthesis and reporting
Engaging stakeholders: This is a
continuous process aimed to keep on
informing all other components
throughout the Evaluation
Stakeholder Engagement Strategy (SES)
To encourage the active participation of:
People with disability, their families & carers
Disability sector, advocacy & peak
organisations
To ensure primarily that stakeholders’ views
inform the research, but also that stakeholders
are informed about the research
State Experts are working in the trial sites/states
Information dissemination through various
methods including www.ndisevaluation.net.au
Asking the question:
What do people with disability, their
families and carers think about the NDIS?
The NDIS Survey of People with Disability, their Families and Carers
• Two waves to be conducted in trial & comparison sites
• Pilot Survey completed in May 2014
• Extensive research and testing
• Registration is continuing as we speak (August 2015)
• Wave 1 is presently in the field (ACT trial and evaluation
start later)
• Wave 2 planned for early 2016 fieldwork (approx. 10-12
months after Wave 1)
NDIS Survey of People with Disability
• Much new thinking and design has gone into:
• How we get as many people with disability as
possible to respond
• Providing choice on how to complete the survey
(online, CATI, hard copy, face to face)
• Making the questionnaires as accessible as possible
• Making sure that the opinions will be those of the
people with disability (see next slide in Part B)
NDIS Survey of People with Disability (cont.)
• Part A: Elicits mainly factual information (Proxy or person with disability)
• Part B: Elicits mainly opinions, perceptions, aspirations and assessments (only person with disability) Pictorial Easy English option where requested
Part A Part B
Disability & Supports Social Participation &
Well-Being
Supports before & since
the NDIS
Education & Employment
Aspirations
Socio-demographics Living & Housing
Needs/Aspirations
Keeping in Touch Opinions on Disability Supports
NDIS Survey of Families and Carers of
People with Disability
• Information collected includes:
• Caring – the type/access/frequency of supports
• Experiences pre- and post-NDIS
• Social participation & wellbeing
• Socio-demographics, including family circumstances
• Caring for children less than 8 years old
These are major data collections: where necessary,
interviews last over 40 minutes and questionnaires
are between 20 and 35 pages long…
NDIS Survey: Highlights/Innovation
• Independent pre-survey registration process provides choice of different completion modes to suit diversity and enhance inclusion
• We offer:
• Hard copy
• CATI
• Online
• Face to face
• All modes are well received (except face to face, which was rarely chosen as the preferred mode)
NDIS Survey: Highlights/Innovation
• Lower reliance on proxies through the use of Pictorial Easy English questionnaires for “Part B” (participation, wellbeing, aspirations, needs)
• Extensive use of experts in accessibility and inclusion
The objective is to allow the voice of the person with disability to be heard as clearly as possible (and as distinct as possible from their proxy’s voice)
Pictorial Part B has been well received
NDIS Surveys of People with Disability, and
their Families and Carers
• The survey of people with disability is well advanced
• The survey of families and carers of the people with
disability is not as advanced, but well under way
Initial response rates may me idiosyncratic: lower than
expected for “units”, but much higher for “items”. Can it
be that the survey does not look as worthwhile from the
outside, but, once started, the experience is good?
Asking the question:
Will we have the providers and the
workforces for the full NDIS roll out?
Survey of Disability Support Providers
and their Workforce
• Workplaces, workers, self-employed providers in the
NDIS trial states (2 waves). We collect information on:
• Staffing, skills & training, recruitment
• Revenue & sources of revenue
• Supports offered (incl. technological innovation)
• Client profile
• Quality assurance and performance monitoring
• Experience of NDIS & future plans
What about the depth of information that
cannot be captured by large scale
quantitative surveys?
Why Qualitative Evidence?
“BIG” quantitative surveys do not handle aspects of
diversity and accessibility with sufficient granularity
Qualitative studies offer more depth and flexibility:
they offer more depth and granularity
they can ask questions that surveys cannot
can facilitate otherwise difficult interpretations
Qualitative is asked in two-three waves (2014/15/16)
Qualitative evidence is built to be combined with our
survey evidence (mixed methods analysis)
What Qualitative Evidence?
People with disability, their families and carers
One-to-One interviews
Focus groups with eligible non-participants and non-
eligible non-participants
Face-to-Face, telephone, online, email
• Sector representatives
• Disability workforce agencies, government
agencies, NDIA, unions, peak bodies
• Specialist disability service providers
(2nd wave beginning now!)
Several important and unique aspects of
the NDIS Evaluation are covered with
additional studies.
The “special” components of the Evaluation
• Northern Territory – a separate evaluation (two waves in 2015 and 2016)
• Older people with disability: aged care/NDIS (one wave 2015/16 only)
• Mainstream providers (one wave 2015/16)
• Administrative data analysis (continuous 2013-2016)
The Northern Territory
• Small sample, remote location study, requires an
individual and flexible evaluation design
• A two-wave, large scale, qualitative study in the
Barkly region (trial) and Alice Springs (comparison)
• NDIS participants and their families and carers
• Providers of supports and peak bodies
• NDIA staff
• Culturally respectful and sensitive approach,
supported by Flinders University NT.
• Started preparations in June 2014, fieldwork in 2015
and will finish in 2016
The Older People with Disability Study
• We will compare supports and satisfaction of older
NDIS participants with older people with similar
disability but not eligible for the NDIS
• People with disability between the ages of 63 and 70
• Comparing experiences of NDIS participants aged
between 59-64 with non-participants aged between
64-70 not in trial sites and some in residential aged
care facilities
• Small samples require an in-depth qualitative study
• One Wave of interviews in 2015
The Mainstream Providers Study
• Services already provided to the general public,
which also need to be available to People with
Disability, separate to the services provided by
the NDIS.
• Content development in progress
• Expected commencement in 2015
How is the whole Evaluation project
travelling?
• Started in June 2013
• Design second half of 2013
• Piloting first half of 2014 with further design
• Wave 1 fieldwork started very end of 2014 and
continues
• Wave 2 design and start second half of 2015
• Wave 2 second half of 2015 and 2016
• Analysis and final reporting second half of 2016 and
first half of 2017