A FOCUS ON QUALITY OUTCOMES
DAO and WANADASession 1st May 2012
ANITA GHOSE B. App Sci; MBA; MAICD
DIRECTORAnita Ghose and Associates
AGENDA1. An overview of outcomes
2. Developing outcomes
3. Measuring outcomes
1. An Overview of Outcomes
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium
What we invest
What we do
Who we reach
What are the results?
What changed?
Long-term
Eg:MoneyStaff timeFacilitiesEquipment
Eg:Provide counsellingProvide diversion servicesProvide residential rehab services
What we produce
Eg:Number of hours of service deliveredNumber of counselling sessions providedNumber of people in residential rehab services
Eg:People who secured stable housingImproved attitudes towards cannabis useIncreased knowledge of the effects of alcohol and drug useReduction in harmful use of alcohol and drugs
REFRESHER – KEY CONCEPTS
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium
What we invest
What we do
Who we reach
What are the results?
What changed?
Long-term
What we produce
Based on Productivity Commission Report 2010; United Way of America (1996)
PROCESSES – eg waiting lists, response times, client/staff ratios
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium
What we invest
What we do
Who we reach
What are the results?
What changed?
Long-term
What we produce
Based on Productivity Commission Report 2010; United Way of America (1996) Population
Inputs—the resources used by the organisation to achieve the objectives of an activity – “what we invest” eg money, staff time, facilities, equipment
Processes – the related tasks and systems of the organisation to fulfil its mission (eg. response times, staff trained, worker safety, client/staff ratio, length of wait lists)
Outputs— the direct products of the organisation’s activities or intervention (“what we produce”) and the participation (“who did we reach”). Eg number of service hours delivered, number of counselling sessions provided, number of clients in residential rehabilitation services
Outcomes—the effects or change on an individual or group as a result of the activities and outputs (“what was the result?) Eg people who secured stable housing, reduction in harmful use of alcohol and drugs, increased knowledge of the effects of cannabis
Impacts—the broader effects of an activity reflected in community wellbeing domains (what changed for the population?) Eg. reduced unemployment, increased mental health, stronger families
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium
What we invest
What we do
Who we reach
What are the results?
What changed?
Long-term
What we produce
The effects or change
as a result of
the activities
or outputs
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium
What we invest
What we do
Who we reach
What are the results?
What changed?
Long-term
Eg:MoneyStaff timeFacilitiesEquipment
Eg:Provide counsellingProvide diversion servicesProvide residential rehab services
What we produce
Eg:Number of hours of service deliveredNumber of counselling sessions providedNumber of people in residential rehab services
Eg:People who secured stable housingImproved attitudes towards cannabis useIncreased knowledge of the effects of alcohol and drug useReduction in harmful use of alcohol and drugs
Government Agencies are moving here in OBP
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium
What we invest
What we do
Who we reach
What are the results?
What changed?
Long-term
Eg:MoneyStaff timeFacilitiesEquipment
Eg:Provide counsellingProvide diversion servicesProvide residential rehab services
What we produce
Eg:Number of hours of service deliveredNumber of counselling sessions providedNumber of people in residential rehab services
Eg:People who secured stable housingImproved attitudes towards cannabis useIncreased knowledge of the effects of alcohol and drug useReduction in harmful use of alcohol and drugs
This isn’t being prescribed by Govt – BUT organisations do need to understand their business here
OUTCOMESThere will generally be effects, results, changes in one or more of these core areas:
KNOWLEDGE OR SKILLS
ATTITUDE OR OPINION
VALUES
BEHAVIOUR
CIRCUMSTANCE
CONDITION OR STATUS
OUTCOMES
KNOWLEDGE OR SKILLS - increased knowledge of the effects of alcohol and drug use on the body
ATTITUDE OR OPINION – improved attitudes or opinion about the dangers of cannabis use
VALUES - enhanced values on the impact of drug use on others around the person
BEHAVIOUR – reduction in the harmful use of drugs and alcohol
CIRCUMSTANCE – increased ability to secure and maintain stable housing
CONDITION OR STATUS – improved general health and wellbeing
OUTCOMES – KEY CONSIDERATIONS
Proximal/DistalAttribution/ContributionPathways approachLink to outputsReasonable numberPerverseMeasurement versus achievementPositive or negativeIntended or unintended
EXAMPLE
Community Drug Service Team
Counselling model
Proximal outcome:
Amphetamine users have reduced their drug and alcohol useAmphetamine users have an improved level of mental health and wellbeing
Distal outcome:Amphetamine users have reduced levels of offending
EXAMPLE
Mother and Child Program
Specialist support
Proximal outcome:
Mothers reduce their alcohol and drug useMothers demonstrate an increased awareness of the harmful effects of alcohol and drug use on their childMothers remain the main caregiver of their child
Distal outcome:Mothers participate more in their children’s schooling program
How efficient is the organisation?
How effective is the organisation?
Efficient and effective
INPUTS OUTPUTS OUTCOMES
Resources Used
Activities
Products
Participation Short Medium Long-term
Efficiency = Inputs Outputs
Example: the average cost per session of counselling delivered
Effectiveness = Inputs Outcome
Example: the average cost per completed as planned episode
Outcomes Frameworks
A Logic Model
Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation
Is a framework for describing the relationships between investments, activities, and results.
Depicts a program showing what the program will do and what it is to accomplish (reasoning)
Provides a series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes
Provides a common approach for integrating planning, implementation, evaluation and reporting.
Applied at a program, process, multi component program, organisational level
Fully detailed logic modelFully detailed logic model
Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation
Logic model and reporting
Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation
RESULTS BASED ACCOUNTABILITY
Strong and stable families, positive family relationships
Rate of family breakdown due to alcohol and drug use
Rate of domestic violence
1. How much did we do? 2. How well did we do it?
3. Is anyone better off?
RESULT or OUTCOME
INDICATOR or BENCHMARK
PERFORMANCE MEASURE
A condition of well-being for children, adults, families or communities.
A measure which helps quantify the achievement of a result.
A measure of how well a program, agency or service system is working. Three types:
= Customer Results
Popu
latio
nPe
rfor
man
cePo
pula
tion
Perf
orm
ance
= Customer Results
Source: Mark Friedman 2005
How much did we do? How well did we do it?
Is anyone better off?
# Clients/customers served
# Activities (by type of activity)
% Common measurese.g. client/staff ratio, unit cost, clients from a target group, staff turnover, staff morale, staff fully trained, worker safety, client satisfaction (were they treated well), wait time, wait lists/turnaways, meeting specific industry standards
% Skills / Knowledge (e.g. parenting skills)
#
% Attitude / Opinion (e.g. toward drugs)
#
% Behaviour (eg drug and alcohol use)
#
% Circumstance (e.g. working, in stable housing)
#
% Activity-specific measures
e.g. timely, clients completing activity, correct and complete processes, meeting standards
Source:Mark Friedman 2005;Anglicare 2011
EFFECTEFFO
RT
QUANTITY QUALITY
How Population
&Performance Accountability
FIT TOGETHER
Contributionrelationship
Alignmentof measures
Appropriateresponsibility
THE LINKAGE Between POPULATION and PERFORMANCE
POPULATION ACCOUNTABILITY
General Physical Health Prevalence of liver diseaseStable Families and Relations Rate of alcohol and drug use Mental Health Rate of co-occurrence with mental health issues
# ofbed days
% receiving assessment withinagreed timeframe
# increased awareness of the impact of alcohol and drug use on physical health
PERFORMANCE ACCOUNTABILITY
POPULATIONRESULTS
Residential Rehab Services
Source:Mark Friedman 2005
% increased awareness of the impact of alcohol and drug use on physical health
2. Developing Outcomes
The Request
Key Assumptions
Partnership and collaboration between agency, organisation and individuals, families to identify community outcomes
Appropriate consultation framework and engagement strategies with key stakeholders
Appropriate mechanisms in place (formal and informal) to capture the information required to get the outcomes and time frames right
This is the stage of developing and identifying outcomes which is outside of scope for this workshop but is the important precursor to defining and writing outcomes
Individual outcomes for people with alcohol and drug issues
Family outcomes
Program or Service level outcomes
Organisational outcomes
Corporate – Financial; Human resource; Risk
Community outcomes
Population outcomes
procurement
Outcomes - different levels and purposes
Types of Levels
Outcomes – Responding to a Request
Key considerations:
•Statement of the desired change or result (what does it look like?)
•Interpretation – get clarity and agreed understanding across stakeholders
•Be clear about target group (eg child, mother, parent)
•Outcomes – at what level (eg individual, service level, population)
•What are you trying to assess?
•Measurable and meaningful; reasonable number
Outcomes – Responding to a Request
Key considerations:
•Attribution – circle of control and influence; proximal and distal
•System – how does this fit within the whole? Communities of practice
•Think about time frame – short term to long term; longitudinal change
•Consider context eg chronic relapsing condition
•Outcomes in the AOD sector are generally about results/effects or change in knowledge or skills; attitude or opinion; values; behaviour; circumstance; condition or status
A useful approach to writing an outcome statement
SMART objectives – specific, measurable, achievable, results oriented, time frame
Who/what Change/desired effect
In what By when
Parents of people with alcohol and drug issues participating in the parenting program
increase their knowledge of alcohol and drug use
at the completion of the program
Young people in a specialist alcohol and drug treatment centre
reduce their harmful alcohol and drug use
after the first year of the program
3. Measuring Outcomes
Indicator - is a measure for which data is available which provides a specific and reliable means to quantify the achievement of the outcome. These results indicators are what we will see if change occurs.
Ask the questions: What will we see if change occurs?What will be the evidence?
Each outcome can have a set of indicators, which, if taken together gives a reasonable approximation of the achievement of the outcome or result
Indicator – can be expressed as a # or % (quantifies the achievement of the outcome)
Eg: The percentage of participants reporting increased awareness about alcohol and drugs
The number of people who have secured stable housing since the programme’s commencement
The percentage of mothers who have retained primary care of their children since the program’s completion
Sometimes, one indicator may be sufficient to evidence the outcome’s achievement.
Often, several indicators are required. Select carefully to be meaningful and not over burdensome.
Outcome: SOBER UP CENTRE – Clients are kept safe overnight
Reduction in complaints about public drunkeness
Reduction in incidence of domestic violence Reduction in street arrests
of clients
Reduction in clients attending A&E depts in hospital
About measurement tools
There is no magical measurement tool!
There are a range of tools (eg. clinical profiling tools, instruments, surveys and questionnaires etc) that can be used but they all have limitations
Develop an “evidencing the outcome” approach that is context specific, in collaboration with the sector organisations, individuals and families which can give a reasonable approximation of the achievement of the outcome
There is always a story, assumptions and a context which needs to be explicitly stated
Reliability refers to the extent to which a measure produces accurateinformation.
Validity refers to the extent to which the measure actually measures what it isintended to measure.
Suitability refers to the criteria for selecting measures that meet the target group or intent – and are appropriate for those administering it
Relevancy refers to measures clearly related to the outcome being evaluated, providing useful information and allowing meaningful conclusions to be drawn
THIS IS A JOURNEY – CONSIDER THESE AND WORK TOWARDS
We need a baseline (pre-post)
Outcomes, Indicators, DataOutcomes, Indicators, Data
Outcomes Indicators Data collection
Sources Methods Sample Timing
Linkages and relationship, alignment, what is important, reducing administrative burdenLinkages and relationship, alignment, what is important, reducing administrative burden
Evidence based practice
SUMMARY OF THE DAY1. An overview of outcomes
2. Developing outcomes
3. Measuring outcomes
QUESTIONS
Consultants to Government and Community
Top Related