Youth Programs Development of a Performance Management ...

22
Youth Programs Development of a Performance Management Framework Consultation Paper February 2005

description

 

Transcript of Youth Programs Development of a Performance Management ...

Page 1: Youth Programs Development of a Performance Management ...

Youth ProgramsDevelopment of a Performance ManagementFrameworkConsultation Paper

February 2005

Page 2: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 2RPR Consulting, February 2005

Table of Contents

1 Developing a youth programs performance management framework –introduction to the project 3

1.1 Introduction 3

1.2 The purpose of the project 3

1.3 What is a performance management framework? 4

1.4 How is the youth performance management framework being developed? 5

2 Improving performance management 7

2.1 What is FaCS trying to achieve through the programs it funds? 7

2.2 What are the youth programs working to achieve? 7

2.3 What is FaCS role in performance management? 8

2.4 What needs to be protected in these programs? 9

2.5 What needs improving in the way performance is managed? 9

2.6 Proposed criteria to judge the framework’s success 12

3 Developing the youth performance framework 13

3.1 Performance expectations of individual services 13

3.2 Assessing the performance of the programs 18

3.3 Developing the methods for reporting 19

3.4 Next steps 20

4 Glossary of terms 21

Page 3: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 3RPR Consulting, February 2005

1 Developing a youth programs performancemanagement framework – introduction to the project

1.1 IntroductionThis consultation paper has been written to give organisations running Reconnect andYouth Activity Services/Family Liaison Worker (YAS/FLW) services some background toa project that will develop a Performance Management Framework for the two programs.Consultation forums will be held across Australia in March. Reading this paper will helpservice providers to think about the issues in advance so they can participate fully in theseforums.

RPR Consulting has been contracted to work with FaCS and funded services on theproject. RPR has worked extensively with both the Reconnect Program and the YAS/FLWProgram and has a good understanding of the issues faced by service providers indelivering services and in reporting to FaCS.

1.2 The purpose of the projectRecent evaluations by the Youth Bureau of the Reconnect and Youth ActivityServices/Family Liaison Worker programs have confirmed the value of both programs.They are seen as an integral to the capacity of FaCS to achieve its own high level outcomesof building stronger families, stronger communities, and helping individuals reach theirfull potential.

While the evaluations found that both programs are making important contributions to thewell-being of individual young people and their families as well as to the communities theyserve, evidence to back up these findings was not always easily accessed. Serviceproviders have generated and contributed a great deal of information about the program viadata collection, forums, action research, plans and reports, hosting visits by FaCSstate/territory office staff, and in the case of Reconnect, also though external assessments.Pulling all of this information together and analysing it to make reliable assessments aboutthe quality and effectiveness of the programs is a costly and time consuming process.

This is because there has not been a systematic and integrated way to link all of these bitsof information about the program. This has meant that neither FaCS nor individual serviceshave been able to derive the full benefit that can be made from the available information.

The purpose of this project is to take a good look at what information we need, for whatpurposes, and what are the best ways to collect it. A more planned and coordinatedapproach will:

• give providers good quality information to plan and improve services

• give FaCS good quality information to make funding decisions

Page 4: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 4RPR Consulting, February 2005

• give FaCS and the government a comprehensive picture of what outcomes the programsare achieving

• provide some consistency across the programs and minimise duplication of reportingwherever possible

• take a risk-management approach: under performing organisations receiving more helpand monitoring

• make reporting requirements realistic to the amount of funding received

• contribute to further development of the programs based on evidence of what works andwhat doesn’t.

Ultimately, all of this should contribute to the capacity of the YAS/FLW and ReconnectPrograms to achieve even better outcomes for the young people, families and communitiesthey work with.

1.3 What is a performance management framework?A performance management framework shows the connection between overall outcomesthat FaCS is working towards at a Departmental level and the service delivery to the client.It clearly defines and documents the processes that measure program and service providerperformance and makes it clear who is responsible at each level of measurement. It is adocumented strategy that:

• contains how all performance that contributes to the program’s outcomes ismanaged

• specifies how, and at what stages, performance is measured

• outlines reporting procedures

• includes how risk will be assessed and managed.

A performance management framework includes:

ß what the program is trying to achieve (planned outcomes – impact or effect onclients)

ß how you will know that planned outcomes have been achieved

ß the program logic – how the inputs, processes, outputs and outcomes fit togetherand describe the program

ß what performance information is required, by when and how that information willbe measured and used to demonstrate results

Page 5: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 5RPR Consulting, February 2005

ß who is responsible at each level of measurement and reporting.

As terms like outcomes, outputs, processes and inputs are often used differently, a glossaryof common terms that will be used in the project is provided at the end of this paper. Termsappearing in italics in this paper are defined in the glossary.

A performance management framework is not just a written document, placed on a file.Performance and its measurement should be considered at all stages of the programincluding the service provider selection process, program guidelines, and within thefunding agreement developed with each provider.

Questions for Consideration

What information would be most useful to collect for your organisation’s needs?

Do you have any particular concerns about the project which you want to ensure areaddressed?

Are you aware of any useful approaches within other performance frameworks whichcould be adapted to these programs?

1.4 How is the youth performance management framework beingdeveloped?

The project started in the second half of 2004, with FaCS staff from national and stateoffices meeting to describe the program logic. Interviews with the state and territory staffabout the current approach to performance management helped to identify areas forimprovement. A workshop with other FaCS programs was held to inform them about whatwas being learned in this project and to hear about approaches that have been appliedsuccessfully in other programs.

At the end of 2004, a day-long workshop was held with a group of service providers – theCritical Friends group – to help inform the consultation process with the sector. Theworkshop discussed the project, its purpose, the work done to that point and madesuggestions for the next stages.

In March 2005 a series of consultation forums will be held in each state and the NorthernTerritory. The forums will both help to inform service providers about the project and togain their views on issues outlined in this paper.

Following the consultation forums, further meetings with FaCS staff and the CriticalFriends group are planned. As well, a workshop will be held with a range of serviceproviders to help develop a new data collection system for both Reconnect and YAS/FLW.

Page 6: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 6RPR Consulting, February 2005

RPR will then document a proposed framework and develop the tools for reporting. Thetools and framework will be sent out to all services for feedback and trialed in someservices. In addition, a workshop with Indigenous specific services will be held to see whatwill need to be adapted for use by these services. The whole project is expected to finish bythe end of 2005 and be introduced during 2006.

1.4.1 Consultation forum dates

Sydney Tuesday, March 1Brisbane Thursday, March 3Adelaide Monday, March 7Alice Springs Monday, March 7Perth Tuesday, March 8Darwin Tuesday, March 8Melbourne Tuesday, March 15Campbelltown, Tasmania Wednesday, March 16

1.4.2 How to contact RPR or FaCS about this project

FaCS National officePierina Lucchesi (02 6212 9475)orMichael Hutchison (02 6212 9447)

RPR ConsultingPenny Ryan (02) 9712 [email protected] Michels (02) 9712 [email protected]

RPR has also set up a ‘blog’ linked to our website – the blog allows discussion on theframework and tools to be seen by all and for anyone to add a comment. Go tohttp://rprblog.typepad.com/ for more details.

Page 7: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 7RPR Consulting, February 2005

2 Improving performance management

The first step in developing a performance management system is being clear about whatthe programs are trying to achieve.

2.1 What is FaCS trying to achieve through the programs itfunds?

FaCS has three main goals for its work:ß individuals reach their potentialß families are strongß communities are strong.

All programs funded through FaCS must contribute ultimately to achieving these goals.Australian Government Budget allocations to FaCS are linked to these high leveloutcomes. FaCS is required to report annually to the Australian government against anumber of performance indicators which are set for each funded program.

2.2 What are the youth programs working to achieve?Reconnect and YAS/FLW have similar goals, or desired outcomes. These can be broadlyclassified into three main and interdependent areas:

• increased individual capacity and resilience of young people• increased family resilience• increased social participation.

The target groups and point of intervention for each program are however different, as arethe range of approaches used by services in order to achieve these goals.

YAS/FLW

YAS/FLW is a primary prevention program that adopts proactive strategies to strengthenthe resilience of young people by:

• helping young people gain access to a range of supervised, structured, andattractive activities matched to their needs, mainly outside school hours

• providing positive peer supports, and

• where possible and appropriate, providing individual practical support andguidance to young people and their families.

The primary target group of the program is young people aged 11–16 years and theirfamilies who live in identified areas of high need.

As a prevention program, it aims to equip young people with skills and experiences thatwill increase their participation, resilience and ability to respond appropriately to difficultfamily, school and social situations. However, young people who are already exhibiting

Page 8: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 8RPR Consulting, February 2005

inappropriate responses and homeless young people are also considered eligible forassistance.

Reconnect

Research has shown that while there are many pathways to homelessness and socialisolation, entry into the youth homeless ‘career’ is often associated with a breakdown infamily relations1. The subsequent in and out stage begins with a tentative break from thefamily situation (leaving home for a night) and involves periodic ‘running away’ fromhome for short periods and episodic homelessness. While chronic homelessness is notinevitable, it does often occur if there is not appropriate intervention or support provided tothe young person and their family.

Reconnect is an early intervention program that targets young people aged 12–18 years atrisk of early home leaving or who are recently homeless, and their families. Reconnectservices seek to help these young people to re-engage or improve their level of engagementwith family, work, education, training and the community. Wherever practicable, servicesare provided to reconcile young people and their families, which may mean the youngperson returning home, or at a minimum stabilising the young person’s accommodationand working with them to establish supportive relationships with at least one member ofhis/her family.

The service delivery model for Reconnect involves a ‘tool box’ of intervention approaches,that include a mix of practical support, counselling, mediation, mentoring, group work andbroader community development approaches. Young people and their families’ strengthsand needs are assessed and identified, and a different combination of the ‘tool box’ isoffered according to the goals and the individual circumstances of the young person andtheir family. By building protective factors that contribute to individual and familyresilience and increased social participation, services attempt to counterbalance the riskfactors that can lead to youth homelessness.

2.3 What is FaCS role in performance management?FaCS as the funder of the programs, has responsibility for:

ß selecting and contracting services (according to overall government guidelines onprocurement)

ß ensuring that contracted services deliver both the amount and the quality of servicesthey have been contracted to do

ß ensuring that the funds that have been given are used for that purpose and areregularly accounted for

ß ensuring that where services are not being provided appropriately, this is identifiedand action is taken to remedy the situation

ß ensuring the government and the Australian community are provided with accurateand relevant information on the outcomes of the program on an annual basis

ß evaluating the program on a regular basis to ensure that it is still relevant and ofvalue

1 Chamberlain and MacKenzie (2003) have identified five stages and four transition points in their youth homeless career.

Page 9: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 9RPR Consulting, February 2005

ß identifying emerging issues and research evidence that can help to develop theprogram and improve ultimate outcomes for young people, families andcommunities.

FaCS staff in the national and state/territory offices do all the above in a variety of waysincluding:

ß developing contracts with each service provider annuallyß receiving data from each serviceß visiting services and by having e-mail and phone contact with each serviceß reading reports provided by each service including six-monthly financial reports,

self-evaluation reports and annual work plansß contracting independent assessments of each Reconnect each three yearsß undertaking a program evaluation of each program each five yearsß sponsoring network meetings, Good Practice Forums and action research training.

2.4 What needs to be protected in these programs?There are many elements of the current programs which are highly valued by FaCS,service providers, clients and communities. Some of these include:

ß the flexibility of service design and delivery to meet individual and local needsß the use of action research in Reconnect to reflect on practice and find better ways to

do thingsß the application of good practice principles within both programs.

This project needs to be able to assure compliance with a minimum level of performance –while at the same time reinforcing the message that innovation and best practice areimportant and encouraged within the program.

2.5 What needs improving in the way performance is managed?As well as the need for a linked and integrated framework, a number of specific issueshave been identified within the current performance framework which will need to beaddressed in this project. These are outlined below.

Accurate and useful data collection

The Reconnect and YAS/FLW evaluations both highlighted the need for improved datacollection. Currently neither program’s collection is fully able to report on informationsuch as: the characteristics of young people and/or parents who have been assisted; whatassistance they were provided and what outcomes resulted. Group work and communitydevelopment projects could also be better captured within the data. The data collections donot provide regular and accurate reports to services or to FaCS staff responsible forperformance management.

Conducting program-wide evaluations is hampered by restrictive data – this in turn makesit difficult for FaCS to make a case for continued funding or enhancement of the programs.

Page 10: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 10RPR Consulting, February 2005

Because the current data systems are limited, youth program reporting annually to thePortfolio Budget Statement (which is where the Australian Government and people getaccess to information about funded programs) does not provide a full picture of the valueof the programs to young people and families.

Consistency in approach

Currently there is not a consistent approach across FaCS in how staff use the variety ofreports and information they receive to manage performance. In some states/territories staffregularly visit all services – in others staff prioritise visits to services perceived as needingmore assistance; some FaCS staff provide written feedback on self-evaluation reports andplans, others do not; some states/territories have regular network meetings that allow FaCSstaff to identify emerging issues or problems, while others either do not have networkmeetings or FaCS staff do not attend. FaCS staff and services are not currently providedwith a clear framework for how contact will occur and for what purpose.

A realistic level of reporting

Reconnect services currently prepare an annual work plan, a self-assessment report eachyear, two financial reports each year (including an audited statement) and submit data oneach client. The self-assessment reports follow a set format, documenting how the GoodPractice Principles are being implemented; feedback from other organisations and clients;action research findings and activity; management and organisational issues. The self-assessment reports could be better linked to the annual work plan both in terms of contentand timing.

YAS/FLW services similarly prepare an annual work plan. These services report progressagainst the plan and on implementing the Good Practice Principles. Many YAS/FLWservices operate on a part-time basis and staff have very little administrative time, and maylack reporting skills where their role is to undertake activities with young people.

Many of the reports from services are overly lengthy. It is not clear for either party howmuch detail the reports should contain or how they are/should be used.

Opportunities for rationalising accountability requirements

Many Reconnect and YAS services are part of larger agencies, such as local government orlarge NGOs, which are required to submit plans and reports to a range of audiences. Todate little consideration has been given in the reporting framework to cutting down onduplication. This is important to ensure that unnecessary time is not spent in administrativework, thus detracting from a services’ ability to work with clients and their community toachieve better outcomes.

Clarity in performance expectations

Currently the reporting system is not very good at identifying services that are underperforming or even at defining what good, satisfactory or under-performance is. While‘good practice’ in both programs has been documented, what is the performanceexpectation for ongoing funding? How should poorer performance be addressed? Evenwhere poorer performance is clearer – for instance not operating the service due to a failure

Page 11: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 11RPR Consulting, February 2005

to employ staff – the current reporting system does not necessarily alert FaCS staff to thesituation in a timely way.

The lack of clarity about performance expectations means that it is hard for FaCS to knowwhen a service should be a preferred provider and not have to re-tender and when it isappropriate to intervene and re-tender a service. The youth performance managementsystem will help to identify good, satisfactory and poor performance so that expectationsare clear to both services and the Department.

Taking a risk-management approach in reporting and support

Currently the reporting requirements are the same for all Reconnect services or for allYAS/FLW services, regardless of the types of services they provide, their funding levels,or where they are located. Other programs differentiate reporting requirements aroundrisks, which can defined in a range of ways. Risks can be defined in relation to client andstaff safety, for example, where the service provides case work or counselling to avulnerable young person and their family or runs risky adventure activities such asabseiling or rock climbing. On the other hand, risk can also be considered in relation to thelevel of investment in the service made by the funder. Services that receive $300,000 mightbe expected to have a greater capacity and pose a higher risk and consequently be asked toreport in greater detail than those receiving $25,000. Other risk factors which might beconsidered could be geographic isolation, or lack of experience of the agency and/or staff.These services may pose a greater risk, and thus require greater support from the programmanager to ensure quality of service delivery to their communities.

Including mechanisms for client feedback

Currently there is no mechanism for gaining client feedback on the benefits of the programat a program (as opposed to an individual service) level. Client feedback helps to supportevidence for immediate outcomes gained as a result of assistance and can help identifyareas for service improvement.

Purposeful collection of information

Finally, some of the current information that is prepared and collected is not fully used. Abetter designed data and reporting system will be linked to the purposes for which theinformation is to be used. The youth performance management project provides anopportunity to re-think the way reporting is done.

Questions for consideration:

Are there strengths in the current approach that you think should be reflected in the newframework?

Are there other problems with the current performance management approach that have notbeen identified? Suggestions of better ways to do things?

Page 12: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 12RPR Consulting, February 2005

2.6 Proposed criteria to judge the framework’s successThe Critical Friends workshop developed some criteria that can be used to test theperformance management framework developed in this project. These are listed below.

An effective Performance Management Framework will have the following features:

Accountabilityß clarifies performance expectations (are providers doing what they have

been funded for; are public monies being managed appropriately; is theprovider/program achieving the intended outcomes?)

Outcomes focusedß reflects the purpose of each programß includes opportunities for young people and families to have input

into the processß captures the range of impacts the programs are making

A two-way processß is developed and implemented within a partnership relationship based

on trust and respectß gives benefits to services as well as the Department

Dynamicß will be regularly reviewed and will develop in response to changes on

the groundß will have capacity for flexibility

Useful and simpleß is purposeful and informativeß is linked to the core work of both partiesß is easy to implement and balances effort of collection with outcome of

information

Reflect diversityß takes into account the diversity of services and their circumstances, both

in what is measured and in how it is measured.

Common understandingß is understood by those implementing itß is logicalß is transparent

There are a wide range of variables to consider in this project, and hence there are likely tobe any number of possible options proposed. It is envisaged that the above criteria can beused throughout the project to assess options as they are developed to ensure that theapproaches finally agreed on capture the really important issues identified by serviceproviders and FaCS staff. They can also be used to re-evaluate the usefulness of theframework after it has been trialled for some time within the two programs.

Questions for consideration:

Are the proposed criteria useful?

Are there other criteria you think are important to guide the development of theperformance framework?

Page 13: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 13RPR Consulting, February 2005

3 Developing the youth performance framework

A starting point in developing the framework is to explore what information is needed, atwhat points and how best the information can be collected and analysed. For FaCS tomanage the program and fulfil the role outlined earlier, information is needed both aboutindividual services (to make funding decisions and ensure accountability), as well as aboutthe program as a whole (to enable reporting to government and ensure that the program isbenefiting the people it is aimed at). The information collected therefore needs to serveboth of these purposes and is clearly interrelated.

3.1 Performance expectations of individual servicesAt present only some of the performance expectations of services are transparent, detailedand documented. To ensure cost-effective use of funds, the Department needs to knowthat:ß organisations have the capacity to deliver the services funded

– in the manner agreed in program guidelines and funding agreements– to an acceptable standard

ß organisations actually deliver on this capacity– to the people the program is intended to benefit– producing outcomes for those people.

As a result, the Department needs to assess performance in relation to both service deliveryoutcomes and organisational capacity.

3.1.1 Assessing service delivery performance

The Department needs to assess the adequacy of service delivery performance so that theycan tell that the funds are being spent for the purposes of the program (for instance theprogram is reaching the young people and families intended to benefit) and that theservices are making a positive difference to the people using them. Currently, the datasystem for both programs either gives none, or very limited information about whom theservices reach, what they do and how this work has benefited young people or families.

The Good Practice Principles for both programs are not integrated into data collection – forexample, in Reconnect immediacy of response has been identified as an important featureof the service delivery model, but no data is collected on how soon young people orfamilies were able to access the service. While Reconnect services report on their use ofthe Good Practice Principles and action research in the self-evaluation reports, it is unclearhow service delivery is then assessed. YAS/FLW services do include information on theGood Practice Principles in their progress report, but this is not integrated into the annualwork plan and again it is unclear how it is assessed.

Examples of approaches used by other programs

As mentioned in section 1.4 a workshop with other FaCS programs was held to informthem about what was being learned in this project and to hear about approaches that havebeen applied successfully in other programs. The intent is not to necessarily introduce

Page 14: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 14RPR Consulting, February 2005

these approaches as they currently exist into FaCS youth programs, but rather to seed somenew thinking about tools and approaches which could be useful to adapt to suit our needs.

One such example which was shared was the data collection system developed for theFamily Homelessness Pilot Program (now called HOME). This data collection formincludes information about service delivery (eg. home visits, collaborating agencies, typeof case goals developed and progress made, case complexity). The data collected in theFHPP pilots was able to demonstrate that the key variables which impact on clientoutcomes are case complexity, intensity of support, and amount of brokerage funds spent.For example, employment outcomes for clients improved regardless of the complexity ofthe case, where there were more agencies involved as part of a co-ordinated case plan.

What indicates good or poor performance in service delivery?The Critical Friends Workshop discussed what might be a sign of good service deliveryand what might indicate that poorer performance is occurring. These types of indicators area beginning step in defining what satisfactory service delivery might be within eachprogram, and provide some indication of when the Department may need to take action tofind out what is happening to cause poorer performance.

The following table outlines some of the ideas that emerged at the workshop andsubsequently about service delivery specifically, rather than organisational capacity.

Reconnect YAS/FLWGood Poorer Good Poorer

• Immediacy ofresponse

• Community links andsupport (gets positivefeedback)

• Young people andfamilies use theservice – range andnumber

• High numbers ofself/friend referrals

• Uses a range ofapproaches – the fulltool box

• Seeing majority ofyoung people beforeleaving home

• Regular reporting ontime

• High % of youngpeople and familiesreport positive benefit

• Positive change inoutcome measuresrelating to familyfunctioning,engagement witheducation & training

• Drop in numbers• Seeing more young

people who have lefthome before/arehomeless

• Few self/friendreferrals

• Not seeing manyparents/familymembers

• Complaints from otherservice providers orclients

• Limited or rigidapproaches (all clientsget same interventionregardless of need)

• Only seeing ‘easy’clients

• Long waiting timesbefore contact orservice

• Not sending in data orother reports

• Small % of youngpeople & familiesindicate positivebenefit

• Numbers of youngpeople using service

• Repeat use of serviceby young people

• Service reached byrange of young peoplewithin target group

• Community networks(number and range)

• Referrals from otherservices and fromyoung people

• Regular reporting ontime

• High % of youngpeople and familiesreport positive benefit

• Outcome measuresindicate positivechange

• Drop in numbers• Servicing young

people outside thetarget group

• Servicing only onegroup of youngpeople – no newyoung people using it

• Complaints fromother providers oryoung people

• Not sending in data orother reports

• Small % of youngpeople & familiesindicate positivebenefit

• Outcome measuresindicate littledifference or negativechange

Page 15: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 15RPR Consulting, February 2005

education & training• Increase in service

system capacity tointervene early in YHpathways

benefit• Outcome measures

indicate littledifference or negativechange

Examples of approaches from other programs

In the Personal Support Program (PSP), providers have to report against key performanceindicators (KPIs) in relation to effectiveness and efficiency. Some examples of PSP KPIsare:

• effectiveness- the extent to which providers engage and maintain program participants

ß how many participants, for how long, compared to the number ofreferrals

ß number referred back to Centrelink early; number suspended andreason

ß how many with Action Plans and time taken to agreeß number of referrals to other services; acceptance

- number and type of economic outcomes- number and type of social outcomes- extent to which a diverse range of assistance to address client needs is

offered (range of intervention in Action Plan, client survey)- level of participant satisfaction (survey, complaints)

• efficiency- management of participant reports and contacts

ß number of reports completedß time taken to complete reportsß number suspended and referred back to Centrelinkß time between referral and first appointmentß time between commencement and Action Plan.

In Western Australia, services funded under the state department give each client a simplefeedback form that allows them to estimate what progress they made against somecommon service type goals.

The feedback form is provided with a reply paid envelope and this is sent back to theDepartment. Each six months the service receives a report on the consumer feedbackreceived compared with the service’s own estimates of progress against the goals made forclients. This has proved to be of benefit to programs as a whole, as outcomes against goalscan be reported to government. It has also been useful to services in confirming theirperspectives of the benefits of their work with clients. For a few services where there issubstantial variation between the service’s estimation and the feedback from consumers, ithas allowed the Department to work with the service to identify reasons for the gap andwhat might be done to address it.

Other programs have set performance indicators as targets, and compare one service withanother similar one to assess satisfactory/poorer performance. For instance in the Job

Page 16: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 16RPR Consulting, February 2005

Network, data is collected across the country on the types of clients seen, the location ofthe service and the outcomes achieved. Each service outlet is then compared with similarservices and a five star rating is provided. The rating is used to alert services to the need toimprove their approach as, if the rating is too low, it is likely that the service will not berefunded at the end of the contract.

The benefits of this approach is the transparency it provides, however it is difficult toensure that ‘apples’ are being compared to ‘apples’ and to take into account changes incircumstances that might have occurred.

Currently Reconnect Independent Assessment reports do provide services with a picture ofwhere they sit in comparison to other services nationally on a few limited indicators. Thismay be a useful approach to expand to assist services in improving their approaches, evenif it is not used to assess performance per se.

Questions for consideration:

Are the indicators suggested in the table useful?

Are there other indicators which might give a picture of performance in relation to servicedelivery in each program?

Is there merit in comparing similar services to each other to assess performance?

3.1.2 Assessing organisational capacity

As part of the process of selecting and funding services, the Department needs to be surethat an organisation has:ß an understanding of the program and its aims and a plan for implementationß respect and support within a community as a providerß the organisational capacity to manage funds effectively and the necessary financial

infrastructure to support the serviceß the organisational capacity to recruit and support appropriate staffß appropriate governance and management systems to ensure effective planning,

monitoring and service improvement occursß effective policies and processes to guide service delivery and work with the

community.

Currently, each FaCS program develops selection criteria independently (with somecommon criteria being used across the Department). In the main, most of the selectionprocess focuses on the capacity of the organisation to understand the program and have aneffective way of implementing a service. While financial statements and boardmembership are checked, on the whole less detailed attention is paid to the internalorganisational capacity that supports service delivery – for example governance andmanagement systems, policies and processes, planning, monitoring and service

Page 17: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 17RPR Consulting, February 2005

improvement approaches. However, these organisational capacity issues are often thecause of organisations failing to perform well and ultimately ‘falling over’ in servicedelivery.

The current reporting system does collect some information on organisational issues –financial reports are received half yearly and Reconnect Independent Assessments(conducted every three years) report in greater detail on management and otherorganisational issues.

Examples of approaches used by other programs

One approach to making the organisational capacity expectations clearer is the use ofApproval Requirements (standards that set out baseline expectations for all services) forservices funded under the Family Relationships Services Program. These services muststate that they comply with the requirements as part of the selection process (althoughcompliance is not validated at the point of selection).

The 15 Approval Requirement standards cover:ß Values and ethical frameworks (need for a values statement, ethical framework,

staff, consumers and other stakeholders are aware of this)ß Governance (effective systems for internal control and transparent decision making,

financial systems to identify and manage risk)ß Strategy, policy and planning (strategic directions, use of plans, stakeholder input)ß Management of data (procedures and staff awareness)ß Entry of practitioners (competencies, qualifications and recruitment)ß Supervision of practitioners (minimum supervision requirements, evidence that this

occurs)ß Training and development (planned approach to training, evidence of training

occurring)ß Staff appraisal (system established and used)ß Safety of staff (risks identified and managed)ß Accessibility of services (evidence of strategies to ensure access)ß Managing client feedback and complaints (complaints system, use of feedback

processes)ß Client confidentiality and privacy (polices and procedures for ensuring

confidentiality and privacy)ß Client safety (risks identified and managed)ß Service design (ensuring core service processes are documented and that quality

improvement occurs)ß Assessing performance (performance trends are monitored and are used to improve

delivery).

FRSP services are externally assessed once for compliance against the requirements usingassessment criteria that outlines what is needed to demonstrate full compliance, partialcompliance and no compliance. Once an organisation is assessed as fully compliant theywill now be required to sign a statement of ongoing compliance each year, against eachstandard. A copy of the full requirements and criteria for assessing compliance is availableon RPR’s website (www.RPRconsulting.com.au).

Page 18: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 18RPR Consulting, February 2005

While the FRSP Approval Requirements reflect the type of work done by FRSP services,their benefit has been to make clear to all stakeholders what the expectations are. If theyouth programs built upon this idea, the criteria developed would need to reflect thecapacity of organisations to operate Reconnect and YAS/FLW services.

Other approaches used to demonstrate organisational capacity include externalaccreditation systems such as the National Child Care Accreditation System, the QualityImprovement Council (QIC), Business Excellence or Standards Australia (ISO). Someauspice organisations have been accredited under quality programs such as these. Manypeople have suggested that such accreditation status should be acknowledged by funders asa valid measure of organisational viability by FaCS and other funders.

Questions for consideration:

Would it be useful to set out clear organisational expectations of service providers to beused as criteria for both selection and ongoing compliance and, if so, what should these be?

Should the performance framework take account of external accreditation systems and inwhat ways?

3.2 Assessing the performance of the programsThe Department needs to report on outputs and outcomes of the programs, including bothquantity and quality measures to answer questions such as:

ß how much did the program do? (for instance: what types of families were reached,how many young people and families were assisted, what sort of activities wereprovided and how many of them?)

ß how well was it done? (for instance: % of clients reporting satisfaction, staff/clientratio, client safety measures put in place, % of clients completing a group, % ofclients reached at an early stage)

ß is anyone better off? (for instance: % of young people with increased life skills, %of young people with improved family connection, % of clients with stabilised orimproved circumstances).

At present, the measures being used to report on Reconnect and YAS/FLW do not reflectthese types of indicators. By developing a clearer description about the outputs andoutcomes of the two programs it will be easier to look at how they might be measured inthe most cost effective way. Part of developing the PMF will result in some improved waysof measuring these types of outcomes. In order to develop better ways of measuring andreporting on what services do, how well they do it and whether anyone is better off as aresult, there needs to be agreement about what the anticipated program outcomes are.

Page 19: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 19RPR Consulting, February 2005

The work done so far in this project has identified the following outcomes as ones thatYAS/FLW and Reconnect services work towards. How many of these outcomes areworked towards will depend on the resources provided to a service, the community it isworking within and the activities provided.

Reconnect YAS/FLW• Improvement in young people’s attitudes and

behaviours (for instance by increasing theirprotective behaviours, their self-esteem or theirresponsibility)

• Increased skills and knowledge (for instanceincreased life skills, relationship skills, pre-vocational skills)

• Improvement in family functioning, familyconnection and reconciliation (by reducing thenumber and intensity of conflict and/or improvingcommunication)

• Improvement or stabilising of young people’s andfamilies circumstances (by referring or advocatingfor young people/families to health,accommodation and/or income support)

• Increased engagement with education, training andemployment

• Increased community connection

• Increased community capacity to address youthhomelessness.

• Strengthening young people’s positive attitudesand behaviours (for instance by increasing theirprotective behaviours, their self-esteem or theircapacity for team work)

• Increased skills and knowledge (for instanceincreased life skills, relationship skills, pre-vocational skills)

• Enhancing family functioning and familyconnection (by reducing the number and intensityof conflicts and/or improving communication)

• Improvement or stabilising of young people’s andfamilies circumstances (by referring or advocatingfor young people/families to health,accommodation and/or income support)

• Increased connection with education

• Increased social networks

• Increased community connection.

Questions for consideration:

Do the outcomes you see in the work you do with young people, families and communityfit within those listed above?

Are there other outcomes that should be considered?

3.3 Developing the methods for reportingOnce it is clear what information needs to be collected, for what purpose and when, it willbecome easier to develop the actual methods of reporting. A good reporting system reflectsthe criteria for an effective performance management framework outlined earlier byideally:

Page 20: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 20RPR Consulting, February 2005

ß minimising duplication of the collection of information – ensuring that theinformation collected allows it to be used multiple times, rather than collecting itmany times

ß integrating the collection of information into the day-to-day work as much aspossible, rather than making it an add-on task that only serves the purpose ofinformation gathering (this will not always be possible, but should be attempted asa starting principle)

ß placing an emphasis on use of the information – information will be collected in amore consistent and reliable way when its use and benefit is apparent.

It is likely that this will mean considerable change to the way things are currently reportedon, however tools and approaches will sent be out for comment and then trialed later in theproject before they are implemented.

3.4 Next stepsThis project is at its beginning stages. The forums being held in each state/territory are akey point for service providers to have input into the development of the new PerformanceManagement Framework.

This paper provides some questions to be considered in advance so that you can comeprepared with some useful thinking about the range of issues involved. Please discuss themat both a management and team level as the framework will impact on people at all levelsof your organisation.

FaCS and RPR are committed to working in partnership with service providers in thisproject to ensure that the end results are useful, workable, and owned by all involved.Thank you for your assistance.

Page 21: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 21RPR Consulting, February 2005

4 Glossary of terms2

Outcomes – all the impacts or consequences of the program – what is different forindividuals or communities as a result of the program. Outcomes are often delayed orlong term and they may be intended or unanticipated. Although programs work towardsbringing about various outcomes, often the outcomes (eg desired changes in individuals'circumstances) are beyond the direct control of the program. The causal relationshipbetween a program and an outcome must be demonstrated before the outcome can beclaimed to have been caused by the program.

Outputs – the products or services which are produced and delivered by a program inorder to achieve the program's outcomes. Outputs of YAS/FLW services includeworkshops, school holiday programs, events, referrals etc. Outputs of Reconnectprograms include counseling, mediation, brokerage, group work, communitydevelopment projects etc.

Process – the means used to produce the outputs designed to bring about outcomes.Examples of processes include service planning and design, assessing needs of individualsand communities, action research, ongoing monitoring and evaluation of services etc.

Input - the resources, both human and other, used to produce program outputs. An inputto a program activity may also be an output of an earlier program activity. For example,the output of a training program may be qualified program staff; qualified program staffmay in turn be the input for a service delivery program.

Program logic – a diagram which illustrates the way in which the program operates bydescribing the relationship between inputs, processes, outputs, and outcomes of theprogram; will also map the cause and effect relationships between program activities (orprocesses) and the outcomes they should produce.

Procurement – the process of “buying” services; allocating resources to an organisationto deliver the services prescribed within a specific program.

Performance information - evidence (quantitative or qualitative) about performance thatis collected and used systematically. Evidence may relate to appropriateness, effectivenessand efficiency and may be about outcomes, about factors that affect outcomes, and aboutwhat can be done to improve outcomes. Performance information also includes evidenceabout the extent to which outcomes can be attributed to an intervention/program.

Performance indicator - a description of what is measured to determine the extent towhich objectives and outcomes have been achieved.3

2 Most of these terms been adapted from the Department of Finance publication: Doing Evaluations - A Practical Guide

3 Dorothy A. Johnson Center for Philanthropy and Nonprofit Leadership – Grand Valley State University, USA -http://www.nonprofitbasics.org/TopicAreaGlossary.aspx?ID=14&curLetter=E

Page 22: Youth Programs Development of a Performance Management ...

Consultation Paper: Youth Performance Management Framework Project 22RPR Consulting, February 2005

Performance measures – a more precise measure than indicators. Performance measuresrelate to outcomes and outputs. They are used when there is a direct causal link betweenan intervention and a measurable change in performance.