Intro to Continuous Quality Improvement –Using the PSAYDN Tool

Today’s Objectives • Explain how CQI supports DHS youth outcomes• Use CQI as a way to tell your story• Explain how to use the PSAYDN self-assessment for CQI• Develop an implementation strategy for the PSYADN self-assessment

Today’s Agenda

• Why? – CQI Rationale

• With What? – Using the PSAYDN Tool

• Who + How? Strategy for Implementing the Tool

• When + How? Submitting the Assessment

• Q & A about CQI

Why CQI?

It Works!Here’s How…

CQI is an on-going process ofassessing the quality of workand making improvements

Continuous Quality Improvement (CQI)

Helps tell the story of your work to multiple stakeholders about achieving desired youth impact

Continuous Quality Improvement (CQI)

Uses feedback tools(e.g. parent surveys, debriefing forms)

Uses supports to make informed improvements

(e.g. staff meetings, coaching, professional development)

Continuous Quality Improvement (CQI)

Designed so improvements can be made throughout the process.

Continuous Quality Improvement (CQI)

Picture of Program Quality

Caregiver/Youth Assessment

External Assessment (Site Visit)

Internal Assessment (PSAYDN/APAS)

Getting a Complete Picture

But Why?

Outcomes.

DHS Mission Statement

The Philadelphia Department of Human Services mission is to provide and promote safety, permanency, and well-being for children and youth at risk of abuse, neglect and delinquency.

DHS OST Project FY 2015 Vision

Create a more intentional connection between the needsof all children and youth, particularly those served by DHS

and the DHS OST Project as an effective prevention strategy and critical community-based resource.

FY 15 Youth Impact

Youth Focus

Program Quality Focus

Evaluation Framework (Desired Youth Impact)

Youth who participate in DHS-funded OST services are less likely to be active within DHS;

and

Youth who were formerly active within DHS are less likely to re-enter the DHS system of services.

GOAL AREA OUTCOMES INDICATORS

Academic success

1. Increased Engagement in Learning

2. Increased Engagement School3. Aspires to Educational

Excellence

1. Improved Quality of Homework Time2. Increased Opportunities for Problem

Solving & Critical Thinking 3. Awareness of High School Choices

Develop positive life skills

1. Improved Life Skills2. Improved Relationships

1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults

Ready for college and/or career

1. Prepared for Higher Education and Employment

1. Awareness of and interest in Careers

DHS OST Goals, Outcomes & Indicators

GOAL AREAS OUTCOMES INDICATORS

Academic success

1. Increased Engagement in Learning

2. Increased Engagement School

3. Aspires to Educational Excellence

1. Improved Quality of Homework Time2. Increased Opportunities for Problem

Solving & Critical Thinking 3. Awareness of High School Choices

Develop positive life skills

1. Improved Life Skills2. Improved Relationships

1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults

Ready for college and/or career

1. Prepared for Higher Education and Employment

1. Awareness of and interest in Careers

DHS OST Goals, Outcomes & IndicatorsElementary School Priorities

Red = Primary Blue = Secondary

GOAL AREA OUTCOMES INDICATORS

Academic success

1. Increased Engagement in Learning

2. Increased Engagement School

3. Aspires to Educational Excellence

1. Improved Quality of Homework Time2. Increased Opportunities for Problem

Solving & Critical Thinking 3. Awareness of High School Choices

Develop positive life skills

1. Improved Life Skills2. Improved Relationships

1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults

Ready for college and/or career

1. Prepared for Higher Education and Employment

1. Awareness of and interest in Careers

DHS OST Goals, Outcomes & IndicatorsMiddle School Priorities

Red = Primary Blue = Secondary

GOAL AREA OUTCOMES INDICATORS

Academic success

1. Increased Engagement in Learning

2. Increased Engagement School3. Aspires to Educational

Excellence

1. Improved Quality of Homework Time2. Increased Opportunities for Problem Solving

& Critical Thinking 3. Awareness of High School Choices

Develop positive life skills

1. Improved Life Skills2. Improved Relationships

1. Goal Setting 2. Personal Accountability3. Work with Others to Accomplish Goals4. Improved Interactions with Peers 5. Improved Interactions with Adults

Ready for college and/or career

1. Prepared for Higher Education and Employment

1. Awareness of and interest in Careers

DHS OST Goals, Outcomes & IndicatorsHigh School Priorities

Red = Primary Blue = Secondary

CQI and You

• Program quality• Empowering site-level staff

to lead improvements• WILL NOT determine FY

2016 funding

CQI and You Data gathered will help:• PHMC and your program work

together to improve quality• Determine whether youth

experiences in your program result in desired DHS youth outcomes

• DHS make decisions about system-wide supports

The CQI Process - 2014Collect data,

Reassess

Provider self-assessment

Data collection

and analysis Goal

setting w/ Provider, Program Specialist

Program Improvements

Site Visit

ProgramImprovements Training on

CQI and Tools

Adapted from the National Institute on Out-of-School-Time

The CQI Process - 2015Collect data,

Reassess

Program Specialist Reviews

Goal setting w/

Provider, Program Specialist

Program Supports and Improvement

Site Visit

Program Supports and Improvement Provider Goal

Setting In Reapplication

Adapted from the National Institute on Out-of-School-Time

Picture of Program Quality

Caregiver/Youth Assessment

External Assessment (Site Visit)

Internal Assessment (PSAYDN/APAS)

Getting a Complete Picture

Here’s How.

PSAYDN Self-Assessment

Section A – Structure & Management• Fundamental to the implementation of a good program

• Will likely require management support to complete• Executive Director• Human Resources• Finance

• Useful for larger strategic planning discussion within the program and the agency

• Program Specialist will provide minimal coaching in this area

Section B – Positive Connections• Helps us examine the relationships between staff, youth,

and caregivers within our programs

• Research and experience tell us that relationships are key to successful youth development work

• Also assessment connections to community partners

Section C – Safety & Health

• Just like sound management practices, safety is critical to success in other areas

• Focuses on emergency policies, facilities, and activities that promote healthy lifestyles

Section D - Activities• Assesses the extent to which activities are:

• Thoughtfully-designed• Age-appropriate• Youth-driven and responsive to youth interests• Foster learning and positive youth development• Varied

• Also examines the planning process for activities

Addendum for Elementary Programs

• The quality of homework time is a priority indicator for DHS elementary programs.

• PSAYDN Self-Assessment does not include a homework time.

• The addendum is a chance to assess how well your program is providing homework support.

• Can be identified as an area for improvement.

Additional Questions for ALL Programs• #46 - Rank the PSAYDN categories in order of one to five based on

the areas your program could use the most additional support.

• #47 - What additional supports would your program like from PHMC/DHS to assist in the area from Q. #46 ranked as a number one (most important).

• #48 - How did program staff feel about the CQI process in FY2015?

• #49 - Why does your program believe the CQI process worked? Select all that apply.

• (If you select ‘No’ to #49) #50 - Why does your program believe the CQI process did not work well? Select all that apply.

Additional Questions for ALL Programs• #51 - What communication process did your program use to share

CQI information with your Executive Director and other stakeholders?

• #52 - What was the first goal you set for FY2015?

• #53 - Did your program achieve the first goal set for FY2015?

• #54 - Provide an explanation for why your program did or did not achieve the first goal.

• #55 - What was the second goal you set for FY2015?

• #56 - Did your program achieve the second goal set for FY2015?

• #57 - Provide an explanation for why your program did or did not achieve the second goal.

Additional Questions for ALL Programs• #58 - What was the third goal you set for FY2015? If you only set

two goals, write N/A below.

• #59 - Did your program achieve the third goal set for FY2015?

• #60 - Provide an explanation for why your program did or did not achieve the third goal.

• For each section, we will ask about your program’s rationale for the score selected for that section’s questions. Be prepared with typed text that you can insert into the online survey.

Scenarios

A.1.3 – Guiding Questions

• Do staff have access to on-going professional development opportunities such trainings and conferences?

• What are the yearly expectations for professional development, in terms of amount and/or content, by staff’s level of responsibility in your program?

• What types of on-site support do staff have to continue building skills?

Scenario #1: A.1.3Agency ABC: Staff members feel well supported.

They are invited to attend paid workshops offered by professional development providers, as well as

occasional on-site in-service trainings. All staff participate in PD in some form, but agency ABC does not monitor requirements for annual PD.

Meetings are held once per month and focus on program planning and policy updates. Staff receive annual performance reviews in writing with some

supervision is provided as needed.

Scenario #1: A.1.31 Only some staff attend professional development

Professional development topics not informed by any assessment of needs

3 Staff participate in on-going professional development to address the unique

characteristics of youth and families and the content focus of program

Staff receive only informal supervision or feedback related to their performance

Staff meetings held at least monthly to address program issues

5 Staff receive written, performance-based, assessments of their work, identify areas for

needed skill-building, and plan for professional development based on identified

priorities

Professional development opportunities are available in multiple formats (on- and off-

site) including workshops, observation, coaching, and peer-to-peer learning

Staff receive regularly-scheduled one-on-one supervision and feedback

Staff meetings held more than once a month to address program issues and support

staff skill-building

B.1.1 – Guiding Questions

• In what ways do staff support positive communication and conflict resolution with youth and co-workers?

Scenario #2: B.1.1Agency DEF: The type of communication happening varies a great deal. Six of eight members of the team get along very

well and have positive communication with the youth. Among these staff, conflicts with co-workers are resolved

quickly. These staff also use the organization’s reward/consequence structure to guide youth behavior.

There are two staff members that don’t gel with the rest of the team. They are quick to snap at the youth and at the other staff members. The kids have started to call them

Oscar #1 and Oscar #2 (the grouch). There are noticeably more behavior problems (put downs, teasing, squabbling, etc.) among the youth these two staff members supervise.

Scenario #1: B.1.11 Staff either ignore youth conflicts, or impose negative consequences without

taking the time to understand the cause

3 Staff relate to all participants in positive ways Program establishes, maintains and communicates code of conduct to participants,

staff, and their families Program applies rewards and consequences for participant behavior appropriately and

consistently Staff communicate with respect and are role models of positive relationships

5 Staff relate to all participants in positive ways and intentionally build individual relationships

Staff approach negative behavior calmly, acknowledging youth’s feelings while stopping hurtful behavior

Staff facilitate constructive conflict resolution when needed

C.1.2 – Guiding Questions

• Does your program have written policies to ensure equal access and safe participation for all youth?

• How do staff promote and maintain a safe physical and emotional environment for all participating youth?

Scenario #3: C.1.2Agency GHI: Prides itself on creating a safe and inclusive

space for youth. Last year, their 70 year-old building added a wheelchair ramp and is now ADA compliant.

Program activities focus on celebrating differences and there is virtually no bullying as a result. Several youth

have special needs and the Site Director met with each caregiver to understand how to best accommodate their

youth’s needs. Six months ago, agency leadership and father of one youth decided the program wasn’t suited

to meet his son’s needs, who had a severe autism diagnosis. The staff and family worked together to find

another program that would work better.

Scenario #3: C.1.21 Program space or offerings are not accessible to all youth

3 Program space and operations meet all local, state and federal regulations regarding accessibility (e.g. ADA)

Program develops and manages effective arrival and dismissal procedures and plans for safe travel home

Participant files include emergency contact information, medical release forms, allergy and health information

No one in the program feels threatened or unsafe because of who they are

5 • Program materials and policies emphasize commitment to serving all youth in the community

• Staff make reasonable accommodations to serve youth with learning differences or special needs

• Staff consistently intervene when youth are adults make hurtful or negative comments

• The emotional climate of the program is positive, characterized by mutually respectful, supportive interactions among staff and youth

D.1.1 – Guiding Questions

• Describe your planning process - how do you determine overall goals and daily activities (e.g., using your logic model, documented program design)?

Scenario #4: D.1.1Agency KLM: Youth are always busy. The staff plan their activities the evening before, using a couple of activity idea books the Site Director bought online.

Examples of activities include friendship bracelets, a soda bottle terrarium, and multiplication jeopardy.

When asked how activities are chosen, the staff say, “Whatever sounds fun.” When the staff don’t have

the materials they need for a particular activity, they have some board games, jump ropes, and a kickball.

Youth of all ages (K-5) participate in activities.

Scenario #4: D.1.11 Activities are not intentionally planned according to program mission,

developmental or learning objectives

3 Activities reflect the mission of the program and promote the physical, mental, and/or social-emotional development of all participants

All activities planned according to the age and skill level of the participants and identify targeted skills that participants will develop during the program cycle

5 Comprehensive written program design and activity plans define learning objectives that build on each other throughout the program cycle and show how activities connect to objectives

Program emphasizes the connections between learning and real life

What’s Your Process?

The Who and How

Self-Assessment Comfort Test

High comfort level with self-assessments • Complete with team and submit online.

Some experience with self-assessments• Program Specialist support for questions as needed.• Work with team to complete and submit online.

Completely new to self-assessments • Share online training with key individuals.• Regular email/conversation with Program Specialist.• Work with team to complete and submit online.

PSAYDN Discussion Guide

IMPORTANTWhen entering the survey online• Complete it in one sitting – It does not save progress.

Have your answers ahead of time• Have your scores for each item that applies• Have the rationale for each section and additional items

answered in advance.

Due c. May 15th• Do not wait until the last minute – plan it on a calendar.

SCORE YOUR PROGRAM ACCURATELY. DHS IS NOT USING THIS TO JUDGE YOUR PROGRAM. THEY USE IT TO

DETERMINE SUPPORTS.

Minimal Investment- CQI Training

- Manager/SD completes form independently- Enter data

Moderate Investment

- CQI Training- Management team

completes form with some staff input

- Enter data

Maximum Investment- CQI Training

- Exec. Staff input for Part A- Line staff input for Parts B-D

- Time to discuss- Align with site visit assessment to

determine evidence of quality - Enter data

How much time will it take?

Tips for Getting Started: Program Directors

• Make an appointment with administrative (managers, directors) staff to ask questions.

• Let your team know that this is coming.

• Consider involving other stakeholders (families, funders, activities partnerships)

• Try to collect “evidence” (sample parent/family survey, newsletters, student artwork, rubrics, etc.) that support your program’s efforts.

Tips for Getting Started: Working With Administration (EDs, Managers)

• Make yourself available on their schedule.

• Set up a brief phone meeting before starting.

• Review the self-assessment before meeting.

• Make suggestions for potential improvement project and program changes.

Submitting the Self-Assessment

EXAMPLE PSAYDNSELF-ASSESSMENT

Timeframe

Goal Setting Reapplication

May - June

PHMC Analysis of

Data May-June

Complete Tool and

Submit Data

May 15

Review Tool & Determine

Support Needed

Apr - May

Intro to CQI Process

Early April

Tools available online

http://afterschoolinphilly.weebly.com

Wallace Foundation: Strategies for Quality After School Programs and Activities

NIOST: Understanding Basic Standard for A Quality Out-of-School Time Program

Quiz Question #1

What does the abbreviation CQI stand for?

A. Continuous Quality ImprovementB. Content Quality ImprovementC. City Quality Instrument

Quiz Question #2

How many sections are on thePSAYDN self-assessment?

A. ThreeB. FourC. Five if you include the HW addendumD. SixE. Seven

Quiz Question #3

Who should ideally be involved in the self-assessment process?

A. Executive Staff, Finance & Human ResourcesB. Program Directors/ManagersC. Site DirectorsD. Direct Line StaffE. All of the Above

Quiz Question #4

Where can I find self-assessment submission instructions?

A. The Scope of WorkB. PCAPSC. PHMC Website D. LearnUpon