Continuous Improvement

Collecting, Analyzing, and Sharing Data

At the end of this presentation, you will know:

1) All the wonderful ways that you can use data for important purposes

2) What “formative evaluation” is, and how it relates to continuous improvement

3) How to organize yourselves for formative evaluation

4) How to collect data and use it to fuel continuous improvement

5) Different approaches to analyzing, reporting on, reflecting on, and sharing your data

6) How to work with C-PAD to accomplish data-related tasks

The Joys of Data!

Data are not just for grant requirements anymore!

Participation

Satisfaction

Tailoring

Effectiveness

By using data for continuous improvement, you will increase:

Using Formative Evaluation for Continuous Improvement

• Evaluation = using data for assessment and learning

• Formative = for the purposes of creating, developing, and improving

• Formative (program) evaluation = Using data to inform how you can continually pursue program excellence

What is formative evaluation?

• Bullet Points

Formative Evaluation as a Tool for Continuous Improvement

Formative evaluation asks:

So that your team can further investigate:

In order to achieve:

Is the program reaching its participation targets?

How can we attract more people to the program?

Increased participation

Are participants satisfied with the offerings?

How can we increase satisfaction levels?

Increased satisfaction

Does the design of services meet the needs of its participants?

How can we better align service offerings with participant needs?

Increased tailoring

Do participants succeed in desired outcomes?

How can we increase the rate of positive outcomes?

Increased effectiveness

•Forget: “evaluation”

•Remember: “leveraging data for program excellence!”

We are busy delivering a program – how can we be

evaluators too?

•Julie Seeley, Spoon River College

Story from the Field:Leveraging data for program

excellence

The Formative Evaluation Toolkit: A Guided Tour

• Many pages (but not to worry!)

• Step-by-step guidance through the phases of evaluation

• Tools and templates for conducting your evaluation

• Templates for working meetings to accomplish evaluation tasks

• Checklists at the end of each chapter to keep you on track

• Blank pages for notes, questions, and insights

• Support for:

– Completing deliverables due to Champion Colleges

– Fulfilling grantee reporting requirements

What you will find in the FET:

•Data collection (chapters 3 & 4)

•Organizing for evaluation (chapter 2)

•Reporting on findings (chapter 5)

•Reflecting on findings (chapter 5)

Step-by-step guidance for:

Data Collection (Chapters 3 and 4)

• Process data

• Outcome data

• Stakeholder feedback

Data Types

• Counts of program participants

– Students completing workforce program courses

– Students participating in support services

• Information about program components, describing:

– Workforce programs and courses

– Math, English, and computer courses

Process Data(Chapter 3)

• Number of students who have:

– Received credits for prior learning

– Received a degree

– Received a certificate

– Received a non-credit certificate

– Become employed

Outcome Data(Chapter 3)

• “Extractable” v. “Real-Time” Data

• Make a data collection plan

– Identify the data you need to collect

– Figure out how you’ll collect it

• Work with an IR Partner

• Plan early for real-time data collection

• Stay organized with data storage

Important Considerations for Collecting Process & Outcome

Data

• Guidance for a working meeting to develop process and outcome data collection plans

• Tools:

– Tool 2: Data Choices Table (p. 42)

– Tool 3: Process Data Collection Plan Worksheet (p. 45)

– Tool 4: Outcome Data Collection Plan Worksheet (p. 54)

– Tool 5: Options for Collecting Employment Data (p. 57)

– Data Storage Excel Workbook (URL on p. 17)

FET Tools for Process and Outcome Data

• Hearing from students, internal partners, and community partners

• Looking for feedback that will help you to:

– Attract more students to the program

– Tailor services better

– Raise satisfaction levels

– Support students better for completion and employment

Stakeholder Feedback(Chapter 4)

• Decide what you want to learn

• Make a data collection plan

– Decide which stakeholders you want to hear from

– Identify the data collection methods you want to use

– Identify people responsible for collecting feedback

• Make data collection instruments

• Don’t collect too much data!

Important Considerations for Collecting Stakeholder Feedback

• Guidance for two working meetings to develop:

– A stakeholder feedback data collection plan

– Data collection instruments

• Tools:

– Tool 6: Stakeholder Feedback Evaluation Questions Worksheet (p. 61)

– Tool 7: Stakeholder Feedback Data Collection Plan Worksheet (p. 63)

– Tools 8 & 9: Survey, Interview, and Focus Group Protocol Templates (pp. 66, 82)

– Tools 10-12: Guidance for Conducting Interviews and Focus Groups (pp. 88, 90, 91)

FET Tools for Stakeholder Feedback

Organizing for Evaluation(Chapter 2)

• Evaluation Team Manager

• IR Partner

• Process and Outcome (P&O) Data Lead

• Stakeholder Feedback Lead

Assembling an Evaluation Team

• Review how the Plus 50 data tasks fit together

• Customize your evaluation timeline

Convene the Team for an Evaluation Launch Meeting

Reflection and Reporting(Chapter 5)

o Preliminary reporting: pulling together data with initial analysis

o Reflecting on findings

o Writing up memos:

Findings/Learnings

Recommendations (based on learnings and reflection)

Analysis, reflection, and reporting as an iterative process

• Guidance for working meetings to reflect:

– With the Plus 50 Team on key findings from the process and outcome data

– With the Plus 50 Team on key findings from the stakeholder feedback

– With the Plus 50 Advisory Committee on all key evaluation findings

• Tools:

– Tool 13: Evaluation Questions Tool (p. 92)

– Tool 14: Qualitative Data Analysis Template (p. 95)

– Tool 15: Memo Write-up Tool (including an memo outline) (p. 99)

FET Tools for Reflecting on and Writing up

Evaluation Findings

Working with C-PAD to Support Continuous

Improvement

1. Collect Process and Outcome Data

2. Gather Feedback from Program Participants and Partners

3. Share Results with Key Stakeholders

4. Plan for Program Improvement

Phase 5

Feeding two birds with one seed:Process and Outcome Data

in C-PAD

P&O data used for continuous

improvement=

P&O data used for grant-required

reporting to AACC

Questions?