Replacing “Ready, Aim, Fire” with

“Research, Inform, Action”

Today’s Session

The Kansas City STEM Alliance and the Kansas City Area Education

Research Consortium collaborate with schools and nonprofits to recruit and

transition youth into STEM programs across the metropolitan area. Come

explore how research with local students, schools, out-of-school programs,

and community volunteers informs program decisions and illustrates a

return-on-investment of resources leveraged by businesses, K-12 schools,

higher education, and foundations. Also participate in discussions about

how what Kansas City is learning can be translated to inform efforts in your

region.

2

Our VisionThe vision of the KC STEM Alliance us is to see that

a diverse, innovative and sustainable STEM

workforce becomes a reality.

By developing an environment that leverages the

strengths of educators, STEM organizations, and

local industry we can create a collaborative network

to encourage and sustain STEM careers.

3

Working together will allow this…

KC STEM Alliance Partners

4

Building on Our Success: 2006-2012

• More than 27 area school districts and higher education

partners to provide curriculum, professional development for

teachers

• Nationally recognized Project Lead the Way (PLTW) and

US FIRST - 15,000 students in the Kansas City Metro

area

• Since 2006 – well over 20,000 students have participated in

these two programs.

• Significant resources – private, public sectors and school

districts

5

Our strategy

• Ongoing program evaluation and data

collection

• Increase participation of existing initiatives

• Seek out new partnerships and opportunities

• Raise awareness about STEM in Kansas City

• Secure support from area companies and

organizations

6

Beyond Descriptive

• KC-AERC works with local education programs to

help create measures of program effects on

students

• Traditional quantifiable measures – Ex. Reading or Math gains pre and post tests

• Non-quantifiable skill measures– Ex. Socio-emotional skills, attitudes towards

science

Methods

• Discuss desired program goals and effects with

program leaders

• Research appropriate instruments for analyzing

effects on students

• Devise appropriate statistical model– Mixed methods of quantitative and qualitative

• Collaborate with program leaders on final design

Quantitative Methods

• Treatment and control groups to identify if effects can be

linked to program– “Gold Standard” in attempting to find causality– Many times difficult to find and engage a control group

• Regression analysis to factor a variety of background

characteristics into results

• Use of Pre and Post tests to identify growth through

program (T-tests, ANOVA)

Qualitative Methods

• Focus groups to gather thoughts from those

closest to subject area

• Use quantitative methods to identify strong and

weak areas of program, qualitative investigation

of why some aspects work

• Interviews with leaders for in depth details of

program implementation

KC STEM Alliance Initial Research Questions

13

• Lower participation rate among minorities (when

compared to overall school population – FIRST)

• Low female participation in STEM engineering

programs – but, high in biomedical programs• Higher participation of females in urban districts

• Urban programs are less robustenrollment, participation, funding and support

What did we learn in the first 6 months:

What are issues raised by these pilots?

Process Data for Program Improvement:

• Comprehensive Program

Implementation with School Districts

• Definitions of Program Participation

• Understanding Within-Group Variation

in Number of PLTW Courses Complete

• Understanding FIRST Robotics

Participation Systematically

• PLTW Course Outcome Data

Outcome Data for

Program Impact:

High School Data

Formats and

Availability

Data on College

Outcomes

Logic Model for KC STEM Alliance Evaluation

Highlights of FIRST Robotics Surveys 2011-12

– FIRST Robotics, the majority of participants are White and Male.

– Majority have lived in the United States their whole life, come from households where English is spoken, and have highly educated parents (i.e., college degree or beyond) who in turn hold high expectations for their children.

– Slightly more than half of students surveyed (54%) were participating in FLL for the first time.

Highlights of FIRST Robotics Surveys

FIRST Coaches reported:

• 72% participating in a FIRST competition

• Of those participating, 44% participated in more than one competition.

• The number of mentors per team ranged from 0-15, with 24% of coaches

reporting their teams did not have even one mentor.

• Every single coach whose team had a mentor(s) reported their mentors

helped in “mechanical component design;” less than half of coaches

reported that their mentors helped with marketing, business plans,

computer applications, or website development.

Highlights of FIRST Robotics Surveys

FIRST Volunteers/Mentors (N = 77) reported:

• Majority of Event Volunteers were White and Male.

• Almost all volunteers lived in the Kansas City metropolitan area.

• Majority employed full-time and represented a variety of occupations.

• When asked about their satisfaction with their work in the FIRST Robotics

program, they gave responses that were generally very positive; however,

there were relatively less positive responses to questions about the

perceived effectiveness of information provided about the volunteer job.

KC 2012 FIRST Robotics Mentors

Think with a partner:

What is the

difference

between

“EVALUATION”

and

“ASSESSMENT?”

Typically,

“EVALUATION” =

program level and

“ASSESSMENT” =

individual level

TYPES of DATA:

• Participant Level (interest,

engagement, achievement)

• Program Level (quality

activities, professional

development)

• Systems Level (Policy

changes)

How can you make this happen for your organization?

Research, Inform, Action

• Two-Way Street

• Evaluation Planning to Implementation to

Utilization– Step 1: Program Purpose– Step 2: Data– Step 3: Plan to Obtain Data– Step 4: Communication/Share Data

Step 1: Document the nature and purpose of the program

• What is the program?

• What is the need?

• How is the program meeting the need?

• What are the goals of the program in terms of measurable

outcomes?– Short-term– Long-term

Note: Increases in knowledge, skills, and attitudes are

laudable goals.

Research, Inform, Action

• Inputs

• Outputs

• Measurable Outcomes/Goals

• Evidence/Possible Sources of Evidence

(Data)

How can you make this happen for your organization?

Step 2: Identify data needed to measure progress against goals

• Choose just the important data; avoid collecting data

you will not use

• Look for data among the people you directly serve

• Consider:– Who, when, and how often participants attend – What participants know and can do

Note: Evidence of learning is more valuable than

what participants say they learned.

Step 3: Develop a plan to obtain the data with current resources

• Brainstorm with team: –What constitutes success?–What evidence can be collected?–What notes/data can frontline staff

record?–What data do you need assistance with

gathering and understanding?• RE – visit goals to see if they capture everything that is important

Step 4: Share how the data will be used

• Provide feedback to stakeholders about how to

improve what we do

• Information to different audiences:– Current and future funders– Schools/School districts– Parents/Students/Families– Participants

Research, Inform, Action

• Two-Way Street

• Evaluation Planning to Implementation to

Utilization– Step 1: Program Purpose– Step 2: Data– Step 3: Plan to Obtain Data– Step 4: Communication/Share Data

Replacing “Ready, Aim, Fire” with “Research, Inform, Action”

Dr. Leigh Anne Taylor Knight

Executive Director, KC-AERC

lknight@kcaerc.org

913-396-3214

Laura Loyac0no

Director, KC STEM Alliance

loyaconol@kcstem.org

816-665-3823