Update on the Student Success Scorecard...Update on the Student Success Scorecard April May June...

Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013

Matthew Wetstein

Vice President of Instruction

San Joaquin Delta College

President Elect of the RP Group

Home Run or Triple Play?

CIO Conference San Diego

October 31, 2013

Update on the Student Success Scorecard


Highlights

• Media coverage of the scorecard

• Page and website visits

• Lineup changes for 2014

• Some research on the scorecard

• Issues to consider as CIO

• Hat tip to Craig Justice and

Sherrie Guerrero

2 Update on the Student Success Scorecard


B E AT W R I T E R S AN D W E B S I T E V I S I T S

Media Coverage

3


Media Coverage

• Chancellor’s Office major media push –

Launch Date April 8, 2013

• Week before launch – social media and talking

points

• Webinar for PIO and other interested parties

• Goal – to build “buzz” about the launch through

Twitter and Facebook accounts

• Indicate what could be found on the scorecard

site

Source: Paige Marlatt Dorr, “Social Media and Talking Points Summary,” California Community Colleges Chancellor’s

Office, September 30, 2013



Media Coverage

• The Scorecard Talking Points…

• New accountability tool with performance metrics

• Perhaps most transparent in the nation

• Demonstrates the importance of becoming

college-prepared in HS

• Disaggregated data across a set of metrics

• Success metrics include “momentum points”

correlated with completion and wage gains

• Not intended to rank colleges – provide

benchmark data for sustained improvement Source: Paige Marlatt Dorr, “Student Success Scorecard Talking Points,” California Community Colleges Chancellor’s

Office, September 30, 2013



Media Coverage



Media Coverage

7

Positive Themes

Positive coverage of

school rates

Transparency of the

system

Usefulness for students

and public

Negative Themes

Students

underperforming

Low rates compared to

state averages or rates


30 News stories on day

of scorecard release

National outlets

Chronicle of Higher

Education

Inside Higher Education

Every major paper in the

state

Small outlets

Willits News

Woodland Daily

Democrat

Hanford Sentinel


Media Coverage

8

Most headlines

were neutral

Positive

headlines

outweighed

negative 8 to 6

Editorials

focused on

transparency


0

2

4

6

8

10

12

14

Positive Neutral Negative

Analysis of News Stories & Editorials Covering the Student Success

Scorecard, April 2013

News

Editorial


Website Visits


April May June July August September

Page Views 111703 31058 18758 18926 22830 15808

Website Visits 36593 17756 11149 11014 13564 8951

0

20000

40000

60000

80000

100000

120000

Student Success Scorecard Web Site Visits and Page Views (April - September 2013)

Source: Paige Marlatt Dorr, “Google Analytics for Student Success Scorecard Website,” California

Community College Chancellor’s Office, October 2013


L I N E U P C H AN G E S F O R 2 0 1 4

Planned Updates



Planned Updates

11

SMALL CHANGES & COSMETICS

Making the size of cohorts and sub-groups more

visible

Fixes the problem of not knowing how many people we’re

really talking about

New problem – you will see cell suppression of 10 students

or less (NA or asterisk)

Revisions in the “look” of the dashboard

Board interaction is still required and minutes of

meeting need to be submitted to the system office for

reporting (March 2014 deadline for 2013 report)



Planned Updates

12

REVISED CALCULATION OF PERSISTENCE

Used to be TERM TO TERM TO TERM enrollment by a

student completing 6 units or more – in other words, 3

consecutive major terms

NOW it will include the above PLUS students who complete

a CERTIFICATE, DEGREE or TRANSFER

The logic is that these are students who in fact have

completed and or persisted elsewhere (UC/CSU/ISP/OOS)

Persistence scores will go up and get recalculated for each

cohort



Planned Updates

13

REVISED REMEDIAL METRICS

Math Cohort:

First attempt of a Math course in two to four levels below

transfer:

CB03 COURSE-TOP-CODE = 1701*

SB01 STUDENT IDENTIFIER STATUS = S (only students

with valid SSN)

CB21 COURSE-PRIOR-TO-COLLEGE-LEVEL = B, C, or D

CB04 COURSE-CREDIT-STATUS = C or D

C = Credit, NOT degree applicable D = Credit, degree

applicable

Prior iteration was just using the CB21 coding (levels below

transfer)



Planned Updates

14

STUDENT-TO-COUNSELOR RATIO

3 options were considered for making this happen

Method 1 – ASA Employee assignment

Method 2 – ASA + TOP Code FTEs for

Counseling/Guidance classes

Method 3 – ASA + TOP but maximum of 1 FTE per

person



Planned Updates

15

How it will work…

Student headcount and counseling Full-Time Equivalent

(FTE) totals. FTEs are obtained from work assignments

and course instruction load. The ratio is calculated with

the following formula:

Sum (Fall Unduplicated Student Headcount)

Sum (Fall Counseling FTEs)

No EOPS or DSPS students or counselors count in the

metric (Unduplicated headcount – EOPS – DSPS)

The denominator is a little tricky…



Planned Updates

16

Counseling Denominator

Employees with ASA assignments of

6300 Student Counseling and Guidance

6310 Counseling and Guidance

6330 Transfer Programs

6340 Career Guidance

6390 Other Student Counseling and Guidance

OR TEACHING FTES IN

493010 Guidance

493011 Interpersonal Skills

493012 Job Seeking/Changing Skills

493013 Academic Guidance



Planned Updates

17

Having seen the data…let me echo the conference

theme… Pay Attention to Your Data

Many schools have good data – BUT preliminary

investigation suggests some do not…

EXAMPLES

13 unnamed colleges had no ASA staffing assignment for

counselors – so they had student to counselor ratios ranging

from 28,412 to 1 (bad data) to 753 to 1 (good data?)

2 of the 13 colleges were in the 18,000 to 1 range

4 colleges have zero ratios if using the planned method –

i.e. no data for either ASA assignments or FTEs for

Guidance/Counseling classes…hmmm…



Planned Updates

18

POINT – for some districts, the HR data for the EB08

Staffing Assignment or EJ01 Employee Assignment Type

are “dirty” “squishy” “missing in action” or “a hanging

curveball”

STUDENT COUNSELOR RATIO Webinar upcoming on

November 7, 1:30-2:30 pm

www.cccconfer.org

(888) 886-3951

Passcode = 614762

Cellphones 913-312-3202


http://www.cccconfer.org/


S AB E R M E T R I C S

Research on the Student Success Scorecard Metrics



Research

20

Bob Pacheco, Mira Costa – Student Success Scorecard

Metrics

Regression analysis

Measurement reliability over time

Multivariate analysis of the so-called SPAR rate

Alice van Ommeren, CCCCO Dean of Research &

Accountability

Equity Index

Multivariate models for peer group clusters

Predictor variables for SPAR rate



Research

21

Predictors of SPAR Rate

Median Household Income in the college area

Educational attainment – BA or higher for adults 25 +

Proximity to CSU or UC

Size of college cohort

SPAR Rate for Hispanics

Same variables as above

Puente Program



Research


y = 0.0002x + 36.572 R² = 0.3333

20.0

30.0

40.0

50.0

60.0

70.0

0 20000 40000 60000 80000 100000 120000 140000 160000 180000

Transfer/Degree Completion Rate by Median Household Income, 2006-07 Cohort

Foothill College

SD City SD Mesa

One of the best

predictors of

student progress

and completion

is the relative

wealth of the

area

Even so, some

colleges out-

perform

expectations on

these measures

Source: Chancellor’s Office Student Success Scorecard; U.S. Census Bureau


Research


y = 0.2695x + 39.78 R² = 0.3708

25.0

35.0

45.0

55.0

65.0

75.0

0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0

Transfer/Degree Completion Rate by Adult Educational Attainment, Age 25+

Another strong

predictor of

student progress

and completion

is the

educational

attainment

rates of the area

Source: Chancellor’s Office Student Success Scorecard; U.S. Census Bureau


Research


Variable Coefficient Std. Error Conf. Level

Cohort Size (in 100s) .100 .001 97.7%

Educational Attainment .154 .053 99.9%

Median Income (in 1000s) .087 .000 99.1%

Miles to CSU/UC -.023 .023 84.1%

Puente Program (Yes = 1) -1.481 1.245 88.1%

Constant 36.9 2.206

R Square .439 5.99

F Test 15.945 99.9%

Predicting the 2012-13 SPAR Rate Using a 5 Variable Model (2006-07 Cohort)


Research


y = -0.0027x + 50.841 R² = 0.0118

25.0

35.0

45.0

55.0

65.0

75.0

0 500 1000 1500 2000 2500

Current Student to Counselor Ratios and Transfer/Degree Completion Rate, 2006-07

Cohort Student counselor

ratio…

Hypothesis: where

this figure is higher,

we should expect

lower SPAR rates,

all other things

being equal

NOTE – outliers of

zero and those

above 4,000 omitted

Source: Chancellor’s Office Student Success Scorecard; Chancellor’s Office


Research

26

SPAR Rate for Hispanics

5 variable model does a poor job predicting

transfer/completion rates for this segment

R Square value of only .085 – variables only explain 8% of

the variance in the dependent variable

Only significant variable was educational attainment (97%

confidence level)

Puente Program indicator had a negative coefficient but

small effect in the equation (-.840, not significant)



M AN A G E R I A L C O N S I D E R AT I O N S

Important Take-Aways for Chief Instructional Officers



Take-Aways

28

Be sure your team cleans your data dugout

Don’t expect metric magic…past patterns help predict

future performance

Using the scorecard for managerial performance ratings

not the best idea

Time lag

Changes at the margins

False hopes

Set reasonable standards of student achievement

Aspirational goal setting versus reality



Take-Aways


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010

Retention Rate

Success Rate

Source: Chancellor’s Office Data Mart, Course Retention & Completion Rates

Patrick Perry, Greg Stoup and Bob Pacheco


Matt Wetstein

San Joaquin Delta College [email protected]

30

mailto:[email protected]

Update on the Student Success Scorecard...Update on the Student Success Scorecard April May June...

Documents

Transcript of Update on the Student Success Scorecard...Update on the Student Success Scorecard April May June...