Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Matthew Wetstein
Vice President of Instruction
San Joaquin Delta College
President Elect of the RP Group
Home Run or Triple Play?
CIO Conference San Diego
October 31, 2013
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Highlights
• Media coverage of the scorecard
• Page and website visits
• Lineup changes for 2014
• Some research on the scorecard
• Issues to consider as CIO
• Hat tip to Craig Justice and
Sherrie Guerrero
2 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
B E AT W R I T E R S AN D W E B S I T E V I S I T S
Media Coverage
3
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Media Coverage
• Chancellor’s Office major media push –
Launch Date April 8, 2013
• Week before launch – social media and talking
points
• Webinar for PIO and other interested parties
• Goal – to build “buzz” about the launch through
Twitter and Facebook accounts
• Indicate what could be found on the scorecard
site
Source: Paige Marlatt Dorr, “Social Media and Talking Points Summary,” California Community Colleges Chancellor’s
Office, September 30, 2013
4 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Media Coverage
• The Scorecard Talking Points…
• New accountability tool with performance metrics
• Perhaps most transparent in the nation
• Demonstrates the importance of becoming
college-prepared in HS
• Disaggregated data across a set of metrics
• Success metrics include “momentum points”
correlated with completion and wage gains
• Not intended to rank colleges – provide
benchmark data for sustained improvement Source: Paige Marlatt Dorr, “Student Success Scorecard Talking Points,” California Community Colleges Chancellor’s
Office, September 30, 2013
5 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Media Coverage
6 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Media Coverage
7
Positive Themes
Positive coverage of
school rates
Transparency of the
system
Usefulness for students
and public
Negative Themes
Students
underperforming
Low rates compared to
state averages or rates
Update on the Student Success Scorecard
30 News stories on day
of scorecard release
National outlets
Chronicle of Higher
Education
Inside Higher Education
Every major paper in the
state
Small outlets
Willits News
Woodland Daily
Democrat
Hanford Sentinel
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Media Coverage
8
Most headlines
were neutral
Positive
headlines
outweighed
negative 8 to 6
Editorials
focused on
transparency
Update on the Student Success Scorecard
0
2
4
6
8
10
12
14
Positive Neutral Negative
Analysis of News Stories & Editorials Covering the Student Success
Scorecard, April 2013
News
Editorial
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Website Visits
9 Update on the Student Success Scorecard
April May June July August September
Page Views 111703 31058 18758 18926 22830 15808
Website Visits 36593 17756 11149 11014 13564 8951
0
20000
40000
60000
80000
100000
120000
Student Success Scorecard Web Site Visits and Page Views (April - September 2013)
Source: Paige Marlatt Dorr, “Google Analytics for Student Success Scorecard Website,” California
Community College Chancellor’s Office, October 2013
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
L I N E U P C H AN G E S F O R 2 0 1 4
Planned Updates
10 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
11
SMALL CHANGES & COSMETICS
Making the size of cohorts and sub-groups more
visible
Fixes the problem of not knowing how many people we’re
really talking about
New problem – you will see cell suppression of 10 students
or less (NA or asterisk)
Revisions in the “look” of the dashboard
Board interaction is still required and minutes of
meeting need to be submitted to the system office for
reporting (March 2014 deadline for 2013 report)
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
12
REVISED CALCULATION OF PERSISTENCE
Used to be TERM TO TERM TO TERM enrollment by a
student completing 6 units or more – in other words, 3
consecutive major terms
NOW it will include the above PLUS students who complete
a CERTIFICATE, DEGREE or TRANSFER
The logic is that these are students who in fact have
completed and or persisted elsewhere (UC/CSU/ISP/OOS)
Persistence scores will go up and get recalculated for each
cohort
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
13
REVISED REMEDIAL METRICS
Math Cohort:
First attempt of a Math course in two to four levels below
transfer:
CB03 COURSE-TOP-CODE = 1701*
SB01 STUDENT IDENTIFIER STATUS = S (only students
with valid SSN)
CB21 COURSE-PRIOR-TO-COLLEGE-LEVEL = B, C, or D
CB04 COURSE-CREDIT-STATUS = C or D
C = Credit, NOT degree applicable D = Credit, degree
applicable
Prior iteration was just using the CB21 coding (levels below
transfer)
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
14
STUDENT-TO-COUNSELOR RATIO
3 options were considered for making this happen
Method 1 – ASA Employee assignment
Method 2 – ASA + TOP Code FTEs for
Counseling/Guidance classes
Method 3 – ASA + TOP but maximum of 1 FTE per
person
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
15
How it will work…
Student headcount and counseling Full-Time Equivalent
(FTE) totals. FTEs are obtained from work assignments
and course instruction load. The ratio is calculated with
the following formula:
Sum (Fall Unduplicated Student Headcount)
Sum (Fall Counseling FTEs)
No EOPS or DSPS students or counselors count in the
metric (Unduplicated headcount – EOPS – DSPS)
The denominator is a little tricky…
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
16
Counseling Denominator
Employees with ASA assignments of
6300 Student Counseling and Guidance
6310 Counseling and Guidance
6330 Transfer Programs
6340 Career Guidance
6390 Other Student Counseling and Guidance
OR TEACHING FTES IN
493010 Guidance
493011 Interpersonal Skills
493012 Job Seeking/Changing Skills
493013 Academic Guidance
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
17
Having seen the data…let me echo the conference
theme… Pay Attention to Your Data
Many schools have good data – BUT preliminary
investigation suggests some do not…
EXAMPLES
13 unnamed colleges had no ASA staffing assignment for
counselors – so they had student to counselor ratios ranging
from 28,412 to 1 (bad data) to 753 to 1 (good data?)
2 of the 13 colleges were in the 18,000 to 1 range
4 colleges have zero ratios if using the planned method –
i.e. no data for either ASA assignments or FTEs for
Guidance/Counseling classes…hmmm…
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Planned Updates
18
POINT – for some districts, the HR data for the EB08
Staffing Assignment or EJ01 Employee Assignment Type
are “dirty” “squishy” “missing in action” or “a hanging
curveball”
STUDENT COUNSELOR RATIO Webinar upcoming on
November 7, 1:30-2:30 pm
www.cccconfer.org
(888) 886-3951
Passcode = 614762
Cellphones 913-312-3202
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
S AB E R M E T R I C S
Research on the Student Success Scorecard Metrics
19 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
20
Bob Pacheco, Mira Costa – Student Success Scorecard
Metrics
Regression analysis
Measurement reliability over time
Multivariate analysis of the so-called SPAR rate
Alice van Ommeren, CCCCO Dean of Research &
Accountability
Equity Index
Multivariate models for peer group clusters
Predictor variables for SPAR rate
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
21
Predictors of SPAR Rate
Median Household Income in the college area
Educational attainment – BA or higher for adults 25 +
Proximity to CSU or UC
Size of college cohort
SPAR Rate for Hispanics
Same variables as above
Puente Program
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
22 Update on the Student Success Scorecard
y = 0.0002x + 36.572 R² = 0.3333
20.0
30.0
40.0
50.0
60.0
70.0
0 20000 40000 60000 80000 100000 120000 140000 160000 180000
Transfer/Degree Completion Rate by Median Household Income, 2006-07 Cohort
Foothill College
SD City SD Mesa
One of the best
predictors of
student progress
and completion
is the relative
wealth of the
area
Even so, some
colleges out-
perform
expectations on
these measures
Source: Chancellor’s Office Student Success Scorecard; U.S. Census Bureau
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
23 Update on the Student Success Scorecard
y = 0.2695x + 39.78 R² = 0.3708
25.0
35.0
45.0
55.0
65.0
75.0
0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0
Transfer/Degree Completion Rate by Adult Educational Attainment, Age 25+
Another strong
predictor of
student progress
and completion
is the
educational
attainment
rates of the area
Source: Chancellor’s Office Student Success Scorecard; U.S. Census Bureau
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
24 Update on the Student Success Scorecard
Variable Coefficient Std. Error Conf. Level
Cohort Size (in 100s) .100 .001 97.7%
Educational Attainment .154 .053 99.9%
Median Income (in 1000s) .087 .000 99.1%
Miles to CSU/UC -.023 .023 84.1%
Puente Program (Yes = 1) -1.481 1.245 88.1%
Constant 36.9 2.206
R Square .439 5.99
F Test 15.945 99.9%
Predicting the 2012-13 SPAR Rate Using a 5 Variable Model (2006-07 Cohort)
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
25 Update on the Student Success Scorecard
y = -0.0027x + 50.841 R² = 0.0118
25.0
35.0
45.0
55.0
65.0
75.0
0 500 1000 1500 2000 2500
Current Student to Counselor Ratios and Transfer/Degree Completion Rate, 2006-07
Cohort Student counselor
ratio…
Hypothesis: where
this figure is higher,
we should expect
lower SPAR rates,
all other things
being equal
NOTE – outliers of
zero and those
above 4,000 omitted
Source: Chancellor’s Office Student Success Scorecard; Chancellor’s Office
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Research
26
SPAR Rate for Hispanics
5 variable model does a poor job predicting
transfer/completion rates for this segment
R Square value of only .085 – variables only explain 8% of
the variance in the dependent variable
Only significant variable was educational attainment (97%
confidence level)
Puente Program indicator had a negative coefficient but
small effect in the equation (-.840, not significant)
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
M AN A G E R I A L C O N S I D E R AT I O N S
Important Take-Aways for Chief Instructional Officers
27 Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Take-Aways
28
Be sure your team cleans your data dugout
Don’t expect metric magic…past patterns help predict
future performance
Using the scorecard for managerial performance ratings
not the best idea
Time lag
Changes at the margins
False hopes
Set reasonable standards of student achievement
Aspirational goal setting versus reality
Update on the Student Success Scorecard
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Take-Aways
29 Update on the Student Success Scorecard
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Retention Rate
Success Rate
Source: Chancellor’s Office Data Mart, Course Retention & Completion Rates
Patrick Perry, Greg Stoup and Bob Pacheco
Home Run or Triple Play? | Matthew E. Wetstein | October 31, 2013
Matt Wetstein
San Joaquin Delta College [email protected]
30
Top Related