Debra D. Bragg, Office of Community College

Research and Leadership, University of Illinois

Transfer

Terminal

Baccalaureate

Transfer

Terminal

Applied Baccalaureate

“…a bachelor’s degree designed to incorporate applied associate courses and degrees once considered as ‘terminal’ or non-baccalaureate level while providing students with the higher-order thinking skills and advanced technical knowledge and skills so desired in today’s job market.”

Townsend, Bragg, & Ruud (2008, p. 4)

Julia Panke Makela, Research Specialist & Project Director

Collin Ruud, Research Associate Stacy Bennett, Graduate Research Associate

http://occrl.illinois.edu/projects/nsf_applied_baccalaureate

Our targeted-research project aims to: ◦ Identify pathways to baccalaureate degrees in

technician education

◦ Analyze pathway designs, implementation, and outcomes

◦ Describe how AB degree programs operate and meet students' and employers' workforce needs

◦ Identify and widely disseminate promising and exemplary practices

Brief survey to identify established formal pathways to baccalaureate degrees

Follow-up survey on identified baccalaureate degree pathways on curriculum and instruction, accreditation and evaluation, enrollments and students served, partnerships with employers and other higher education institutions, and perceived impacts of ATE.

Case studies with 7–10 ATE projects and centers to uncover promising ideas and proven practices

Contacted all NSF-ATE Principal Investigators (PIs) with grants awarded between1992 and 2011 (~700 grants)

Inquired about: • degrees affiliated with the NSF-ATE project or center • fields of study • retention and recruitment of underrepresented

student populations at the baccalaureate-level • access to student-level data for baccalaureate degrees

Received 234 responses (36% of the sample)

24% of survey respondents reported associate degrees affiliated with their ATE project or center with no established pathway to the baccalaureate

Some survey non-participants offered insights into their decision not to participate: ◦ “Our Civil Engineering Practitioner Degree is an AAS

and therefore is a terminal degree. Our participation in the survey is probably not warranted.”

Baccalaureate degree pathways affiliated with ATE projects and centers fit both: • Traditional transfer patterns of AS or AA degrees

transferring to BS or BA degrees • Emerging pathways such as applied baccalaureate (AB)

and community college baccalaureate (CCB) degrees 42% (98 of all respondents) indicated that associate degree

programs had established formal baccalaureate degree pathways

20% (47 of all respondents) indicated at least one pathway began from an applied associate degree

0% 5% 10% 15% 20% 25% 30% 35%

Agricultural Technology

Marine Technology

Transportation Technology

Multimedia Technology

Civil and Construction Technology

Geospatial Technology

Chemical Technology

Nanotechnology

Telecommunications

Cyber Security and Forensics

Environmental Technology

Electronics

Energy

Biotechnology

Other

Computer and Information Technology

Manufacturing and Engineering Technology

Percent of Respondents Indicating Baccalaureate Degree Pathways

Analysis of 87 of established degree pathways

• Applied Associate Technical Baccalaureate (22) • Applied Associate Traditional Baccalaureate (32)

• Traditional Associate Technical Baccalaureate (11) • Traditional Associate Traditional Baccalaureate (47)

Degree Examples

Applied Associate AAA, AAS, AAAS, AAT, AET, AT Traditional Associate AA, AS Technical Baccalaureate BAA, BAS, BAAS, BAT, BT Traditional Baccalaureate BA, BS

20 respondents identified the following fields of study: • Biotechnology • Chemical technology • Computer and

information technology • Cyber security and

forensics • Electronics • Energy • Environmental

technology

CCB Defined…

Any form of baccalaureate degree awarded by an institution identified as a community college, technical college, two-year college, two-year or technical branch campus of a university system, or any other institution that primarily awards associate degrees.

• Manufacturing and engineering technology

• Marine technology • Nanotechnology • Telecommunications • Transportation technology

External Influences

Institutional Influences

Unit Influences

Latucca & Stark (2009), Contextual Influences on Academic Plans

Theoretical and Methodological Frameworks

• Program Quality

• Educational Significance

• Evidence of Effectiveness and Success

• Replicability and Usefulness to Others

Bragg et al. (2002), Sharing What Works: Exemplary and Promising

Programs Evaluation Criteria

Variety makes baccalaureate pathways in technician education challenging but compelling to study

Many questions: • How are programs designed? • What perceived needs are they addressing? • What features contribute to their effectiveness? • What do we know about student outcomes? • What can be learned from one program that can be

adopted or adapted in other settings?

• Debra D. Bragg • Email: dbragg@illinois.edu

OCCRL ◦ http://occrl.illinois.edu ◦ PH: 217-244-9390 ◦ E-mail: occrl@illinois.edu

• Check out our website:

occrl.illinois.edu

• Participate in our webinars

• Get on our listserv

• Receive the e-Info

• Friend our Facebook

• Receive our tweets

BUILDING REFLECTIVE LEADERSHIP: RESEARCH INTO PRACTICES ATE LEADERS USE TO DEVELOP AND MAINTAIN INDUSTRY-RELEVANT CURRICULUM, PROGRAMS, & INSTRUCTION

Louise Yarnall, Raymond McGhee, & Joseph Ames

Research goals

Deepen understanding about the industry-CC collaborative cycle to develop workforce programs

Analysis framed by research model based on past research and our findings; use model to: Tell rich stories about ATE Center cases Describe mechanisms for iteratively translating industry input

into curriculum, programs, and instruction Describe mechanisms for sustaining the curriculum, program,

and instruction collaboration with industry over time Describe common metrics of program success

Research background

Title: Community College Partnership Models for Workforce Education Sustainability and Integrated Instruction

4-year project, beginning Year 3 4 ATE Centers/Projects:

Wind energy, biotechnology, engineering technology, telecommunications and information technology

Different stages of engagement with industry in instructional program development: beginning, mid-life, mature

6-7 associated colleges Case studies

Research Team and Advisors

Louise Yarnall, PI Ray McGhee, co-PI Geneva Haertel Robert Murphy Carolyn Dornsife Joseph Ames, Ames

Assoc.

Nick Smith, Evaluator, Syracuse University

Frances Lawrenz, University of Minnesota

Cynthia Wilson, The League for Innovation in the Community College

Manjari Wijenaike, former ATE Center director

Steve Wendel, NCME

David Jonassen, University of Missouri

SRI Team and Ames Associates Evaluator and Advisory Panelists

Project Overview

Partnership sub-study: Evolution of relationships between industry and community

college in workforce programs Unique stories, common mechanisms to translate industry goals

into instructional programs

Classroom instruction sub-study: Tracing industry and ATE Center influences on instructional

programs Characterizing range of workforce education instructional

practices and curricula

Research products - Partnership

Cases of ATE Center activities contributing to life cycle of collaboration with industry in workforce program development ATE principal investigator activities Instructional goals Rapid development mechanisms Sustainability challenges

Research products - Instruction

Cases of ongoing, classroom-level processes that support continual instructional updates

Cases of technician education instruction

Peek at findings so far

Model of industry-community college instructional partnerships

Partnership sub-study: Early highlights & starting cases

Model: Findings and Uses

ATE community members can use this model to strengthen partnerships: Stepping back, seeing “big picture” of your work Using the categories in the model to “make sense” of

challenges you face, identify potential opportunities

Researchers use models to make sense of complex phenomena across multiple settings

Models emerge from past empirical research and theory; they evolve based on current data

Model: Strategic Need

Model: Formation Processes

Model: Partnership Capital

Model: Outcomes/Outputs

ATE-CC Partnership Conceptual Model

Strategic Need Address labor supply needs Retrain incumbent workers Improve technician training

ATE center role

Talking with industry

Organizing work groups with faculty

Marketing/outreach

Trust-building meetings

Industry community link Historic presence In region Articulates labor need first

FORMATION PROCESSES

Establishing trust/norms/comm. (Fusing social & org. capital)

PARTNERSHIP CAPITAL

Resource Leveraging Productive meetings: PD, new technology, standards alignment Establish agreements around equipment, labs / resources Instructional materials sharing Industry adjuncts

Creating partnership capital (Partnership implementation)

Student Certificate testing (student pays) Degrees/certificates obtained Job placement/internships

Classroom/ Faculty Degrees/certificates offered New courses created Instructional materials development

Workplace Prepared workers placed Employee training

OUTCOMES/ OUTPUTS

Sustaining the partnership (Producing results)

Organizational boundary maintenance

Partnership Complexity -# organizations -# sectors -# states

STAGES: Emergence Transition Maturity Critical Cross Roads

External Resources State & local funding 1/x

CC support Administrator support for ATE leader

Partnership sub-study: Early findings

Cases Uses: ATE community leaders can compare their own

situations to these cases, deriving insights

Case 1: Regionally scaling a program

ATE leader role: Facilitate regional industry, educators

Goal: Sequence for multi-college ET program

Rapid Development Mechanisms: Identify core courses that transfer

across local fields (boating & medical devices)

Crosswalk industry standards to courses

Sustainability Challenges: Sustain adults past 1 course

Case 2: National dissemination

ATE leader role: Moving national industry materials to

colleges Goal:

Provide low-cost, up-to-date, industry-made IT materials

Rapid Development Mechanisms: Identify IT platform providers with

materials Outreach to educators, pass costs to

students, free training & materials Sustainability Challenges:

Staying current

Case 3: Local industry exchange

ATE leader role: Develop instructional materials,

communicating with industry Goal:

Enhance existing industry-college partnership in biotech

Rapid Development Mechanisms: “SWAT” team capacity Division of labor around “safety

training” Sustainability Challenges:

Rust belt economy Biotech jobs pay half of old jobs Global companies, no local loyalty

Case 4: Boot camp to program

ATE leader role: Workforce program development

Goal: Expand boot camp to college program

Rapid Development Mechanisms: DACUM

Sustainability Challenges: Timing market need: VC dry up Keeping industry engaged Facilitating discussions between

educators/industry “shop math” vs. “college math”

Next steps

Partnership Study: Follow up interviews with stakeholders Development of cases, and possibly other tools

Instruction Study: Interviews to build cases: Describe 2 contrasting

partnerships’ specific classroom instructional goals and programs

Classroom data to build cases: Select tech classes representing different levels of technical content and different emphases on technical vs. professional skills: Instructional practice: Classroom observations and interviews Curriculum: Artifacts rated by expert panels

Thank you

louise.yarnall@sri.com

Stephen Magura Kelly N. Robertson

The Evaluation Center

Western Michigan University

Presented at the 2011 National ATE PI Conference Washington, DC, October 27, 2011

Funded by NSF grant # 0832874

Began 1992

Funding FY 11 - $64 million by NSF

Approximately 40 centers & 200 projects

Encompasses biotechnology, manufacturing, engineering, energy, IT

Located in community colleges nationwide

1. “Producing more science & engineering technicians to meet workforce demands”

2. “Improving the technical skills & general science, technology, engineering, & mathematics (STEM) preparation of these technicians” and

3. “(Of) the educators who prepare them”

Objective 1: Formulate a model for standardized measurement of outputs pertinent to ATE central goals 1, 2 and 3 that is relevant across different Projects and Centers.

Objective 2: Determine which outputs individual Project and Centers are measuring as concrete steps toward achievement of ATE’s central goals and propose additional outputs that could feasibly be measured.

Objective 3: Determine what types of evaluation designs individual ATE Projects and Centers are employing to determine impact and propose alternative or improved evaluation designs.

Promote scientific assessment of effectiveness

Application of objective effectiveness measurement strategies

Better understanding of variations in success of grantees

Return on investment of ATE portfolio to Congress

Objective 1. Existing material on ATE compiled from four sources: Selected ATE Project/Center progress and final

reports solicited by an NSF program official Project/Center evaluator reports previously

submitted to the ATE Resource Center ATE Project/Center websites ATE Projects/Centers described in the ATE

Impact publications (Patton, 2008 a,b).

Objectives 2 and 3. One ATE Project was analyzed in each of ten

industries and one ATE Center in each of seven industries.

The Project and Center chosen within each industry based on the most information available.

Purpose was to demonstrate that the proposed framework is applicable to ATE Projects and Centers across the range of applicable industries.

Projects and Centers are anonymous in the report.

Post-Secondary Secondary

Enrolled Completed/ Graduated

Retention Enrolled Completed/ Graduated

Retention

(1) (2) (2 ÷ 1) (1) (2) (2 ÷ 1)

A. Program ⃞ ⃞ ⃞ ⃞ ⃞ ⃞

B. Course ⃞ ⃞ ⃞ ⃞ ⃞ ⃞

C. Internship/ Apprenticeship

⃞ ⃞ ⃞ ⃞ ⃞ ⃞

D. Dual Program/ Dual Credit

⃞ ⃞ ⃞ ⃞ ⃞ ⃞

Post-Secondary Exposed* Secondary Exposed*

E. Software/ Materials ⃞ ⃞

Secondary Post-Secondary

Number of Educators who Complete… Elementary Middle High Faculty Industry Professional

Professional Development Workshops ⃞ ⃞ ⃞ ⃞ ⃞ ⃞

Professional Development Courses ⃞ ⃞ ⃞ ⃞ ⃞ ⃞

Professional Development Fellowships/Mentoring ⃞ ⃞ ⃞ ⃞ ⃞ ⃞

Professional Development Software/Materials* ⃞ ⃞ ⃞ ⃞ ⃞ ⃞

Note: *Including hard copy and audio/visual materials for professional development purposes

Study Objectives Current Project Current Center

Description Creates simulations that teach the underlying science principles of

biotechnology & nanotechnology.

Providing educators with professional development in

manufacturing.

2. Current Outputs

Pre/post test to assess student achievement in relation to the topics

the simulations intend to teach.

Track # of teachers trained & self-assessment of learning. Plan

to start asking teachers about implementation of learning.

3. Recommend Outputs Quality of the simulations.

Quality of PD course. Test teacher skills, changes in

classroom practices, & student learning.

4. Current Evaluation Design

Pre-test with repeated post-test. Post-training satisfaction measures.

5. Recommend Evaluation Design

Expert panel to assess quality of simulations. Compare student learning with

cohort receiving standard course.

Pre-test with multiple post test for PD.

Common ATE Project and Center outputs can be specified and potentially aggregated to yield output statistics for the national ATE program as a whole.

The proposed framework, consisting of the figures and the tables in the report, narrows down and partly standardizes the types of data collected across ATE projects and centers.

This standardization can result in meaningful aggregation of output measures that will make it possible to better determine program effectiveness.

Additional instrumentation must be developed to assess the quality of STEM educational and outreach resources and their impact on students’ and educators’ learning and behavior.

The evaluation framework is also useful because it identifies the gaps in instrumentation more precisely.

The evaluation framework is very comprehensive, but all elements are not always applicable to any individual ATE Project or Center.

This inherently quantitative data framework does not diminish the value of additional qualitative and narrative data that speak to the value, merit or worth of ATE programs.

Some aspects of the proposed framework are outside the scope of any individual ATE grant and would better be pursued through targeted research.

This report is not a final prescription, but may help frame further discussion of ATE evaluation.

My email: stephen.magura@wmich.edu

Thanks for your attention!

Ron Anderson rea@umn.edu

This project was funded by the National Science Foundation ATE Program for Targeted Research. The grant was to Colorado University’s DECA Project, Liesel Ritchie, PI, with a subcontract to Rainbow Research for Project I, Strategies for Improving Recruitment, Retention and Placement.

October 27, 2011

1

Community College completion rates embarrassing low at 20 to 40% within 8 years.

Advanced Technology Programs (ATP), while not as bad as non-ATP programs, still lose over 50% of their students before completion.

Gender inequality, a serious problem in NSF ATE projects

Recruitment of racial minorities improving in NSF ATE projects.

NSF ATE projects neglect student advising & other strategies to retain students

2

3

*Data from Program Improvement Projects in Western Michigan State annual ATE Survey by www.evalu-ate.org

4

Data from Program Improvement Projects in Western Michigan State annual ATE Survey: www.evalu-ate.org

5

Data from Program Improvement Projects in Western Michigan State annual ATE Survey: www.evalu-ate.org

6

Advanced Technology Programs (ATP) fail to Attract Women. Data graphed are First-term Enrollments by Gender for ATP & Non-ATP

Data are based on all students enrolled in Connecticut Community Colleges 1999-2009. (N=120,000)

Many organizations are trying to address the completion/success gap in 2-year colleges

Analytics movement attempting to forecast student dropouts

Whitehouse Committee on Measures of Student Success ◦ Appointed in 2010 ◦ Sept. 2011 interim report ◦ April, 2012 target for preliminary report ◦ Years before impact likely

7

Common Completion Metrics (National Governors Assoc.)

Voluntary Framework of Accountability (AAAC) Foundations of Excellence in the First College Year

(Gardner Institute) Complete College America Achieve, Inc (35 State network) Achieving the Dream (Database and Dashboards) Western Interstate Commission for Higher

Education (WICHE) – Human Capital Database Project Gates Foundation - funded analytics initiatives National Agenda for Analytics (EDUCAUSE)

8

Predictive Analytics (Capella U & others) Data Analytics (Sinclair Community College) Incisive Analytics (IncisiveAnalytics.com) Platinum Analytics (AstraSchedule.com) Action Analytics (Symposia in 2009 & 2010,

and EDUCAUSE in 2011) Learning Analytics (1st International Conference on

Learning Analytics, Feb. 27, 2011) Student Success Analytics (Purdue U., etc.)

9

Analytics is sometime used as synonymous with ‘analysis’ to sound impressive.

More precisely, ‘analytics’ refers to ‘predictive analytics,’ or analysis of trend data to predict future events of individuals or populations.

Current analytics does not follow individual course-taking histories across time, thus it is weak in providing individualized information that students can use.

10

37 40 39 37 46

0

10

20

30

40

50

60

70

80

90

100

2003 2004 2005 2006 2007

Percent of Students Completing Program X in each year, 2003-2008

Typical Analytics Data: Trend Line, not a Trajectory

(Trend lines fail to give any information about change in individual attributes overtime, only aggregates.)

11

Cohorts Showing Student Trajectories for 120,000 student histories in Conn.

12

Student-Pathway Trajectories showing Race Gaps

Data are all 2,407 students first enrolled Fall, 2005 in the Community College of Rhode Island system. Completion is defined as graduation, articulation, or completion of 48+ credits within 7 terms (4.5 years).

13

Recent, dynamic microsimulation techniques make it possible to follow individual course-taking histories (trajectories) across time

Thus, using student transcript data records, models can be built that simulate student enrollment decisions term by term..

The results give information that students and student advisors can use to greatly improve their chances of completing a program successfully.

14

Microsimulation model developed in Modgen programming language from Statistics Canada

Hundreds of thousands of student transcript records from the CCs of Connecticut and Rhode Island were used as test data sets.

For any given set of data, each scenario simulation is repeated for an equivalent sample of 5 million students to eliminate random variability, which only takes about 2-3 minutes.

15

MicroCC developed with Targeted Research funds from NSF ATE program.

Initial model includes 4 student choices or behaviors (details on next slide)

Model’s core (predictive factors) are derived from data at hand ◦ 28 separate logistic (and ordered logit) regression

models run to calculate coefficients for each factor and interaction that predicts success or completion

Multiple scenarios can be simulated by modifying either ◦ starting populations (mostly demographic factors) Gender, race, age, and initial full-/part-time status

effect coefficients for student decisions, or

16

1) Enrollment /re-enrollment choice in each term

2) Full vs Part Time enrollment in each term

3) Number of courses attempted

4) Successful completion of each course attempted

Process Decision Points: MicroCC Completes this Decision Sequence for each term of each Student

17

Success = completion of program (graduate, certificate, successful transfer, or completion of a required number of courses)

Total courses completed = completion of 12 or more courses within 10 terms (5 years)

18

Momentum Point One Passed - student completed 3 courses in first term

Momentum Point Two Passed - student completed 6 courses in year one

Stopout - student temporarily does not enroll in term X

Stopouts -total terms student stopped out

19

Used in MicroCC ◦ Gender (M/F) ◦ Race (W/B/L/O) ◦ Age (to 21/22+) ◦ Starting term enrollment full-time vs part-time

Data not available in 2010 for MicroCC model ◦ Financial aid in term X ◦ Concurrent job ◦ Marital status ◦ Prior postsecondary education

20

Data Restructuring – Creation of longitudinal file from term-level files can be done but it is time consuming.

Missing Data – Records on transfer status, graduations, and certificate completions may be incomplete or nonexistent.

Summer Term Challenge – can summer credits be ignored completely because there are so few regular students enroll in summer terms, or should credits and courses completed during the summer, be added into the counts for the previous term?

Developmental Courses -- Developmental courses were tracked but institutions handled them differently.

Transfer credits -- Are they added to new credits, and if so, when?

Simultaneous enrollments -- In Connecticut we found many students enrolled in multiple colleges during a single term.

21

Screen print from MicroCC with Student Success Model for Baseline scenario with RI and CT data

22

◦ Data for MicroCC microsimulations came from two State enrollment databases: Rhode Island Community College – 5 annual cohorts

with most analysis just on the 2,502 students first enrolled in Fall 2005 for 4.5 years

Connecticut Community College system – 276,469 students in 10 cohorts beginning Fall 1999 to 2009.

23

Screen print from MicroCC with Student Pathways Models for Baseline scenario with RI and CT data

Sample output table for student success rates by term

0

0.05

0.1

0.15

1 2 3 4 5 6

% c

ompl

eted

terms 1 to 6

Sample chart of growth of student completions from above table

24

25

Gaps in success can be deconstructed, identifying the student pathways that created specific portions of the gap.

These results have direct relevance for students and guidance counselors, toward improving success rates.

26

1) Enrollment /re-enrollment choice in each term

2) Full vs Part Time enrollment in each term

3) Number of courses attempted

4) Successful completion of each course attempted

Process Decision Points: MicroCC Completes this Decision Sequence for each term of each Student

27

Most (90%) CT students in ATPs were in engineering and manufacturing programs. The remainder were in IT, network, and misc. science and technology programs.

The 7,310 ATP enrollees in CT were only 6% of all CC students.

As shown in the next chart, ATP students has a 17% higher completion rate than non-ATP students.

28

29

Source: 7,310 ATE Students in Connecticut CCs 2000-2009

The amount of impact they have on success depends upon specific regions, schools, and curricular programs.

If a student enrolls full time plus works full time and has children to raise, s/he might not do well in coursework and thus not keep up the momentum toward completion.

30

But both students and their advisors need to understand how crucial these decisions are to pathway success: 1. To enroll continuously – no stop outs 2. To enroll full time 3. To take the larger numbers of courses each term,

within reason 4. To pass the courses attempted.

The simulation model incorporates these decisions, not just at first enrollment, but at every term in which the student is enrolled.

31

Remaining charts from microsimulations illustrate how student decisions influence different subgroups of students within ATP programs in CT.

Example 1, shows elements of gap between CT and ATP White and Hispanic men

Example 2, highlights the higher completion rates of women over men in CT ATPs

32

33

34

Source: 7,310 ATE Students in Connecticut CCs 2000-2009

35

43

35 37

49 50 48

0

10

20

30

40

50

60

White Black Hispanic

Men

Women

Women Outpace Men in all Race Categories - Percent of Students Completing their Programs

by Gender & by Race in Conn. N=7,310 ATE students

Microsimulation can uncover enrollment decisions that have huge effects on student success.

These student decisions can sometimes explain demographic differences.

Adding additional data, e.g., job history, financial aid and retention interventions, e.g., mentoring, as factors in the models, can make the methodology even more powerful.

Enrollment forecasting can be done with greater precision. The model could also be extended to include post-schooling

job trajectories as well.

36

For More information contact Ron Anderson rea@umn.edu or 952-473-5910

1. The ATE program should invest in student tracking data systems, either in conjunction with existing student record systems or, better yet, a separate data system to which ATE-funded projects had to contribute.

2. ATE-funded projects should be encouraged or required to address and report on student advising practices.

3. Training should be developed for high school and community college student advisors regarding the needs of STEM students

4. Recruitment of women (with improved advising) into STEM pathways needs to be given greater priority

37

NSF ATE projects may be neglecting student advising & related strategies to retain students.

Of the 305 projects and centers recently funded by the NSF ATE program, only two mentioned “student advising” or “guidance counseling” in their title or abstract. However, 10 projects (1%) mentioned “counselors.”

ATE projects could utilize the findings of MicroCC simulations as guides for student advising. A system for student progress coaching and advising is needed with every ATE funded project 38

39

Microsimulations should be run on many more States, college populations, and ATE program populations, so that findings could be tailored to specific groups of at-risk students.

Input data for simulations should be expanded to include job status, financial aid, and other items relevant to student success.

Microsimulation should be extended to include articulation and job acquisition processes.

40

For more information contact:

Ron Anderson rea@umn.edu

41