Using Available Data and Analysis to Inform Comprehensive Program Review

Using Available Data and Analysis to Inform Comprehensive Program Review Comprehensive Program Review Training, March 2008Kristina Kauffman, Associate Vice Chancellor Institutional Effectiveness

Comprehensive Program Review Training, March 2008Kristina Kauffman, Associate Vice Chancellor Institutional Effectiveness

Data and AnalysisData and Analysis

Data –noun

1.a pl. of datum 2.(used with a plural verb ) individual facts, statistics, or items

of information 3.(used with a singular verb ) a body of facts; information —Usage note Data is a plural of datum, which is originally a

Latin noun meaning “something given.” Today, data is used in English both as a plural noun meaning “facts or pieces of information” (These data are described more fully elsewhere) and as a singular mass noun meaning “information”: Not much data is available on flood control in Brazil. It is almost always treated as a plural in scientific and academic writing. In other types of writing it is either singular or plural. The singular datum meaning “a piece of information” is now rare in all types of writing. In surveying and civil engineering, where datum has specialized senses, the plural form is datums.*

Data for RCCD Program Review purposes = numbers*Dictionary.com Unabridged (v 1.1), Based on the Random House Unabridged Dictionary, © Random House, Inc.

2006.

Data –noun

1.a pl. of datum 2.(used with a plural verb ) individual facts, statistics, or items

of information 3.(used with a singular verb ) a body of facts; information —Usage note Data is a plural of datum, which is originally a

Latin noun meaning “something given.” Today, data is used in English both as a plural noun meaning “facts or pieces of information” (These data are described more fully elsewhere) and as a singular mass noun meaning “information”: Not much data is available on flood control in Brazil. It is almost always treated as a plural in scientific and academic writing. In other types of writing it is either singular or plural. The singular datum meaning “a piece of information” is now rare in all types of writing. In surveying and civil engineering, where datum has specialized senses, the plural form is datums.*

Data for RCCD Program Review purposes = numbers*Dictionary.com Unabridged (v 1.1), Based on the Random House Unabridged Dictionary, © Random House, Inc.

2006.

Data at RCCD Comes From

Data at RCCD Comes From

Information Services (Rick Herman) Raw data from Datatel Reports organized around regular requests

such as registration statistics Institutional Reporting (Raj Bajaj)

Current term enrollments Current term FTES Distributions Current term assignments Scheduling patterns

Institutional Research (David Torres) Past term(s) data including all in

Institutional Reporting’s list above

Information Services (Rick Herman) Raw data from Datatel Reports organized around regular requests

such as registration statistics Institutional Reporting (Raj Bajaj)

Current term enrollments Current term FTES Distributions Current term assignments Scheduling patterns

Institutional Research (David Torres) Past term(s) data including all in

Institutional Reporting’s list above

InformationInformation

Information1. knowledge communicated or received concerning a particular fact or

circumstance; news: information concerning a crime. 2. knowledge gained through study, communication, research,

instruction, etc.; factual data: His wealth of general information is amazing.

3. the act or fact of informing. 4. an office, station, service, or employee whose function is to provide

information to the public: The ticket seller said to ask information for a timetable.

5. (in information theory) an indication of the number of possible choices of messages, expressible as the value of some monotonic function of the number of choices, usually the logarithm to the base 2.

6. Computers. a.important or useful facts obtained as output from a computer by means of processing input data with a program: Using the input data, we have come up with some significant new information. b.data at any stage of processing (input, output, storage, transmission, etc.).*

Information for RCCD Program Review Purposes = Data in a meaningful context organized for decision making

*Dictionary.com Unabridged (v 1.1), Based on the Random House Unabridged Dictionary, © Random House, Inc. 2006.

Information1. knowledge communicated or received concerning a particular fact or

circumstance; news: information concerning a crime. 2. knowledge gained through study, communication, research,

instruction, etc.; factual data: His wealth of general information is amazing.

3. the act or fact of informing. 4. an office, station, service, or employee whose function is to provide

information to the public: The ticket seller said to ask information for a timetable.

5. (in information theory) an indication of the number of possible choices of messages, expressible as the value of some monotonic function of the number of choices, usually the logarithm to the base 2.

6. Computers. a.important or useful facts obtained as output from a computer by means of processing input data with a program: Using the input data, we have come up with some significant new information. b.data at any stage of processing (input, output, storage, transmission, etc.).*

Information for RCCD Program Review Purposes = Data in a meaningful context organized for decision making

*Dictionary.com Unabridged (v 1.1), Based on the Random House Unabridged Dictionary, © Random House, Inc. 2006.

Information at RCCD Comes From

Information at RCCD Comes From

Institutional Reporting (Raj Bajaj) Summaries or organized reports about the current

term Institutional Research (David Torres)

Reports on Employment in the Region Summaries or organized reports about past terms Matriculation reports Reports on Student Engagement and/or satisfaction

Institutional Effectiveness (Kristina Kauffman) Reports to the State, Federal Government and

Accrediting Agencies Reports such as the McIntyre Environmental Scans

Discipline Program Review Documents Information used to “lobby” administration

Institutional Reporting (Raj Bajaj) Summaries or organized reports about the current

term Institutional Research (David Torres)

Reports on Employment in the Region Summaries or organized reports about past terms Matriculation reports Reports on Student Engagement and/or satisfaction

Institutional Effectiveness (Kristina Kauffman) Reports to the State, Federal Government and

Accrediting Agencies Reports such as the McIntyre Environmental Scans

Discipline Program Review Documents Information used to “lobby” administration

AnalysisAnalysis

Analysis1. a method of studying the nature of something or of

determining its essential features and their relations: the grammatical analysis of a sentence.

2. a presentation, usually in writing, of the results of this process: The paper published an analysis of the political situation.

3. a philosophical method of exhibiting complex concepts or propositions as compounds or functions of more basic ones.*

Analysis for RCCD Program Review Purposes = Projections or qualitative observations often resulting in recommendations aligned with the college mission, community need or transfer requirements

*Dictionary.com Unabridged (v 1.1), Based on the Random House Unabridged Dictionary, © Random House, Inc. 2006.

Analysis1. a method of studying the nature of something or of

determining its essential features and their relations: the grammatical analysis of a sentence.

2. a presentation, usually in writing, of the results of this process: The paper published an analysis of the political situation.

3. a philosophical method of exhibiting complex concepts or propositions as compounds or functions of more basic ones.*

Analysis for RCCD Program Review Purposes = Projections or qualitative observations often resulting in recommendations aligned with the college mission, community need or transfer requirements

*Dictionary.com Unabridged (v 1.1), Based on the Random House Unabridged Dictionary, © Random House, Inc. 2006.

Analysis in RCCD program review

documents comes from:

Analysis in RCCD program review

documents comes from: Institutional Research (David Torres)

Employment projections Institutional Effectiveness (Kristina Kauffman)

Scenarios such as those found in the McIntyre Report Statewide and local economic and demographic projections Analysis of trends in Education law and politics Analysis of trends in Accreditation requirements

Units undergoing Program Review (you) Each unit must determine for itself which data, information and

the analysis from other respected sources is relevant for its review.

Ultimately the unit must prepare its own analysis of its unit and its future as a capstone to data, information and analysis provided by others.

Units are not expected to generate their own data or information, but they are expected to be able to identify additional data or information needs and advise Institutional Effectiveness and Institutional Research of those needs. As resources permit units will be provided with data and information or Institutional Effectiveness and Institutional Research will collaborate with units in collection of data.

Institutional Research (David Torres) Employment projections

Institutional Effectiveness (Kristina Kauffman) Scenarios such as those found in the McIntyre Report Statewide and local economic and demographic projections Analysis of trends in Education law and politics Analysis of trends in Accreditation requirements

Units undergoing Program Review (you) Each unit must determine for itself which data, information and

the analysis from other respected sources is relevant for its review.

Ultimately the unit must prepare its own analysis of its unit and its future as a capstone to data, information and analysis provided by others.

Units are not expected to generate their own data or information, but they are expected to be able to identify additional data or information needs and advise Institutional Effectiveness and Institutional Research of those needs. As resources permit units will be provided with data and information or Institutional Effectiveness and Institutional Research will collaborate with units in collection of data.

External Documents Containing Important Considerations Units May

Wish to Include in Their Analysis

External Documents Containing Important Considerations Units May

Wish to Include in Their Analysis

2025 Report: http://www.ppic.org/main/ca2025.asp The University and the Creative

Economy: http://creativeclass.com/rfcgdb/articles/University_andthe_Creative_Economy.pdf

Report of the Task Force on General

Education at Harvard: http://www.fas.harvard.edu/~secfas/General_Education_Final_Report.pdf

Basic Skills Report: http://www.cccbsi.org/Websites/basicskills/Files/Content/83979/http://www.cccbsi.org/Websites/basicskills/Files/Content/83979/

Lit_Review_Student_Success.pdfLit_Review_Student_Success.pdf

2025 Report: http://www.ppic.org/main/ca2025.asp The University and the Creative

Economy: http://creativeclass.com/rfcgdb/articles/University_andthe_Creative_Economy.pdf

Report of the Task Force on General

Education at Harvard: http://www.fas.harvard.edu/~secfas/General_Education_Final_Report.pdf

Basic Skills Report: http://www.cccbsi.org/Websites/basicskills/Files/Content/83979/http://www.cccbsi.org/Websites/basicskills/Files/Content/83979/

Lit_Review_Student_Success.pdfLit_Review_Student_Success.pdf

Key Points in 2025 ReportKey Points in 2025 Report

Dramatic change in labor marketMismatch between skills needed and

skills possessed by the populationDemand for high skilled workers will

require foreign immigration, unless educational changes occur in current residents

By 2025 = 2 of every 5 jobs 41% will require a college degree (now less than 1/3)

Dramatic change in labor marketMismatch between skills needed and

skills possessed by the populationDemand for high skilled workers will

require foreign immigration, unless educational changes occur in current residents

By 2025 = 2 of every 5 jobs 41% will require a college degree (now less than 1/3)

Key Points in The University and the Creative Economy

Key Points in The University and the Creative Economy

Increased need to transfer research to industry

3 T’s Technology: Universities are on the cutting

edge Talent: Drawn to the University and the

surrounding community Tolerance: culture of meritocracy with

openness to difference and eccentricity Riverside ranks 24th in nation in licensing

income per faculty and university generated spin off companies Are we tapping into this potential for our

students?

Increased need to transfer research to industry

3 T’s Technology: Universities are on the cutting

edge Talent: Drawn to the University and the

surrounding community Tolerance: culture of meritocracy with

openness to difference and eccentricity Riverside ranks 24th in nation in licensing

income per faculty and university generated spin off companies Are we tapping into this potential for our

students?

Key Points in Report of Task Force on General Education at

Harvard

Key Points in Report of Task Force on General Education at

Harvard “ambition of the program of ge… is to enable

undergraduates to put all the learning… outside as well as inside the classroom, in the context of the people they will be and the lives they will lead after college.”

Harvard education is liberal education - spirit of free inquiry undertaken without concern for topical relevance or vocational utility Heightens students’ awareness of the human and

natural worlds Results in student who are more

reflective about their beliefs and choices self-conscious and critical of their presuppositions and

motivations, creative problem solving, perceptive of the world around them

Ultimately able to inform themselves about the issues that arise in their lives, personally, professionally, and socially.

“ambition of the program of ge… is to enable undergraduates to put all the learning… outside as well as inside the classroom, in the context of the people they will be and the lives they will lead after college.”

Harvard education is liberal education - spirit of free inquiry undertaken without concern for topical relevance or vocational utility Heightens students’ awareness of the human and

natural worlds Results in student who are more

reflective about their beliefs and choices self-conscious and critical of their presuppositions and

motivations, creative problem solving, perceptive of the world around them

Ultimately able to inform themselves about the issues that arise in their lives, personally, professionally, and socially.

Harvard, con’tHarvard, con’t

GE Goals to prepare students: for civic engagement to understand themselves as products of –

and participants in – traditions of art, ideas and values

to respond critically and constructively to change

Understand the ethical dimensions of what they say and do

GE Goals to prepare students: for civic engagement to understand themselves as products of –

and participants in – traditions of art, ideas and values

to respond critically and constructively to change

Understand the ethical dimensions of what they say and do

Harvard, con’tHarvard, con’t

Courses in Categories Aesthetic and Interpretive Understanding Culture and Belief Empirical Reasoning Ethical Reasoning Science of Living Systems Science of the Physical Universe Societies of the World The United States in the World

Courses in Categories Aesthetic and Interpretive Understanding Culture and Belief Empirical Reasoning Ethical Reasoning Science of Living Systems Science of the Physical Universe Societies of the World The United States in the World

Harvard, con’tHarvard, con’t

PedagogyStrive for interactiveBetter retention of learning through

concrete exercisesaccomplishment of specific tasks creation of actual objects out-of-classroom experiences

Activity-Based Learning Initiative

PedagogyStrive for interactiveBetter retention of learning through

concrete exercisesaccomplishment of specific tasks creation of actual objects out-of-classroom experiences

Activity-Based Learning Initiative

Key Points in Basic Skills Report

Key Points in Basic Skills Report

Review of literature and effective practices

Self assessment tool for collegesCost revenue model for

development education

Review of literature and effective practices

Self assessment tool for collegesCost revenue model for

development education

Basic Skills, con’tSelected Examples of Effective

Practices

Basic Skills, con’tSelected Examples of Effective

Practices Organizational and Administrative Practices

A.1 Developmental education is a clearly stated institutional priority.

A.3 The developmental education program is centralized or highly coordinated.

A.4 Institutional policies facilitate student completion of necessary developmental coursework as early as possible in the educational sequence.

A.5 A comprehensive system of support services exists, characterized by a high degree of integration among academic and student support services.

Program Components B.1 Orientation, assessment, and placement are

mandatory for all new students. B.3 Counseling support provided is substantial,

accessible, and integrated

Organizational and Administrative Practices A.1 Developmental education is a clearly stated

institutional priority. A.3 The developmental education program is

centralized or highly coordinated. A.4 Institutional policies facilitate student

completion of necessary developmental coursework as early as possible in the educational sequence.

A.5 A comprehensive system of support services exists, characterized by a high degree of integration among academic and student support services.

Program Components B.1 Orientation, assessment, and placement are

mandatory for all new students. B.3 Counseling support provided is substantial,

accessible, and integrated

Basic Skills (con’t)Basic Skills (con’t)

Staff Development C.2 The faculty play a primary role in needs assessment,

planning, and implementation of staff development programs and activities

C.4 Staff development opportunities are flexible, varied, and responsive

C.5 Faculty development is clearly connected to intrinsic and extrinsic faculty reward structures.

Instructional Practices D.1 Sound principles of learning theory are applied in

the design/delivery of courses in the developmental program.

D.3 The developmental education program addresses holistic development of all aspects of the student - social and emotional development as well as cognitive growth.

D.7 Programs align entry/exit skills among levels and link course content to college-level performance requirements.

Staff Development C.2 The faculty play a primary role in needs assessment,

planning, and implementation of staff development programs and activities

C.4 Staff development opportunities are flexible, varied, and responsive

C.5 Faculty development is clearly connected to intrinsic and extrinsic faculty reward structures.

Instructional Practices D.1 Sound principles of learning theory are applied in

the design/delivery of courses in the developmental program.

D.3 The developmental education program addresses holistic development of all aspects of the student - social and emotional development as well as cognitive growth.

D.7 Programs align entry/exit skills among levels and link course content to college-level performance requirements.

Internal Documents Containing Important

Analysis

Internal Documents Containing Important

Analysis CCSEQ:

http://www.rccdfaculty.net/pages/Evidence/Insitutionalhttp://www.rccdfaculty.net/pages/Evidence/Insitutional%20Effectiveness/CCSEQ%20Report%20compiled.doc %20Effectiveness/CCSEQ%20Report%20compiled.doc

McIntyre Reports: External Scan 2007:

http://www.rcc.edu/administration/academicaffairs/effectivhttp://www.rcc.edu/administration/academicaffairs/effectiveness/sp_docs/ScanExtCondsRCCDFinal.pdfeness/sp_docs/ScanExtCondsRCCDFinal.pdf

Internal Scan 2007: http://www.rccdfaculty.net/pages/Evidence/Insitutionalhttp://www.rccdfaculty.net/pages/Evidence/Insitutional%20Effectiveness/IntScan.pdf%20Effectiveness/IntScan.pdf

Scenarios 2007: http://www.rccdfaculty.net/pages/Evidence/Insitutionalhttp://www.rccdfaculty.net/pages/Evidence/Insitutional%20Effectiveness/Scenarios.pdf %20Effectiveness/Scenarios.pdf

Educational Master Plans Moreno Valley:

http://morenovalley.maasco.com/clients_morenovalley.asphttp://morenovalley.maasco.com/clients_morenovalley.asp Norco:

http://www.rcc.edu/norco/files/NorcoCollegeEMP.pdfhttp://www.rcc.edu/norco/files/NorcoCollegeEMP.pdf Riverside: http://www.maasco.com/clients_riverside.asphttp://www.maasco.com/clients_riverside.asp

CCSEQ: http://www.rccdfaculty.net/pages/Evidence/Insitutionalhttp://www.rccdfaculty.net/pages/Evidence/Insitutional%20Effectiveness/CCSEQ%20Report%20compiled.doc %20Effectiveness/CCSEQ%20Report%20compiled.doc

McIntyre Reports: External Scan 2007:

http://www.rcc.edu/administration/academicaffairs/effectivhttp://www.rcc.edu/administration/academicaffairs/effectiveness/sp_docs/ScanExtCondsRCCDFinal.pdfeness/sp_docs/ScanExtCondsRCCDFinal.pdf

Internal Scan 2007: http://www.rccdfaculty.net/pages/Evidence/Insitutionalhttp://www.rccdfaculty.net/pages/Evidence/Insitutional%20Effectiveness/IntScan.pdf%20Effectiveness/IntScan.pdf

Scenarios 2007: http://www.rccdfaculty.net/pages/Evidence/Insitutionalhttp://www.rccdfaculty.net/pages/Evidence/Insitutional%20Effectiveness/Scenarios.pdf %20Effectiveness/Scenarios.pdf

Educational Master Plans Moreno Valley:

http://morenovalley.maasco.com/clients_morenovalley.asphttp://morenovalley.maasco.com/clients_morenovalley.asp Norco:

http://www.rcc.edu/norco/files/NorcoCollegeEMP.pdfhttp://www.rcc.edu/norco/files/NorcoCollegeEMP.pdf Riverside: http://www.maasco.com/clients_riverside.asphttp://www.maasco.com/clients_riverside.asp

Key Points in CCSEQKey Points in CCSEQCCSEQ Gains, District, 2004 and 2006

0 0.5 1 1.5 2 2.5 3 3.5 4

Understanding own abilities & interests

Developing ability to learn on own

Writing clearly and effectively

Developing ability to work with others in new settings

Clarifying own values

Understanding/getting along with others

Developing clearer career goals

Learning about different fields of knowledge

Becoming aware of different philosophies, etc.

*Putting ideas together

Presenting ideas effectively in speaking

Acquiring ability to use computers to access information

Acquiring ability to use computers for papers, etc.

Acquiring skills for specific job

Gaining info about career opportunities

Seeing importance of history

*Understanding mathematical concepts

Developing understanding of literature

Developing good health habits

*Understanding role of science & technology

Learning about other parts of the world

Interpreting information from charts & graphs

*Understanding art, music, theater

Interest in political and economic events

Speaking another language

2006

2004

CCSEQ Gains, Factors, District, 2004 and 2006

0

0.5

1

1.5

2

2.5

3

3.5

4

Career Preparation *Arts and Communication Computers Personal and SocialDevelopment

*Mathematics, Science,and Integrated Technology

Perspectives of the World

2004

2006

CCSEQ Quality of Effort, District, 2004 and 2006

0 0.5 1 1.5 2 2.5 3 3.5 4

Writing Activities

Course Activities

* Computer Technology

* Counseling/Career Planning

Student Aquaintances

* Faculty

Career/Occupational Skills

Library Activities

* Science Activities

* Art/Music/Theater

Clubs/Organizations

Athletic Activities

2006

2004

Key Points in External Scan

Key Points in External Scan

Robust area growth 4.1% annuallyDiverse service area Increasing number of HS

graduatesEnvironmental trends (water,

energy, etc.)Transportation costs highTechnology impacting learning

Robust area growth 4.1% annuallyDiverse service area Increasing number of HS

graduatesEnvironmental trends (water,

energy, etc.)Transportation costs highTechnology impacting learning

Key Points in External Scan (con’t)

Job growth in region

Key Points in External Scan (con’t)

Job growth in region Chart K. ANNUAL INDUSTRY JOB OPENINGS,

2004-14 IN RIV,SB,ONT MSA

0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500

Natural Res.& Mining

Construction

Durable Manufacture

Nondurable Manufac.

Wholesale Trade

Retail Trade

Utilities

Logistics (Tr&Wh)

Information

Finance and Ins.

Real Estate,Rnt/Lse

Prof, Sci, & Tech

Firm Management

Business Support

Private Education

State Education

Local Education

Health & Social

Arts/Entertainment

Hospitality & Food

Other Services

Federal Gov

State Gov

Local Gov

External Scan (con’t)Recent Trends in Education

External Scan (con’t)Recent Trends in Education

Shift from teaching to learning paradigms

Longer-term education (with work) balanced with shorter-term job training

Imparting knowledge and meaning, rather than just data and information

Active, rather than passive, learningCooperative or collaborative, rather

than competitive, learning approaches.

Shift from teaching to learning paradigms

Longer-term education (with work) balanced with shorter-term job training

Imparting knowledge and meaning, rather than just data and information

Active, rather than passive, learningCooperative or collaborative, rather

than competitive, learning approaches.

Key Points in the Internal Scan

Key Points in the Internal Scan

Assessment of Fundamental Skills and needed preparation for success

Transfer rates and success Retention patterns Trends in occupational programs

compared to job openings Opportunities for Economic

Development programs Market penetration and marketing

concerns Assessment of district financial situation

Assessment of Fundamental Skills and needed preparation for success

Transfer rates and success Retention patterns Trends in occupational programs

compared to job openings Opportunities for Economic

Development programs Market penetration and marketing

concerns Assessment of district financial situation

Key Points in ScenariosKey Points in Scenarios

Potential enrollment and market penetration Relatively low market penetration now Students are sensitive to changes in the real

cost of enrollment Enrollments are positively related to area

unemployment (unless there is a fee hike) Enrollments are positively impacted by RCCD budget

increases Enrollments are driven by population increases Changes in state funding significantly impacts RCCD Enrollments are negatively impacted to changes in

enrollment fees at Cal State Norco and MV positively impact market penetration

and enrollment

Potential enrollment and market penetration Relatively low market penetration now Students are sensitive to changes in the real

cost of enrollment Enrollments are positively related to area

unemployment (unless there is a fee hike) Enrollments are positively impacted by RCCD budget

increases Enrollments are driven by population increases Changes in state funding significantly impacts RCCD Enrollments are negatively impacted to changes in

enrollment fees at Cal State Norco and MV positively impact market penetration

and enrollment

Scenarios (con’t)Scenarios (con’t)

RCCD Market Penetration: Fall Enrollment per 1,000 15+ Area Population

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

1974

1976

1978

1980

1982

1984

1986

1988

1990

1992

1994

1996

1998

2000

2002

2004

2006

2008

2010

2012

2014

2016

2018

2020

Actual

Estimate

Chart K. Model Estimate 1974-2006; Forecast C: 2007-2021

Scenarios (con’t)Scenarios (con’t)

Chart L. Model Forecasts of RCCD Fall Enrollment,Scenarios A, B, and C, and COCCC, 2007-2021

0

10,000

20,000

30,000

40,000

50,000

60,000

70,000

80,000

Actual

A

B

C

COCCC

Use Data, Information and Analysis To…

Use Data, Information and Analysis To…

Understand where your discipline and its students are now

Tie the future of your discipline and its students to needs/forces outside RCCD

Anticipate the needs of your disciplineLobby effectively for the resources to

meet those needsParticipate in prioritization of

resources

Understand where your discipline and its students are now

Tie the future of your discipline and its students to needs/forces outside RCCD

Anticipate the needs of your disciplineLobby effectively for the resources to

meet those needsParticipate in prioritization of

resources