California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO...
-
Upload
isai-chisley -
Category
Documents
-
view
217 -
download
2
Transcript of California State University First Year Experience Assessment Presenter: Joseph Pica, Ed.D. CEO...
California State UniversityFirst Year Experience
Assessment
Presenter:Joseph Pica, Ed.D.
CEO Educational Benchmarking (EBI)
June 25, 2004
Fundamentals of Structuring and
Implementing Successful First Year Seminars
The statistical analysis for this presentation was based on the Educational
Benchmarking (EBI) First Year Initiative (FYI) Assessment
and conducted by
Randy L. Swing, Ph.D.Co-Director, Policy Center on the First
Year of College &Fellow, The National Resource Center
on The First-Year Experience and Students in Transition
in conjunction with Educational Benchmarking
Value of a Collaborative
Group
• Shared mission and purpose
• Common problems and barriers
• Varied expertise
• Benchmarks for performance
0%5%
10%15%20%25%30%35%
California State UniversityFirst Year Experience Survey Question:What is the MOST important topic you would like to see
addressed at future First Year Experience Meetings?
California State UniversityFirst Year Experience Survey Question:What is the MOST important topic you would like to see
addressed at future First Year Experience Meetings?
Direct Quote:
“The elements that research indicates are the most important
to have as components of an effective first year program.”
Presentation Overview
• The elements that research indicates are the most important to have as components of an effective first year program.
• Principles for leveraging assessment to initiate and sustain improvement
FYS – Goals• Retention 37%• Academic Achievement 37%• Participation in College Life 11%
The Policy Center on the First College Year
• Ease transition to college• Increase “student skills”• Increase persistence rates• Increase graduation rates
First-Year Initiative Survey
FYI• Developed by the Policy Center on the First
College Year and Educational Benchmarking (EBI)
• 62 institutions (limited to 4-year institutions*)• Over 30,000 students• 7- point scale• Learning Outcome Factors• Administered in the last week of fall 2001
*4-year or 2-year regional campuses - a 2-year version is in development.
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/ Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
FYI Assessment and Research
ContextThe following principles
and practices provide the foundation to the
research and the First Year Initiative Assessment
(FYI)
Research/Assessment Context
1.Factors: A more powerful indicator than individual question results.
2. Regression: Allows you to identify predictors of performance
3.Predictor Factors: Factors, if improved, have the greatest impact on improving overall effectiveness
Factors
• Factors (also called “constructs”) are groupings of related questions that share a relationship.
• The basic assumption of factor analysis is that underlying dimensions, or factors, can be used to explain more complex phenomena.
Factor MeansCourse Learning Outcomes mean
Course Improved Knowledge of Campus Services . . . . . 4.72
Course Improved Knowledge of Campus Policies. . . . . 4.58
Course Improved Connections with Peers. . . . . 4.54
Course Improved Connections with Faculty. . . . 4.50
Course Improved Managing Time/Priorities. . . . 4.50
Course Improved Critical Thinking. . . . 4.40
Course Improved Study Strategies. . . .4.19
Course Improved Knowledge of Wellness Issues. . . . 3.90
Course Improved Academic/Cognitive Skills. . . . 3.74
Course Increased Out-of-Class Engagement. . . .3.63 1.0 2.0 3.0 4.0 5.0 6.0 7.0
1/2 3/41/4
Factor Reliability
Once it has been determined that a certain set of questions
do share a relationship and therefore constitute a factor,
there is an additional (and necessary) statistical test to
assess the psychometric soundness of the factor.
• A Cronbach’s Alpha of zero would mean that there is no internal consistency at all, i.e., subjects are likely to respond with anything from 1 to 7 on any of the questions in a factor with no discernable pattern.
• An Alpha of 1 would mean that every subject answered every question comprising the factor consistently (e.g., all subjects answered with all 7’s, or all 1’s). This is a highly unlikely event.
Factor Reliability
FYI Factor Reliabilities Factor Descriptions ReliabilityCourse Improved Study Strategies 0.90Course Improved Academic and Cognitive Skills 0.89Course Improved Critical Thinking 0.91Course Improved Connections with Faculty 0.84Course Improved Connections with Peers 0.90Course Increased Out-of-Class Engagement 0.90Course Improved Knowledge of Campus Policies 0.90Course Improved Knowledge of Academic Services 0.87Course Improved Managing Time and Priorities 0.92Course Improved Knowledge of Wellness 0.90Sense of Belonging and Acceptance 0.90Usefulness of Course Readings 0.90Satisfaction with College/University 0.90Course Included Engaging Pedagogy 0.92Overall Course Effectiveness 0.92
Regression• A correlation establishes the
relationship between two variables.
• Regression analysis, by contrast, allows us to determine the relationship between some dependent variable (e.g. overall effectiveness) and multiple independent variables (e.g., engaging pedagogy, etc.).
• Major predictors identify, in order of descending importance, the factors that have the greatest impact on overall effectiveness.
• While minor predictors are statistically significant, individually and combined they add little to the predictability of overall effectiveness.
Regression
Predictors
Therefore, the return on the investment for
improving the major predictors would far
outweigh the return on improving the minor
predictors.
Horizontal Cross Barset at Mean = 5.50 (75%
satisfaction level)
Vertical Cross Barsets the line between Major and
Minor Predictors of Overall Satisfaction
Top PriorityHigh Impact, Low
Performance
MonitorLow Impact,
Low Performance
MaintainLow Impact,
High Performance
Maintain or
Improve High Impact, High Performance
Impact: Predictors of Overall Satisfaction
Perf
orm
an
ce
4 Quadrants
Priority Matrix
Ex e
cuti
ve S
um
ma r
y
Top PriorityHigh Impact, Low
Performance
MonitorLow Impact,
Low Performance
MaintainLow Impact,
High Performance
Maintain or
Improve High Impact, High Performance
Impact: Predictors of Overall Satisfaction
Perf
orm
an
ce
Priority Matrix
Ex e
cuti
ve S
um
ma r
y
Engaging Pedagogy #1Predictor of Course
Effectiveness
Engaging Pedagogy Factor
• To what degree did the course include:– A variety of teaching methods– Meaningful class discussions– Challenging assignments– Productive use of classroom time– Encouragement to speak in class– Encouragement for students to work
together– Meaningful homework
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/ Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
Four Types of Academic Seminars
1. COLLEGE TRANSITION THEMEorientation/study skills/managing transitions
2. SPECIAL ACADEMIC THEMEinterdisciplinary/problem-focused/selected topic
3. DISCIPLINE BASEDintroduction to major/department/discipline
4. REMEDIAL/STUDY SKILLSstudy skills for a high risk population
Transition-Theme
75%
What is a First-Year Seminar?
A course offered specially for new students to assist with the transition into college.
Four Types of Seminars
Remedial2%
Discipline-based10%
Special Academic
13%
58%
Cal
College Transition Theme courses deal directly with orientation to college and academic skills. This is the University 101 (USC) model.
Special Academic Theme courses deal with a selected topic other than college transition. These are often taught as interdisciplinary seminars where a small group of students and a model learner/teacher use a variety of methods to investigate an important theme.
Discipline based are often an introduction to a major or department. They are based in individual academic departments.
Remedial/Study Skills, was considered, but there were too few cases to draw any meaningful conclusions.
Study StrategiesAcademic Skills
Critical ThinkingFaculty Connections
Peer ConnectionsOut-of-Class
Policies/ProceduresCampus Services
Time/PrioritiesWellness/Spirituality
BelongingCourse SatisfactionEngaging Pedagogy
19.213.327.9
28.234.418.034.139.426.423.058.533.630.5
17.316.031.127.732.112.719.222.5
23.115.5
55.137.036.5
9.79.116.819.717.210.0
35.932.215.310.7
52.821.318.3
Theme Format:Percent of students with mean 5.50 or greater
SpecialTransition Academic Discipline
FYI Finding
Transition-theme and Special Academic-theme courses were about equal on learning outcomes and student satisfaction.
Discipline-theme courses produced lower learning outcomes and student satisfaction.
Remedial courses – too few in the study to draw valid conclusions.
Why?Engaging Pedagogy explains the
difference.
Transition 30.5%Special Academic 36.5%Discipline 18.3%
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/ Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
Credit Hours2001 Pilot Administration
% of institutions (N=62)
40%
6%6%
24%
24%
No credit
4 or more credits
1 credit
2 credits
3 credits
Cal
21%
16%
32%
32%
0%
Study StrategiesAcademic Skills
Critical ThinkingFaculty Connections
Peer ConnectionsOut-of-Class
Policies/ProceduresCampus Services
Time/PrioritiesWellness/Spirituality
BelongingCourse SatisfactionEngaging Pedagogy
Percent of students mean 5.50 or greater 15.9 20.6 19.4
10.9 12.7 17.323.1 26.1 34.224.9 28.3 29.626.1 35.5 37.314.8 18.5 18.633.3 35.3 30.435.9 41.0 35.424.6 27.7 25.218.5 24.0 23.055.3 58.5 60.228.1 34.9 35.424.1 30.1 37.0
Contact Hour(s)1 Hr 2 Hrs 3
Hrs
1 contact hour courses
Orientation to Services2 contact hour courses
Study Strategies Peer Connections Faculty
Connections3 contact hour courses
Academic SkillsCritical Thinking
most effective with learning outcomes associated with a basic orientation to campus services.
perform well at producing effective study strategies, peer connections, and faculty connections
significantly impact academic skills (reading, writing, oral presentation skills) and critical thinking skills
FYI Finding
The number of contact hours should match the
intended goals of a first-year seminar.
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
Required/ Not Required
2001 Pilot Administration% of institutions (N = 62)
11%
35%17%
37%
Cal FYI42% 37% No students in any section required to enroll32% 11% Some students in some sections required to enroll16% 35% All/most students in every section required to enroll10% 17% Mixed formats – no one format constitutes 80%
42%
16%10%
32%
Cal
Study StrategiesAcademic Skills
Critical ThinkingFaculty Connections
Peer ConnectionsOut-of-Class
Policies/ProceduresCampus Services
Time/PrioritiesWellness/Spirituality
BelongingCourse SatisfactionEngaging Pedagogy
Requi
red
Not
Req
uire
d (E
lect
ive
)
Percent of students mean 5.50 or greater 16.1
12.727.824.827.412.925.927.722.616.756.027.929.0
19.514.028.129.438.017.834.442.426.724.058.237.731.9
Required Not Required
Study Strategies 4.18 4.43Academic Skills 3.47 3.76
Critical Thinking 4.36 4.56Faculty Connections 4.35 4.66Peer Connections 4.31 4.84
Out-of-Class 3.36 3.76Policies/Procedures 4.28 4.66
Campus Services 4.35 4.93Time/Priorities 4.27 4.57
Wellness/Spirituality 3.59 4.14Belonging 5.41 5.51
Course Satisfaction 4.29 4.88Engaging Pedagogy 4.43 4.74
Factor Means for Required/not requiredControlled for Contact Hrs, Theme-types, and Grade-formats
Required Not Required
Study Strategies 4.18 4.43Academic Skills 3.51 3.72
Critical Thinking 4.39 4.55Faculty Connections 4.37 4.65Peer Connections 4.34 4.83
Out-of-Class 3.36 3.77Policies/Procedures 4.21 4.72
Campus Services 4.29 4.98Time/Priorities 4.26 4.59
Wellness/Spirituality 3.55 4.18Belonging 5.43 5.51
Course Satisfaction 4.32 4.86Engaging Pedagogy 4.47 4.70
Factor Means for Required/not requiredControlled for student characteristics (gender, race/ethnic, HS Grades, and commuter/residential)
FYI FindingCourses that are NOT REQUIRED
. . . produced greater outcomes than required courses.
WHY?
Engaging Pedagogy
Required Not Required 4.48* 4.71*
* after controlling for differences in grading and contact hours
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/ Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
Grading 2001 Pilot Administration
% of institutions (N = 62)
2%
16%
82%
82% Letter Graded16% Pass/Fail
2% Mixed
Cal
47%
Study StrategiesAcademic Skills
Critical ThinkingFaculty Connections
Peer ConnectionsOut-of-Class
Policies/ProceduresCampus Services
Time/PrioritiesWellness/Spirituality
BelongingCourse SatisfactionEngaging Pedagogy
Graded
Pass
/Fa
il
Mean Scores *controlling for Contact Hrs, Required, UGTAs, & Themes
4.25 4.333.63 3.454.40 4.394.49 4.544.56 4.453.68 3.444.51 4.794.70 4.794.38 4.503.93 3.815.48 5.414.56 4.624.58 4.43
FYI FindingOverall, grading format produces
mixed results, even when controlled for Contact Hours, Theme-types, Required, and
UGTAs.Key Finding:
Graded courses are associated with higher
scores on Engaging Pedagogy
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/ Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
Linked Courses2001 Pilot Administration
% of institutions (N = 62)
11%16%
73%
Few or no sections linked
Some Linked Most Linked
16%43%
42%
Cal
Study StrategiesAcademic Skills
Critical ThinkingFaculty Connections
Peer ConnectionsOut-of-Class
Policies/ProceduresCampus Services
Time/PrioritiesWellness/Spirituality
BelongingCourse SatisfactionEngaging Pedagogy
Link
ed
Not
Li
nked
Mean Scores controlling for Required, Grading, Contact, & Theme
4.46 4.383.78 3.644.62 4.504.60 4.655.02 4.633.63 3.814.55 4.704.75 4.914.60 4.574.02 4.155.55 5.504.89 4.724.76 4.66
FYI FindingLinking the seminar to other
courses produced greater learning outcomes for:
Academic SkillsStudy Skills
Critical ThinkingEngaging Pedagogy
Limitations in this study reduce the capability to make definitive statements about the impact of
linking courses.
Issues Addresses byFYI Findings
Course Theme
Credit/Contact Hours
Required/ Not Required
Letter Graded/Pass-Fail
Linked Course Format
Undergraduate Teaching Assistants
Use of Undergraduate Teaching Assistants
38%
30%
32%
Less than 80% of sections have UTAs
Cal
58%
21%16%
No sections
have UTAs
More than 80%of sections have
UTAs
Study StrategiesAcademic Skills
Critical ThinkingFaculty Connections
Peer ConnectionsOut-of-Class
Policies/ProceduresCampus Services
Time/PrioritiesWellness/Spirituality
BelongingCourse SatisfactionEngaging Pedagogy
UGTA
No
UGTA
Mean Scores controlling for Required, Grading, Contact, & Theme
4.28 4.313.56 3.744.40 4.584.54 4.444.66 4.423.73 3.234.51 4.214.83 4.234.47 4.304.04 3.495.48 5.414.57 4.514.57 4.60
FYI FindingUndergraduate Teaching Assistants are associated with higher mean learning outcomes - except for academic skills, critical thinking skills, and engaging pedagogy.
.... even after controlling for:
Required/NotGrading formatThemeContact Hours
Summary:
Engaging Pedagogy best predicts the learning
outcomes and student satisfaction with the
seminar
Student rating of Engaging Pedagogy:
• No significant difference by gender
• African-Americans, Latino, and Native Americans gave higher ratings than Whites and Asians
• “A” high school students gave lower ratings
Right Reason• Help staff calibrate
performance
• Identify where to focus effort
• Identify where training is needed
• Provide motivation to improve
• Use assessment for Continuous Improvement, not Evaluation
• Don’t punish staff for past performance they are not capable of changing.
• Join with staff to use information to improve performance
Right Reason
Right Method
• Credible
• Confidential
• Comparative
• Comprehensive
• Continuous
Right Reason
Right Info
• reallocate budgets
• focus resources on a limited number of initiatives with the greatest probability of producing improvement
Results should be of research quality, but analyzed and presented to provide decision-makers with the information they need to:
Right Method
Right Reason
Right Info
Right People
• Put the information directly in the hands of the people in the best position to initiate and implement change
• Put information into the hands of the people responsible for results
Right Method
Right Reason
Right Info
Right Time
Right People
• Immediacy of feedback brings relevancy and impact to results
• Provide results at a time staff can evaluate performance, focus on impact factors to create innovative initiatives to improve.
• The shortest assessment cycle leads to the most rapid improvement
Right Method
Right Reason
Right Info
Repeatedly
Right Time
Right People
• Essential to establish expectation in staff they will be able to determine the impact of their efforts to improve through future assessments
• Assessment is a management tool not an isolated event
• Continuous assessment is paramount to continuous improvement
Right Method
Right Reason