Assessment Update 2007-08 Presented by Darby Kaikkonen Director of Institutional Research.
-
Upload
augustine-stewart -
Category
Documents
-
view
213 -
download
0
Transcript of Assessment Update 2007-08 Presented by Darby Kaikkonen Director of Institutional Research.
Assessment UpdateAssessment Update2007-082007-08
Presented by Darby KaikkonenPresented by Darby Kaikkonen
Director of Institutional ResearchDirector of Institutional Research
BackgroundBackground
July 2007- Received notice that College would July 2007- Received notice that College would have a focused interim visit in Fall 2008 from have a focused interim visit in Fall 2008 from Northwest Commission (accrediting body) on Northwest Commission (accrediting body) on standards pertaining to assessmentstandards pertaining to assessment
December 2007- College adopted new strategic December 2007- College adopted new strategic plan which included strategy: “Implement plan which included strategy: “Implement systematic academic assessment that evaluates systematic academic assessment that evaluates student learning outcomes in major program student learning outcomes in major program areas and in general education.” areas and in general education.”
Progress in Assessment Process:Progress in Assessment Process:2007-082007-08
Creation of assessment plansCreation of assessment plans General education (College Wide Ability) General education (College Wide Ability)
assessment: Critical Thinkingassessment: Critical Thinking Linking assessment, strategic planning, Linking assessment, strategic planning,
and budgetand budget
Assessment PlansAssessment Plans
ProcessProcess
All departments submitted assessment All departments submitted assessment reports AND assessment plans in reports AND assessment plans in December 2007December 2007
ARC reviewed assessment plans and ARC reviewed assessment plans and provided feedback for improvement to provided feedback for improvement to departmentsdepartments
Assessment plans implemented Spring Assessment plans implemented Spring quarter 2008quarter 2008
Critical Thinking AssessmentCritical Thinking Assessment
BackgroundBackground
Each course that includes critical thinking Each course that includes critical thinking as part of a course outline would include as part of a course outline would include an assessment of that ability on their an assessment of that ability on their 2007-09 assessment plan2007-09 assessment plan
Interdisciplinary task force developed a Interdisciplinary task force developed a common rubric to use for measuring common rubric to use for measuring critical thinking in all types of coursescritical thinking in all types of courses
First quarter of data collection designed to First quarter of data collection designed to be formative and set baselinesbe formative and set baselines
Pilot Test and Reliability AnalysisPilot Test and Reliability Analysis
Pilot Test:Pilot Test: Used for editing purposes and ease of Used for editing purposes and ease of
understandingunderstanding Reliability Analysis:Reliability Analysis:
Tested how consistently evaluators measured Tested how consistently evaluators measured the same work the same work Level of agreement measured using Cronbach’s Level of agreement measured using Cronbach’s Alpha reliability statisticAlpha reliability statisticAcceptable level is .700 or above, Critical Acceptable level is .700 or above, Critical thinking rubric scored .893thinking rubric scored .893
Results from Benchmark QuarterResults from Benchmark Quarter
Nearly 2,000 records of student work from Nearly 2,000 records of student work from 29 different disciplines were evaluated 29 different disciplines were evaluated using the rubricusing the rubric
Student Work Samples by Division:Student Work Samples by Division:Division
975 48.8 48.8 48.8
876 43.9 43.9 92.7
85 4.3 4.3 96.9
61 3.1 3.1 100.0
1997 100.0 100.0
Natural and Applied Sciences
Social Science and Business
Humanities/Communication
Applied Technology
Total
Valid
Frequency Percent Valid PercentCumulative
Percent
Overall Mean Scores by DimensionOverall Mean Scores by Dimension
931 1 4 2.65 1.105
971 1 4 2.50 1.028
354 1 4 2.78 .898
994 1 4 2.79 .970
870 1 4 2.59 1.074
229
Creates, integrates, andevaluates ideas, concepts,and/or information
Analyzes and evaluatesproblems and/or solvesproblems using multipleprocesses
Examines attitudes, values,and assumptions andassesses their implications
Integrates experience,reason, and evidence tomake meaningfulconclusion and judgments
Justifies key results andprocedures, explainsassumptions and reasons
Valid N (listwise)
N Minimum Maximum Mean Std. Deviation
Dimension Mean Scores by Course Level Dimension Mean Scores by Course Level CategoriesCategories
Course Level Category Dimension A Dimension B Dimension C Dimension D Dimension E
Overall N N Mean N Mean N Mean N Mean N Mean
Pre College 186 186 2.10 0 * 0 * 0 * 186 2.06
100-120 743 338 2.87 391 2.62 198 2.99 396 2.72 290 2.56
121-140 11 61 2.20 95 2.63 0 * 49 3.27 56 3.00
141-160 102 10 2.50 18 2.89 10 2.70 102 2.55 38 2.63
161-180 146 8 3.50 8 3.38 8 3.63 139 3.24 8 3.38
201-220 467 237 2.73 347 2.19 110 2.35 207 2.57 208 2.80
221-240 58 58 2.71 58 2.69 0 * 0 * 58 2.95
241 and above 185 33 3.42 54 2.83 28 2.79 101 2.93 26 2.92
Key Findings from Baseline Quarter:Key Findings from Baseline Quarter: Dimension C called “Examines attitudes, values, and assumptions Dimension C called “Examines attitudes, values, and assumptions
and assesses their implications in a variety of contexts” was and assesses their implications in a variety of contexts” was evaluated significantly less than each of the other four dimensionsevaluated significantly less than each of the other four dimensions
The overall means scores for each dimension were very similar to The overall means scores for each dimension were very similar to one another, and fell somewhere between a “beginning” and a one another, and fell somewhere between a “beginning” and a “usually or largely” level of demonstration“usually or largely” level of demonstration
There is evidence that students in highest course levels There is evidence that students in highest course levels demonstrate greater critical thinking skills than students in Pre demonstrate greater critical thinking skills than students in Pre college and beginning college level courses; however, not all college and beginning college level courses; however, not all dimensions of critical thinking are significantly positively correlated dimensions of critical thinking are significantly positively correlated with course level. Students who were evaluated at the end of a with course level. Students who were evaluated at the end of a series of progressive courses focusing on critical thinking skills series of progressive courses focusing on critical thinking skills tended to score higher than their counterpartstended to score higher than their counterparts
The scores of each critical thinking dimension are very highly The scores of each critical thinking dimension are very highly correlated with one another, so much so that the scores could be correlated with one another, so much so that the scores could be combined into a single index that represents the overall concept of combined into a single index that represents the overall concept of critical thinking. This means that a student was likely to receive the critical thinking. This means that a student was likely to receive the same or very close to the same score on each dimension of the same or very close to the same score on each dimension of the rubric rubric
See full report on the intranet:See full report on the intranet:“Links” “Institutional Research” “College “Links” “Institutional Research” “College
Wide Ability Assessment: Critical Wide Ability Assessment: Critical Thinking Preliminary Report”Thinking Preliminary Report”
Linking Assessment, Linking Assessment, Strategic Planning, and Strategic Planning, and
BudgetBudget
Institutional Assessment LoopInstitutional Assessment Loop
Conceptual Level
Operational Level
Measurement Level
College Mission and Goals
Philosophy Statements
Student Learning Outcomes
Program Outcomes
Assessment and Results
2007-2010 Strategic Plan2007-2010 Strategic PlanGoal:Goal: South Puget Sound Community College is student-centered. South Puget Sound Community College is student-centered.
Objective:Objective: Offer dynamic and balanced educational programs in a global context. Offer dynamic and balanced educational programs in a global context. Strategy: Support and recognize curricular innovation.Strategy: Support and recognize curricular innovation. Strategy:Strategy: Deliver programs that anticipate and respond to stakeholder needs.Deliver programs that anticipate and respond to stakeholder needs. Strategy:Strategy: Respond to the increasing demand for online and hybrid offerings. Respond to the increasing demand for online and hybrid offerings. Strategy:Strategy: Implement systematic academic assessment that evaluates student learning outcomes in major program Implement systematic academic assessment that evaluates student learning outcomes in major program areas and in general education areas and in general education ObjectiveObjective: Provide comprehensive and effective services to students: Provide comprehensive and effective services to students .. Strategy:Strategy: Develop and implement a one-stop philosophy for services to students.Develop and implement a one-stop philosophy for services to students. Strategy:Strategy: Develop and implement a strategic enrollment management plan. Develop and implement a strategic enrollment management plan. Goal:Goal: South Puget Sound Community College is effective, efficient, and fiscally accountable. South Puget Sound Community College is effective, efficient, and fiscally accountable. Objective:Objective: Develop and employ technology that reflects and supports instructional, internal, and external Develop and employ technology that reflects and supports instructional, internal, and external community needscommunity needs
Strategy:Strategy: Implement and support a learning management system.Implement and support a learning management system. Strategy:Strategy: Focus on maintaining and replacing instructional and institutional equipment to meet the needs of Focus on maintaining and replacing instructional and institutional equipment to meet the needs of tomorrow’s learnerstomorrow’s learners Objective:Objective: Actively create and cultivate connections and partnerships with the community. Actively create and cultivate connections and partnerships with the community. Strategy:Strategy: Address community and business environmental changes through dynamic planning. Address community and business environmental changes through dynamic planning. Objective:Objective: Integrate assessment, planning, and resource allocation at the institutional and program levels. Integrate assessment, planning, and resource allocation at the institutional and program levels. StrategyStrategy: : Implement a systematic instructional program review process that integrates academic assessment and Implement a systematic instructional program review process that integrates academic assessment and planning and influences resource allocation. planning and influences resource allocation. Strategy:Strategy: Systematically assess and improve processes and procedures (institutional effectiveness plan).Systematically assess and improve processes and procedures (institutional effectiveness plan). Strategy:Strategy: Diversify college funding sources. Diversify college funding sources. SStrategytrategy: : Develop and implement a marketing plan. Develop and implement a marketing plan. ObjectiveObjective: Maintain high standards for recruiting, retaining, hiring, and evaluating employees.: Maintain high standards for recruiting, retaining, hiring, and evaluating employees. Strategy:Strategy: Recruit and retain faculty and staff that reflect the college’s diverse population and commitment toRecruit and retain faculty and staff that reflect the college’s diverse population and commitment to excellenceexcellence
Strategy:Strategy: Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a diverse, multi-cultural society. diverse, multi-cultural society.
Strategy:Strategy: Enhance leadership and professional development opportunities.Enhance leadership and professional development opportunities. Strategy:Strategy: Design and implement a staffing model that supports efficient and accessible operations. Design and implement a staffing model that supports efficient and accessible operations. Objective:Objective: Create a sustainable physical environment. Create a sustainable physical environment. Strategy:Strategy: Develop and implement a responsible master plan. Develop and implement a responsible master plan. Strategy:Strategy: Assess current practices and reduce college consumption.Assess current practices and reduce college consumption.
2007-2010 Strategic Plan- Budget Priorities2007-2010 Strategic Plan- Budget PrioritiesGoal:Goal: South Puget Sound Community College is student-centered. South Puget Sound Community College is student-centered.
Objective:Objective: Offer dynamic and balanced educational programs in a global context. Offer dynamic and balanced educational programs in a global context. Strategy: Support and recognize curricular innovation.Strategy: Support and recognize curricular innovation. Strategy:Strategy: Deliver programs that anticipate and respond to stakeholder needs.Deliver programs that anticipate and respond to stakeholder needs. Strategy:Strategy: Respond to the increasing demand for online and hybrid offerings. Respond to the increasing demand for online and hybrid offerings. Strategy:Strategy: Implement systematic academic assessment that evaluates student learning outcomes in major program Implement systematic academic assessment that evaluates student learning outcomes in major program areas and in general education areas and in general education ObjectiveObjective: Provide comprehensive and effective services to students: Provide comprehensive and effective services to students .. Strategy:Strategy: Develop and implement a one-stop philosophy for services to students.Develop and implement a one-stop philosophy for services to students. Strategy:Strategy: Develop and implement a strategic enrollment management plan. Develop and implement a strategic enrollment management plan. Goal:Goal: South Puget Sound Community College is effective, efficient, and fiscally accountable. South Puget Sound Community College is effective, efficient, and fiscally accountable. Objective:Objective: Develop and employ technology that reflects and supports instructional, internal, and external Develop and employ technology that reflects and supports instructional, internal, and external community needscommunity needs
Strategy:Strategy: Implement and support a learning management system.Implement and support a learning management system. Strategy:Strategy: Focus on maintaining and replacing instructional and institutional equipment to meet the needs of Focus on maintaining and replacing instructional and institutional equipment to meet the needs of tomorrow’s learnerstomorrow’s learners Objective:Objective: Actively create and cultivate connections and partnerships with the community. Actively create and cultivate connections and partnerships with the community. Strategy:Strategy: Address community and business environmental changes through dynamic planning. Address community and business environmental changes through dynamic planning. Objective:Objective: Integrate assessment, planning, and resource allocation at the institutional and program levels. Integrate assessment, planning, and resource allocation at the institutional and program levels. StrategyStrategy: : Implement a systematic instructional program review process that integrates academic assessment and Implement a systematic instructional program review process that integrates academic assessment and planning and influences resource allocation. planning and influences resource allocation. Strategy:Strategy: Systematically assess and improve processes and procedures (institutional effectiveness plan).Systematically assess and improve processes and procedures (institutional effectiveness plan). Strategy:Strategy: Diversify college funding sources. Diversify college funding sources. SStrategytrategy: : Develop and implement a marketing plan. Develop and implement a marketing plan. ObjectiveObjective: Maintain high standards for recruiting, retaining, hiring, and evaluating employees.: Maintain high standards for recruiting, retaining, hiring, and evaluating employees. Strategy:Strategy: Recruit and retain faculty and staff that reflect the college’s diverse population and commitment toRecruit and retain faculty and staff that reflect the college’s diverse population and commitment to excellenceexcellence
Strategy:Strategy: Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a Expand and encourage opportunities to learn the skills and knowledge necessary to work and live in a diverse, multi-cultural society. diverse, multi-cultural society.
Strategy:Strategy: Enhance leadership and professional development opportunities.Enhance leadership and professional development opportunities. Strategy:Strategy: Design and implement a staffing model that supports efficient and accessible operations. Design and implement a staffing model that supports efficient and accessible operations. Objective:Objective: Create a sustainable physical environment. Create a sustainable physical environment. Strategy:Strategy: Develop and implement a responsible master plan. Develop and implement a responsible master plan. Strategy:Strategy: Assess current practices and reduce college consumption.Assess current practices and reduce college consumption.
2008-09 Equipment Request2008-09 Equipment Request How does your proposal address budget priorities?
(30 points max)
Develop and implement the strategic enrollment management plan* Identify our enrollment problem and implement plan to address. * Support the enrollment goals identified in the 2007-10 SEM plan. * Improve retention - identify why students leave and create strategies to address. * Develop a research plan and build capacity to obtain and use data.
Design and implement a staffing model that supports efficient and effective operations. * Assess new and vacant positions to align with institutional priorities and needs.
Develop and implement marketing plan. Respond to increasing demand for online and hybrid offerings. Maintain and replace equipment to meet needs of today's learning
*Create a replacement plan for equipment - computers, maintenance equipment, carpeting, instructional equipment, printers, copiers and software.
Implement systematic academic assessment that evaluates student learning outcomes in major program areas and in general education.
Implement a systematic instructional program process that integrates academic assessment, planning and influences resource allocations.
2008-09 Equipment Request2008-09 Equipment Request
How does your proposal address budget priorities?
(30 points max) Clear ties to priorities with specific measurable
outcomes The proposal addresses findings based on
assessment results The proposal supports an activity in a 2007-09
assessment plan
Automotive Department Automotive Department ExampleExample
Outcome from Automotive Outcome from Automotive Assessment PlanAssessment Plan
Outcome 1 Measurable Criteria Measurement ToolTime Frame
A program defined outcome is to have employment ready student and graduates of the Automotive Technology Program demonstrate hands-on competency in the 8 ASE task categories: engine repair, automatic transmission & transaxle, manual drive train & axles, suspension & steering, brakes, electrical & electronics, heating and air conditioning, and engine performance.**This outcome is required annually by NATEF for program accreditation**
Students will achieve the following lab completion scores required by NATEF: 95% of the priority 1 tasks, 80% of the priority 2 tasks, and 50% of the priority 3 tasks, at the completion proficiency level of at least 3 on a 1 to 4 scale
Use a task tracking program, completion data, and weekly feedback reports will be provided to students on completed tasks that meet the measured criteria. Quarterly reports will be kept for comparison with goal criteria
Every AUTO course for:2007-08 academic year and 2008-09 academic year
Automotive Equipment RequestAutomotive Equipment Request
Connecting Goals, Strategic Plan, and Connecting Goals, Strategic Plan, and BudgetBudget
College Goal (Student Centered)College Goal (Student Centered)
Strategic Priority (Academic Assessment)Strategic Priority (Academic Assessment)
Budget Decision (Connect to Assessment Plan)Budget Decision (Connect to Assessment Plan)==
Ability to analyze student learning in order to make Ability to analyze student learning in order to make judgments about effectiveness of curriculumjudgments about effectiveness of curriculum
ACCOMPLISHES GOALACCOMPLISHES GOAL
Next Steps…Next Steps…
Assessment Plan UpdatesAssessment Plan Updates
Logan model assessment report is NOT Logan model assessment report is NOT due this falldue this fall
ARC requests that each department ARC requests that each department complete an assessment update formcomplete an assessment update form
Form will be due to your division dean no Form will be due to your division dean no later than October 31, 2008later than October 31, 2008
Assessment Update FormAssessment Update Form2007-09 Assessment Plan:
Mid-Cycle Report - Fall 2008Please complete the following nine items and return them to your division dean no later than October 31, 2008. 1. Name of person submitting report: 2. Department:3. Has your department begun collecting assessment data?
a) Yesb) No
4. If your department has not begun collecting data, in which quarter will the collection begin? a) Fall 2008b) Winter 2009c) Spring 2009
5. Has your department analyzed any assessment data and compiled results?a) Yesb) No
6. If yes, please briefly explain any changes that will be implemented within this assessment cycle as a result of your analysis. If you have already entered your results into a Logan Model grid, feel free to attach it to this report.7. Please briefly describe any anticipated budget needs that your assessments may have identified.8. Please briefly describe any assistance from the Office of Institutional Research that you may need to complete your assessment plan. 9. Any other questions or concerns?
Questions?Questions?