Beyond SLOs and SAOs: The Importance of Rubrics and

Completing the Assessment Cycle CSSO Southern Drive-In Workshop

Menifee Campus, Mt. San Jacinto CollegeFriday, October 31, 2008

Juan Avalos, Ph.D.Dean Student Services and CounselingCosumnes River College

A little bit about myself…

Formal training in Higher Education. 10 years of direct experience with educational

assessment and research (national research institutes, 4-yr private, and 4-yr public).

12 years of management experience in higher education (4-yr private, 4-yr public, and 2-yr public).

Happily married w/ four beautiful kids and a black & white Chihuahua named “Oreo”.

Presentation Deliverables:

ConceptsProgram ModelsSample ProcessesExamplesHands-on ExerciseResources

Presentation Goals:

Everyone walks away with at least one new understanding or piece of information.

Everyone walks away with at least one process or concept that can be implemented or used on your campus next week.

Taking a Quick Pulse…“Everyone, Please Close Your Eyes…”

Thumbs Up, Down, or Side Ways? I am confident in my ability to lead my Student Services

division through the development of a robust “continuous improvement” system on my campus?

My understanding of assessment concepts and theoretical frameworks guides my day-to-day conversations with program leads in development of said system?

I am aware of specific assessment processes and am confident in my ability to lead my managers, faculty, and staff through said processes?

Some Sort of Break….

Main Points

Understanding the core elements of continuous improvement models, having a research conceptual framework, and being able to gauge the relative strength of the evidence you collect is critical to keeping you from being overwhelmed by the complexity of the assessment process.

Main Points…(‘cont)

Attention has been and will continue to be devoted to the articulation of SLOs and SAOs, don’t forget about the importance of rubrics and going through the entire assessment cycle.

…There will be time for further refinement, increased rigor, and broader scope with each iteration.

Core Elements of “Continuous Improvement” Models

Desired outcome(s) Activities designed to address stated outcomes Assessment of progress on said outcomes Dialog & reflection of research/findings Implementation of improvement plan based on

previous four steps (a.k.a. “closing the loop”).…..then back around again.

Various Business and Education “Continuous Improvement”

MethodologiesMore in-depth descriptions and additional information can be found at http://www.isixsigma.com/me/

Balanced Scorecard

Baldrige

Benchmarking

Business Process Reengineering

Deming

Document Control

DMADV

DMAIC

Financial Analysis/Cost Quality

ISO 9000

Lean

Plan, Do, Check, Act—PDCA

Process Management

Project Selection

Simulation

Six Sigma

Total Quality Management

Theory of Inventive Problem Solving

Common Images of “Continuous Improvement” Models…

Common Images of “Continuous Improvement” Models…

Common Images of “Continuous Improvement” Models…

Common Images of “Continuous Improvement” Models…

Common Images of “Continuous Improvement” Models…

Plan: Design or revise business process components to improve resultsDo: Implement the plan and measure its performanceCheck: Assess the measurements and report the results to decision makersAct: Decide on changes needed to improve the process

Common Images of “Continuous Improvement” Models…Baldridge Framework Compared to ISO

BEWARE OF….

NOISELots of terms, concepts, models, tools, vendors, and phrases tied to this process that can be tremendously overwhelming, be counter productive, and turn into NOISE.

One Example: Over 200 terms present in Instructional Assessment Resources website.

//www.utexas.edu/academic/diia/assessment/iar/glossary.php

Resource from 2006 CSSO SLO Conference…

Resource from 2006 CSSO SLO Conference…

Resource from 2006 CSSO SLO Conference…

One of our challenges…

Overcoming a “Transactions” FrameworkDo you teach?

If yes, you have SLOs.

If no, you are likely to only have SAOs.

One of our challenges… (‘cont)

We are probably “ok” with Service Area Outcomes since Student Service areas are traditionally “used to” thinking about operational efficiencies, service to students, student satisfaction w/ services, etc.

…the quality of service we provide and it’s impact on the student (i.e. student satisfaction, responsiveness of service, indirect impact on other student success measures, etc.).

One of our challenges… (‘cont)

Student Learning Outcomes will be more difficult as Student Service areas have not traditionally viewed themselves as contributing to Student Learning...viewed themselves primarily as administrative support units.

The necessary paradigm shift from “transactions” to “student learning” will require us to view what we do from a “do I teach?” lens. If “yes”…What do I teach? Where is my “classroom”? What are my “lesson plans”? How do I know students are learning what I’ve intended for them to learn? etc.

The Cosumnes River College Example…..

Walking the Walk at CRC

Prior to 2004 at CRC

Limited opportunity for Student Services faculty and staff to come together and dialog about organizational success.

Student Services participation in campus Program Review and Unit Planning processes were isolated and misaligned.

Noel-Levtiz SSI findings pointing to potential performance gaps, in key Student Services, over an eight-year span, viewed as “inaccurate” due to perceived concerns over rigor of survey instrument and proctoring process.

Focus of many departmental meetings on “updates” and reasons why improvements cannot be made without additional people and/or funding.

Walking the Walk…

Over the Past Two Years at CRC

Increased opportunity for Student Services faculty and staff to come together and dialog on organizational success. Second Student Services Summit scheduled for October 24, 2008.

Dean’s counseling meeting only focuses on issues for dialog and reflection…need for “updates” handled via weekly email. Weekly email also includes a “Leadership Quote for the Week”, which provides an opportunity for reflection on important concepts connected to issues we are experiencing at the time.

Walking the Walk…

Over the Past Two Years at CRC (‘cont)

Student Services participation in campus Program Review more inclusive, but product a bit disjointed. Unit Plan, however, more clear and precise in articulating department-wide issues, identified strategies, and necessary resources, where applicable.

Could have dwelled on the “imperfections” of our Performance Review document, but decided to accepted it as is, take from it as much insight as we could, and focus on production of our Unit Plan. Moving forward paid off.

Walking the Walk

Walking the Walk…

This Year…reaping the benefits of laying our foundation four years ago.

Reduction in resistance to accept that twelve-year trend of Noel-Levtiz SSI findings pointing to potential performance gaps may be “real”. Counselors willing to consider conversation on what actions we can take in response to concerns surfaced by SSI findings.

Counseling faculty engaged in Fall 08 Convocation SLO Dialog Exercise. Willing to “go off task” and entertain initial conversation on SLO competence levels, as a precursor to a broader discussion about appropriate assessment measures.

Walking the Walk…

This Year…(‘cont)

Faculty willing to continue work started during Convocation. They will propose a method for going through the exercise. It may involve smaller groups of faculty initially (…producing draft work to share with Counseling core) and a process that is not as “boring”.

Next series of conversations likely to focus on how we will ensure that we provide opportunities for students to achieve our desired SLOs within the context of our regular counseling sessions (i.e. “what will you do, in sessions with students, to help them learn the skills, abilities, etc. we expect them to learn?”).

What we did at Convocation…

Getting to Work

Getting to Work…

Instructions:

•You will be walking to Bldg. #800, Community Room.

•You will be grouped by candy wrapper.

•Exercise directions and worksheets are at your table.

•I need you back in this Assembly Hall by 11:19 am.

Getting to Work…

Break-Out Session Debrief

Think about the PROCESS….

1.What were your conversations like?

2.What were you able to do with relative ease?

3.What did you need to M-u-d-d-l-e Through?

…and what you were feeling as you went through the process.

An Educational Research Conceptual Framework to

Consider…

One Research Framework to Consider

Astin’s I-E-O Framework

INPUTS

ENVIRONMENT

OUTCOME

What you need to know about Evidence…

A Guide to Using Evidence in the Accreditation Process: A Resource to Support Institutions and Evaluation

TeamsWorking Draft (January 2002)

WASC Senior Colleges & UniversitiesPrimary Author: Peter Ewell

What is Evidence?(from WASC Evidence Guide)

Evidence is the substance of what is advanced to support a claim that something is true.

Evidence… Intentional and purposeful. Entails interpretation and reflection. Integrated and holistic. Can be both quantitative and qualitative. Can be either direct or indirect.

What is Evidence? (‘cont)(from WASC Evidence Guide)

Syndromes to avoid when using evidence: Trying to measure everything. Trying to be too precise. Premature closure.

What Constitutes Good Evidence?(from WASC Evidence Guide)

Five principles to using good evidence: 1. Relevant…demonstratively related to the question

being investigated.2. Verifiable…is documentable and replicable.3. Representative…typical, not an isolated case.4. Cumulative…gains credibility as additional sources or

methods are employed.5. Actionable…reflectively analyzed and interpreted in

way to provide guidance for action/improvement.

Check for Understanding…

Are you asking yourself…

How do I ensure that my Student Service operations are effective?

Are you asking yourself… (‘cont)

In addition to working closely with your Student Services faculty and staff in the identification of SAOs and SLOs, attention must be given to the following:

1) Moving your operations through the entire Assessment Cycle. It’s okay not to be perfect the first time around. There will be time for further refinement, increased rigor, and broader scope with each iteration….

Review…

Outcomes

Strategies

Analysis

Improvement

Are you asking yourself… (‘cont)

In addition to moving through the Assessment Cycle:

2) For SLOs particularly, focus immediate attention to articulating levels of competency attainment…since students will be at various skill levels prior to enrolling at our college or coming to us for services. This is a critical step towards accurately and intentionally connecting all elements of the Assessment Cycle.

Review…

Astin’s I-E-O Framework

INPUTS

ENVIRONMENT

OUTCOME

Why Bamboo?

Development of a continuous improvement system has been compared to the growth of

Bamboo……it takes several years to get established (i.e.

difficult to see growth initially), then will dramatically grow from month to month.

Be patient, you may be making progress even if you can’t see it…stay the course.

Some Resources…

Resources…CRC Student Service Areas First Attempt at SLOs

Resources…CRC Student Service Areas First Attempt at SLOs

Resources…CRC Student Service Areas First Attempt at SLOs

Resources…CRC Student Service Areas First Attempt at SLOs

Resources…CRC Student Service Areas First Attempt at SLOs

Resources…

Resources…

Resources…

Resources…

Resources…Assessment Planning (ABET)

http://www.abet.org/assessment.shtml

Resources…Instructional Assessment Resources

http://www.utexas.edu/academic/diia/assessment/iar/

Resources…

Bloom’s Taxonomy

Resources…Bloom’s Taxonomy (con’t)

Knowledge…Ability to memorize and recall facts. Action verbs: Cite, define, identify, label, list, match, name, recognize, reproduce, select, state.

Comprehension…Ability to read course content, understand and interpret important information, and put other’s ideas into own words. Action Verbs: Classify, convert, describe, distinguish between, explain, extend, illustrate, interpret, paraphrase, summarize, translate.

Application…Ability to take new concepts and apply them to another situation. Action Verbs: Apply, compute, construct, demonstrate, discover, modify, operate, produce, relate, show, solve, use.

Analysis…Ability to take new information and break it down into parts to differentiate between them. Action Verbs: Analyze, associate, classify, arrange, diagram, discriminate, distinguish, infer, separate.

Resources…Bloom’s Taxonomy (con’t)

Synthesis…Ability to take various pieces of information and form a whole creating a pattern where one did not previously exist. Action Verbs: Combine, compile, compose, construct, create, design, develop, devise, formulate, integrate, modify, organize, plan, propose, rearrange, reorganize, revise, rewrite.

Evaluation…Ability to look at someone else’s ideas or principles and see the worth of the work and the value of the conclusions. Action Verbs: Appraise, assess, compare, conclude, contrast, criticize, evaluated, judge, justify, support, weigh, rank, grade, recommend, explain.

Taking a Quick Pulse…“Everyone, Please Close Your Eyes…”

Thumbs Up, Down, or Side Ways? I am confident in my ability to lead my Student Services

division through the development of a robust “continuous improvement” system on my campus?

My understanding of assessment concepts and theoretical frameworks guides my day-to-day conversations with program leads in development of said system?

I am aware of specific assessment processes and am confident in my ability to lead my managers, faculty, and staff through said processes?

Open your eyes…

I need a show of hands….

Who feels like they gained at least one new understanding or piece of information this morning?

Who feels like you are walking away with at least one process or concept that you can use on your campus next week?

Don’t Forget…

Thank You!