What’s in the works for General Education Assessment?

Dan McCollum, Ph.D.

Associate Director of Academic Assessment

Objectives Define Assessment Explain the Assessment Cycle of Improvement Explore Purposes of Assessment Review Six Assessment Terms Identify Present Elements Describe Present into Future Elements Describe Desired Future Elements Address Ambiguities & Challenges of the Future Ask, what’s with these new G.E. outcomes? Leave time for Q & A

Define Assessment Trudy Banta – “the systematic collection, review

and use of information about educational programs for the purpose of improving student learning and development”

By definition, all assessment is formative assessment

Logically, if it’s not formative, it’s not assessment

Expanded from: Commentary by Trudy Banta, 2008 IUPUI Assessment Institute

Explain the Assessment Cycle of Improvement

PLANNING(Re) Invent Outcomes

(Re) Design Curriculum(Re) Develop Measures

LEARNINGOpportunity to Learn

ASSESSINGUse MeasuresAnalyze Data

Discuss Results

Adapted from: Suskie, L. (2004). Assessing student learning: A common sense guide.

Explore Purposes of Assessment1. Committing to student success

What are students learning? How well? (baseline) Comparison for improvement (cyclical/longitudinal) Comparison to local standards (local criteria for success) Planning curriculum improvements & innovations

2. Meeting accreditation standards of NWCCU How do we know what and how well students are learning? Accreditation – a catalyst for student success

3. Informing outside constituents of program/university performance Accountability – a catalyst for student success Comparison for external standards (e.g., national criteria/ranks)

Adapted in part from: Linda Suskie, Middle States Commission on Higher Education (MSCHE)

Review Six Assessment Terms

1. Direct Measure

2. Indirect Measure

3. (Course) Embedded Measure

4. Common Measure

5. Local Measure

6. External Measure

Adapted in part from Andrea Leskes of AAC&U: http://www.aacu.org/peerreview/pr-sp02/pr-sp02reality.cfm

Identify Present Elements A set of desired Learning Outcomes Course-embedded, local and external,

direct measures A common, local, indirect measure

Identify Present Elements Learning Outcomes

English Composition World Literature Constitutions Mathematics Multicultural International Distribution: (1) Humanities & Fine Arts, (2) Life and

Physical Sciences & Analytic Thinking, (3) Social Sciences

Source: UNLV Undergraduate Catalog Fall 2008 – Spring 2010

Identify Present Elements Course-embedded, local, direct measures

Aligned to student learning objectives within courses, but sometimes aligned to learning objectives within sections of a course (a faux pas)

Aligned with learning outcomes?

Currently being expanded Studying online tracking, analysis, and reporting

systems

Identify Present Elements Course-embedded, external, direct

measure: A Standardized Test Collegiate Assessment of Academic Proficiency

(CAAP) – Writing Essay and Writing Skills Used to assess English Composition course-

sequence objectives aligned to the English Composition General Education Outcome

For more on CAAP: http://www.act.org/caap/

Identify Present ElementsCommon, Local, Indirect measure: Graduating Senior Exit Survey (GSES) includes items to address how well students believe they achieved in:

1. Improving writing skills (English Composition)2. Comparing literature of different cultures and times (World Literature)3. Able to interpret the U.S. and Nevada constitutions in broad contexts

(Constitutions)4. Learn about mathematical/statistical reasoning (Math)5. Analyze multiple, contemporary U.S. Cultures (Multicultural)6. Learn about sociopolitical complexities of interconnected world cultures

(International)7. Appreciate fine arts (Distribution – Fine Arts)8. Understand individual & group behaviors and the impacts of policies on

them (Distribution – Social Sciences)9. Learn about the role of science & engineering in society (Distribution –

Life and Physical Sciences & Analytic Thinking)

Describe Present into Future Elements Sampling plans being developed for senior pilot of

CAAP’s Writing Essay, Critical Thinking, & Math A Baseline Measurement

Sampling plans being developed for first full, freshman implementation of same CAAP tests for VSA

Learning Outcomes being re-invented Will be the focus of future General Education Assessment See: http://generaled.unlv.edu/geac.html

For more on Voluntary System of Accountability (VSA): http://www.voluntarysystem.org/index.cfm?page=homePage

Describe Desired Future Elements Common, Local, Indirect Measure of self-

perceptions of outcomes (entering, graduating, alumni)

Common, External, Indirect Measure of engagement (entering & graduating)

Common, Local, Direct Measure of Outcomes (entering, mid-level, graduating)

Common, External, Direct Measure of Writing, Critical Thinking & Math via CAAP for VSA (entering & graduating)

For more on Voluntary System of Accountability (VSA): http://www.voluntarysystem.org/index.cfm?page=homePage

Describe Desired Future Elements: Full Model with Timing

Entry(1) External Direct (2) Local Direct (3) Local Indirect (4) External Indirect

Mid-Level (Entry for Transfers)

(2) Local Direct

Graduating(1) External Direct (2) Local Direct (3) Local Indirect (4) External Indirect

Alumni(3) Local Indirect

For more on External Direct and Indirect Measures: http://www.voluntarysystem.org/docs/cp/CollegePortraitDetails.pdf

Address Ambiguities and Challenges - CAAP

Pilot Senior Sampling (Spring 2009) ~600 students will be sampled Proportional Quota Sampling

Sample students from each College (college is the Condition) proportionate to the College’s overall part of the University’s graduating population

Motivators for Students Required by course policy Explain use of data for public “portrait” of UNLV Rewards: (1) gift cards (2) graduation fee rebates (3) cash

(4) extra credit (5) public recognition of performance (6) raffle

Address Ambiguities and Challenges - CAAP Full Entry Sample (Fall 2009?)

~ 750 students needed to represent the population Simple Random Sample Proportional Quota Sample Stratified Random Sample No sample Census

Motivators for Students Required part of orientation/registration by university policy Scores on transcripts Rewards

Address Ambiguities and Challenges - CAAP Full Senior Sample (Spring 2010?)

~ 750 students needed to represent population Simple Random Sample Proportional Quota Sample Stratified Random Sample No sample Census

Motivators for Students Required for graduation Scores on transcripts Rewards

Address Ambiguities and Challenges – Desired Full Model

Development of Local Measures by Assessment Measures Development Committees

One committee per outcome To include developing blueprints and/or rubrics To include item writing and/or assignment development To include standard setting for each level

Assessment Days to administer measures to entering, mid-level, and graduating students

Scoring of Local, Direct Measures by Assessment Measures Scoring Committees

One committee per outcome

Address Ambiguities and Challenges – Committees

Gaining faculty participation in measure development and scoring

Having expert measurement people to train and guide the developers and scorers

Address Ambiguities and Challenges: Costs

Faculty stipends and/or releases Proctors Rewards for students CAAP: ~ $39/student for test materials Indirect, external engagement measure

Ask, what’s with these new G. E. outcomes?

What will they be? See: http://generaled.unlv.edu/geac.html

Possibly implemented alongside the curriculum that follows the current General Education outcomes

Quasi-experimentation can be used to compare students’ performance in new and current curriculums

Leave time for Q & A

Questions?