Assessment Techniques for Curricular Improvement Roxanne Canosa, Rajendra K. Raj Department of...
-
Upload
mae-leonard -
Category
Documents
-
view
212 -
download
0
Transcript of Assessment Techniques for Curricular Improvement Roxanne Canosa, Rajendra K. Raj Department of...
Assessment Techniques for Curricular Improvement
Roxanne Canosa, Rajendra K. Raj
Department of Computer Science
Rochester Institute of Technology
Overview
What is Assessment?– Analytic vs. Holistic Approaches– Assessment == Grading?
Terminology– Assessment vs. Accreditation– Outcomes vs. Objectives
Performance Criteria– Direct vs. Indirect
Evaluation and Continuous Improvement
What is Assessment?
“Assessment is one or more processes that identify, collect, and prepare data to evaluate the achievement of program outcomes and educational objectives”– 2006-2007 Criteria for Accrediting
Computing Programs – Appendix A (Proposed Changes)
– From Section II.D.1 of the ABET Accreditation Policy and Procedure Manual
Analytic vs. Holistic Approaches
Analytic approach– All students/courses analyzed to diagnose areas
in need of improvement
Holistic approach – Focus on overall performance of the program
• Input from employers, alumni, advisory board
Develop efficient and effective processes– “Lean, mean assessment machine”– Don’t commit “random acts of assessment”
• Gloria Rogers
What is Your Assessment Goal?
Assessing all students or specific groups of students?
Assessing students, department, or program?
Assessing for short-term improvement or long-term effect?
Assessing for formative or summative purposes?
Grading vs. Assessing
Grading– Measures extent to
which a student meets faculty requirements and expectations for a course
• Can grades infer student’s achievement of an outcome?
– Factors• Student knowledge• Work ethic• Faculty variance in
course content, grading components, beliefs, bias, …
Assessing– Measures extent to
which a student achieves each course (program) outcome
• Can we leverage grading components for assessment?
– Use rubrics, which are pre-announced performance criteria
Assessment vs. Accreditation
Institutional accreditation through Middle States, SACS, etc. are increasingly requiring direct assessment of program objectives and outcomes
Jargon may be different, but the essential ideas are the same
Terminology (Jargon)
Standards, rubrics, metrics
Statements to measure performance on outcomes and backed by evidence
Performance criteria
Goals, outcomes,
standardsDescribe expected accomplishments of graduates 3-5 years after graduation
Objectives
Educational strategies
Mapping curriculum (coursework, internships, etc.) to outcomes
Educational practices
AssessmentProcess to review results of data collection and analysis to determine value of findings and future action(s)
Evaluation
EvaluationProcesses to identify, collect, analyze & report data to evaluate achievement
Assessment
Objectives, goals, standards
Describe what students are expected to know & be able to do by graduation
Outcomes
Other TermsDefinitionTerm
From ABET perspective
Terminology Lessons
Use terminology for your situation– Sometimes dictated by institutional
accreditation (SACS, Middle States)– Sometimes dictated by program
accreditation (ABET)– Keep a glossary of terms handy for any
external evaluatorsStick to your terminology
– Terms are not fungible without causing too much grief
Proposed Changes toABET Criteria for Computing
Old criteria– Intents and Standards
New criteria (2008-2009 cycle)– General – Program Specific
New ABET Criteria
8 General Criteria– Students– Program Educational
Objectives– Program Outcomes
(a) through (i)– Assessment and
Evaluation– Curriculum– Faculty– Facilities– Support
CS Program Specific Criteria– Outcomes and
Assessment (a) and (b)
– Faculty Qualifications
– Curriculum (a), (b), and (c)
IT/IS Program Specific Criteria
Program Audit Concern
Concern– “A criterion is currently satisfied; however,
potential exists for this situation to change in the near future such that the criterion may not be satisfied. Positive action is required to ensure full compliance with the Criteria.”
Program Audit Weakness
Weakness– “A criterion is currently satisfied but lacks
strength of compliance that assures that the quality of the program will not be compromised prior to the next general review. Remedial action is required to strengthen compliance with the Criteria.”
Program Audit Deficiency
Deficiency– “A criterion is not satisfied. Therefore, the
program is not in compliance with the Criteria and immediate action is required.”
Program Objectives
“Program educational objectives are broad statements that describe the career and professional accomplishments that the program is preparing graduates to achieve.”– Long-term goals– Should be distinct to your program– Should be publicly available– Must be measurable!
Program Outcomes
“Program outcomes are narrower statements that describe what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire in their matriculation through the program.”– Should be publicly available– Must be measurable!
- Gloria Rogers
Objectives vs. Outcomes
Example objective:– Graduates will exhibit effective
communication skillsExample outcomes:
– By the time of graduation, students will:• demonstrate effective written communication
skills• demonstrate effective oral communication skills
Performance Criteria
Define and describe progression toward meeting important components of work being completed, critiqued, or assessed– Student provides adequate detail to support
his/her solution/argument– Student uses language and appropriate word
choice for the audience– Student work demonstrates an organizational
pattern that is logical and conveys completeness– Student uses the rules of standard English
Provide solid evidence of progression
What is Solid Evidence?
Direct Evidence– Easier to measure– Familiar to most faculty - exam or project
grades, presentation skills, etc.Indirect Evidence
– Difficult to measure– Attitudes or perceptions– For example, a desired outcome of a
course may include “improving students’ appreciation of team work”
Direct vs. Indirect Assessment
The assessment process should include both indirect and direct measurement techniques
A variety of sources should be used– Employers, students, alumni, etc.– Converging evidence from multiple sources
can reduce the effect of any inherent bias in the data
Direct Assessment
Direct examination or observation of student knowledge or skills using stated, measurable outcomes
Faculty typically assess student learning throughout a course using exams/quizzes, demonstrations, and reports– Sample what students know or can do – Provide evidence of student learning
Direct Assessment of PEOs
Employment statisticsPromotions and career advancement of
graduatesJob titles, advanced degrees earned,
additional course work taken after graduation, etc.– PEOs must be assessed separately from
POs
Direct Assessment of POs
Common final examsLocally developed exit examsStandardized regional or national exit
examsExternal examinerCo-op reports from employersPortfolios of student work
Indirect Assessment
Indirect assessment of student learning ascertains the perceived extent or value of learning experiences – Assess opinions or thoughts about student
knowledge or skills– Provides information about student
perception of their learning and how this learning is valued by different constituencies
Indirect Assessment Measures
Exit and other kinds of interviewsArchival dataFocus groupsWritten surveys and questionnaires
• Industrial advisory boards• Employers• Job fair recruiters• Faculty at other schools
Survey of Assessment Methods
Indirect
Standardized Exams
Oral Exams
Portfolios
Written & Other Surveys
External Examiner
Locally Designed
Exams
Method
Direct IndirectDirectMethod
Performance Appraisal
Focus Groups
Archived Records
Behavioral Observations
Simulations
Exit & Other Interviews
Direct and Indirect
Duality of some instruments, e.g., an exit interview– Indirect
• Survey of opinions about the perceived value of the program components
– Direct• If person asking the questions uses it as a way
of assessing student’s skills (e.g., oral communication), then the survey is being used as a direct measure of the achievement of that outcome
Evaluation
“Evaluation is one or more processes for interpreting the data and evidence accumulated through assessment practices. Evaluation determines the extent to which program outcomes or program educational objectives are being achieved, and results in decisions and actions to improve the program.”
Continuous Improvement
Accreditation boards are moving towards outcomes-based assessment of CS, IS, and IT programs– Programs must have an established outcomes-
based assessment plan in place (or at least be making progress in that direction)
– Process must be documented– Process must show continuous improvement (both
quantitatively and qualitatively)
Faculty Responsibility
All faculty must have a commitment to and be directly involved in the evaluation of program educational objectives and program outcomes, as well as the process for continuous improvement of the program
Need for Faculty And Staff Buy-In
What makes most academics tick?– Rewards
• Money?• Fun?• Appreciation?• Recognition?
How to encourage involvement? – We all resent any extra work!
Where to Begin?
Define your Mission StatementDefine your Program Educational
Objectives (PEOs)Define your Program Outcomes (POs)Define Course Outcomes (COs)
– Include specific course outcomes on each course syllabus
Make publicly available
Then What?
Show how course outcomes map to program outcomes
Show how program outcomes map to program educational objectives
Choose measurement tools, both direct and indirect
Collect data
Finally
Present data to faculty in an easily digestible form– Charts, graphs, tables, etc.
Faculty evaluates the data – Are students actually learning the material
that the faculty believe (and claim) they are learning?
Faculty make recommendations for improvement as necessary
The Big PicturePerformance
Criteria
Assess: Collect and Analyze
Evidence
EducationalPractices/Strategies
Stakeholders(students,
alumniemployers faculty, …)
Revise
Assess: Collect and Analyze
Evidence
Evaluate:Interpret Evidence
Take Action
Course Outcomes
Mission Statement
Program Outcomes
ProgramObjectives
The Big Picture
Show relationship between mission statement, objectives, and outcomes
Assess and evaluate objectives and outcomes independently
Map program outcomes to program objectives
Map course outcomes to program outcomes Identify weaknesses and implement focused
improvements in targeted areas
Issues
All assessment methods have their limitations and contain some bias
Meaningful analysis requires both direct and indirect measures from a variety of sources– Students, alumni, faculty, employers, etc.
Multiple assessment methods provides converging evidence of student learning
Assessment Lessons
Cannot do everything at once– Try an approach for first round; learn and refine
Having data isn’t all there is to it!– Easy to generate lots of bad data
One size fits all … NOT!– Programs, courses, instructors all differ
Be ready to compromise– Perfection is neither possible nor desirable
Faculty evaluation and promotion– Do not tie to data generated from assessment
Resources
http://www.cs.rit.edu/~rlc/Assessment/http://www.abet.org/assessment.shtml