ASSESSMENTS

ASSESSMENT BASICS1WelcomeThe attendee will understand assessment basics with a focus on creating learning activities and identifying assessment expectations.

Apply the definition of formative assessment to a specific programApply the definition of summative assessment to a specific programDesign formative learning activities and experiences that are linked to course outcomes to measure student learning and progress (assessment FOR learning)Design summative learning activities and experiences that are linked to course outcomes to measure student learning and progress (assessment OF learning)Create Direct and Indirect data-collection methods utilized in each program used to assess student competencyIdentify the assessment expectation(s)/benchmark(s) as a result of the assessment process in each program area2

3Did you know thatStudents can learn better when their college experiences are not collections of isolated courses and activities but are purposefully designed as coherent, integrated learningexperiences in which courses and out-of-class experiences build on and reinforce one another.Suskie, 2004

www.CartoonStock.com4Assessment of Student LearningHigher Learning Commission defines assessment of student learning:Assessment of student learning is a participatory, interactive process that:provides data/information you need on your students learningengages you and others in analyzing and using this data/information to confirm and improve teaching and learning5produces evidence that students are learning the outcomes you intendedguides you in making educational and institutional improvementsevaluates whether changes made improve/impact student learning and documents the learning and your efforts

http://www.uni.edu/assessment/definitionofassessment.shtmlAssessment of Student Learning6

7Drivers of AssessmentLearning-centered paradigmRequirements for accreditationDiscipline/program accreditationAccountabilitySupport for faculty and students to improve their performance

8Assessment CycleAssessment cycle is an on-going process of:

establishing clear, measurable expected outcomes of student learning or service.(established learning goals/expected learning outcomes)

ensuring that students or service users have sufficient opportunities to achieve those outcomes.(provide learning opportunities/curriculum map)9As well as

analyzing and selecting assessment methods used to monitor the alignment of the curriculum with the student learning outcomes.(assessment methods used: Direct and Indirect)

Assessment Cycle10And

systematically gathering, analyzing, and interpreting evidence to determine how well student learning or service matches our expectations.(assess student learning/expectations/benchmarks)

using the resulting information to understand and improve student learning or service.(use the results/improvement plan)Assessment Cycle11Adapted from: http://manoa.hawaii.edu/assessmentAssessment Cycle12Linking Assessments to Curriculum In linking assessment to curriculum, assessment demonstrates what students know and are able to do, thus, this becomes the ACHIEVED curriculumwhat the assessment data says that students know and are able to do.13

14Quantitative AssessmentsUse structured, predetermined response options that can be summarized into meaningful numbers and analyzed statistically:GPAGradesExam scoresStandardized test scoresStandardized teaching scores

15Qualitative AssessmentsUse flexible, naturalistic methods and are usually analyzed by looking for recurring patterns and themes:Ethnographic studiesExit interviewsParticipant observationsWriting samplesOpen-ended questions on surveys and interviews16Formative AssessmentInformal product or performance designed to provide student with instant feedback for self-monitoring of strengths and weaknesses of personal learning expectations.

Assessment FOR learning17Summative AssessmentFormal product or performance designed to inform student and others about personal achievement regarding learning expectations.

Assessment OF learning18Formative vs SummativeFORMATIVESUMMATIVEImprove teaching and learning (or service and satisfaction)Used while learning is taking placeFocus on feedback and adjustmentDocument learning or service and satisfactionOccurs at the end of a course or service periodFocus on sum or total with little feedback19

20Assessments in 2004PERFORMANCETRADITIONALStudents asked to demonstrate skillsAuthentic assessments real-life tasksField experiencesStudio assignmentsProjectsResearch papers

Blue Book essay questionsOral examinationsControlled and timed exam settingObjective tests21Assessments in 2009CONTEMPORARYTRADITIONALCarefully aligned with goals, the most important information students must learnFocused on thinking and performance skillsDeveloped from research and best practices on teaching and assessmentOften planned and implemented without considering learning goalsOften focused on memorizing knowledge low on Blooms TaxonomyFrequently poor quality tests without consideration of mastery of subject 22

23Assessment at the Program LevelEmbedded course assignmentsCapstone experiencesField experiencesPortfoliosCertification testsCommon Student Learning Outcome Rubric24Direct & Indirect Evidence of Student LearningData-collection methods for assessment purposes typically fall into two categories: direct and indirect. Direct evidence of student learning comes in the form of a student product or performance that can be evaluated. Indirect evidence is the perception, opinion, or attitude of students (or others). Both are important. But, indirect evidence by itself is insufficient. Direct evidence is required. Ideally, a program collects both types.

25Types of EvidenceDIRECTINDIRECTTangible, visible, self-explanatory, compelling and acceptableScores/pass rates on licensure/certification examsPortfolios of student workCapstone experiencesLess convincing indicatorsGradesStudent self-ratingsStudent/Alumni satisfaction with learningHonors, awards, scholarships26

Types of Direct Data-Collection Methods27

Direct Data-Collection Methods28

Types of Indirect Data-Collection Methods29Indirect Data-Collection Methods

30Guidelines for Selecting Assessment MethodsThe evidence you collect depends on the questions you want to answerDoes the program meet or exceed certain standards?How does the program compare to others?Does the program do what is sets out to do?How can the program experience be improved?

Using these assessment questions to guide method selection can help your data collection priorities.

Adapted from Volkwein, J., Program evaluation and assessment: Whats the question? (1996).31Use multiple methods to assess each learning outcome

Include both direct and indirect measures

Include qualitative and quantitative measures

Choose assessment methods that allow you to asses the strengths and weaknesses of the programfinding out what is working well is only one goal of program assessment

Adapted from Umass Amherst, OAPA Handbook, Program-Based Review and Assessment

Guidelines for Selecting Assessment Methods32

33Your Turn

34Student Competency Assessments - Direct MethodsDIRECT METHODSEXAMPLES35Student Competency Assessments - Indirect MethodsINDIRECT METHODSEXAMPLES36

37Questions?

Comments?

Thank you for Attending!38ReminderEvaluation Measures(with Guest Speaker Dr. Henry Oh)

Wednesday, April 204 6 pmThursday, April 218 10 am and 12 2 pm39DIRECT METHODSExamples

Licensure or certificationNursing program students' pass rates on the NCLEX (Nursing) examination.

National exams or standardized testsa) Freshmen and seniors' scores on the Collegiate Learning Assessment (CLA) or Collegiate Assessment of Academic Proficiency (CAAP)b) Senior-level biology students' scores on the GRE Subject Test on Biology.

Local exams (external to courses)Entering students' scores on the SJC Placement Exams.

Embedded testing or quizzesa) Students' pass rates on course tests or final exam. Two questions from final exams are scored by a team of faculty members and results used for program-level assessment.

Embedded assignmentsThe program selects course assignments that can provide information on a student learning outcome. Students complete these assignments as a regular part of the course. The assignments are scored using criteria or a scoring rubric and the scores are used for program-level assessment and not typically used to give students a grade on the assignment.

Grades calibrated to clear student learning outcome(s)Professors give grades based on explicit criteria that are directly related to particular learning outcomes.

PortfoliosA collection of student work such as written assignments, personal reflection, and self assessments. Developmental portfolios typically include work completed early, middle, and late in the students' academic career so growth can be noted. Showcase portfolios typically include students' best work and aim to show the students' highest achievement level.

Pre- post-tests

When used for program assessment, students take the pre-test as part of a required, introductory course. They take the post-test during their senior year, often in a required course or capstone course.

Employer's or internship supervisor's direct evaluations of students' performances

Evaluation or rating of student performance in a work, internship, or service-learning experience by a qualified professional.

Observation of student performing a taskProfessor or an external observer rates each students' classroom discussion participation using an observation checklist.

Culminating project: capstone projects, senior theses, senior exhibits, senior dance performanceStudents produce a piece of work or several pieces that showcase their cumulative experiences in a program. The work(s) are evaluated by a pair of faculty members, a faculty team, or a team comprised of faculty and community members.

Student publications or conference presentationsStudents present their research to an audience outside their program. Faculty and/or external reviewers evaluate student performance.

Description or list of what student learnedStudents are asked to describe or list what they have learned. The descriptions are evaluated by faculty in the program and compared to the intended student learning outcomes.Example: After completing a service learning project, students are asked to describe the three most important things they learned through their participation in the project. Faculty members evaluate the descriptions in terms of how well the service learning project contributed to the program outcomes.

INDIRECT METHODSExample

Student surveysStudents self-report via a questionnaire (online, telephone, or paper) about their ability, attitudes, and/or satisfaction.

End-of-course evaluations or mid-semester course evaluationsStudents report their perceptions about the quality of a course, its instructor, and the classroom environment.

Alumni surveysAlumni report their perceptions via a questionnaire (online, telephone, or paper). E.g., alumni answer questions during a telephone survey about the importance of particular program learning outcomes and whether they are pertinent to their current career or personal life.

Employer surveysPotential employers complete a survey on job skills that graduates should possess. Employers can be asked about the quality of current employees who graduated from SJC.

InterviewsFace-to-face, one-to-one discussions or question/answer session.

Focus group interviewsFace-to-face, one-to-many discussions or question/answer session.

Percent of time or number of hours/minutes spent on various educational experiences in and out of classStudents' self reports or observations made by trained observers on time spent on, for example: co-curricular activities homework classroom active learning activities verses classroom lectures intellectual activities related to a student learning outcome cultural activities related to a student learning outcome

Grades given by professors that are not based on explicit criteria directly related to a learning outcomeGrade point averages or grades of students in a program.

Job placement dataThe percent of students who found employment in a field related to the major/program within one year.

Enrollment in higher degree programsThe number or percent of students who pursued a higher degree in the field.

Maps or inventories of practiceA map or matrix of the required curriculum and instructional practices/signature assignments.

Transcript analysis or course-taking patternsThe actual sequence of courses (instead of the program's desired course sequence for students).

Institutional/Program Research dataInformation such as the following: Registration or course enrollment data Class size data Graduation rates Retention rates Grade point averagesSpecific examples:a) Number of sections in each course. b) Percent of seats filled. c) Number of students who dropped a course after first day of classes.d) Average enrollment in sections by course level.

Pilot ReportsProgram-Specific Student Learning Outcomes Assessment Plan (DRAFT)Program: School:Report submitted by:Approval of the Dean:(CURRICULUM 101) (CURRICULUM MAPPING) (ASSESSMENT BASICS) (EVALUATION)Program Outcomes (Observable Student Learning Outcome)Course(s) in which the competency will be assessedMethod used to assess student competency(Process/Instrument/rubric used to assess performance level)D = DIRECT I = INDIRECT(min. 1 each)Assessment Expectations/Benchmark

How ResultsWill Be Used(to make improvements in courses, instruction, and/or student support activities)