A D D E N D U Mto

Assessment Guide

January 25, 2017

ProgramReviewandAssessment1“AtaGlance”Promoting Loma Linda University Success • Preparing for WSCUC 2020

ProgramResponsibilitiesWhat When How Why 1. Program Assessment

Plan• Program Learning

Outcomes (PLOs)• Curriculum Map• Assessment Matrix

If program doesn’t have one yet, develop one now.

If program has one, review annually; update it, if needed.

See the “LLU Assessment Guide2,” pp. 4-11

• To guide, maintain, and strengthenthe program’s curriculum andassessment.

• To prepare the program tocomplete the new annual Inventoryof Educational EffectivenessIndicators (IEEI) survey forWSCUC and the federalgovernment.

2. Program Review (PR)Self-study Report

Required: Due at the end of the LLU program review cycle

See the “LLU Program Review Guide3,” pp. 3-6; 12-18

To evaluate quality, effectiveness, currency, viability, sustainability, alignment with mission, and priorities.

3. PR External ReviewTeam Visit and Report

Required: Due at the end of the LLU program review cycle

See the “LLU Program Review Guide,” pp. 23-29

To provide an outside perspective that provides a constructive, expert analysis of program quality and recommendations for future planning and improvements.

4. PR Action Plan Full plan is the last step of the PR process during the PR cycle.

Annual updates: Due end of October

See the “LLU Program Review Guide,” pp. 8-9, 22 See the Sample Action Plan in the “LLU Program Review Guide,” pp. 30-32

To be derived from the Overview of Proposed Changes of the self-study document and reflects recommendations in the External Review Report. It is the program’s blueprint for planning, implementing, and tracking program development

5. School AdministrativeReview Feedback

First time ever requested - TBA

In development To give programs essential feedback that will help them continue to grow and improve.

6. ILO andProfessional ILOAssessment

Annual updates: Due end of October

See the “LLU Assessment Guide,” pp. 17-18

To ensure that all LLU graduates at every level and discipline have these important skills.

LLUProgramReviewandAssessmentTermsAssessment: Processes that identify, collect, use, and prepare data that can be used to evaluate students’ achievement.

Assessment includes Institutional Learning Outcomes, Program Learning Outcomes, Course Learning Outcomes, and so on. Assessments for Learning Outcomes are usually done with rubrics. Course or assignment grades and tests are not considered to be assessments.

LLU Institutional Learning Outcomes (ILOs): (1) Written Communication, (2) Oral Communication, (3) Quantitative Reasoning, (4) Information Literacy, and (5) Critical Thinking. Critical Thinking is the ILO for this academic year (2017).

1 Assessment based on ILOs and PLOs and is tracked over time. 2 http://home.llu.edu/academics/academic-resources/educational-effectiveness/assessment 3 http://home.llu.edu/academics/academic-resources/educational-effectiveness/program-review

Academic Deans Council January 25, 2017 1

School Assessment Specialist: Each school assigns at least one Assessment Specialist to coordinate school assessment activities. These individuals have evaluation and measurement experience and receive additional training and support from the Office of Educational Effectiveness.

Student Learning Outcomes (SLOs): Knowledge, skill, attitudes, values, etc., that students should be able to demonstrate by the end of the program. This is a large category term for three types of SLOs: (1) institutional (ILOs), (2) program (PLOs), and (3) course (CLOs).

WSCUC-only: Programs that do not have professional or external accreditation. These programs must follow the full LLU program review process.

WSCUC-plus: Programs that have professional, discipline-specific accreditation. These programs follow their professional accreditors’ guidelines for the self-study report, external site review, and action plan. For LLU they will also do ILO/Professional ILO assessment.

ProgramReviewandAssessmentResourcesAll of these resources are listed and linked at myLLU.edu Faculty Page:

Resource Why It Is Needed Assessment Management System (AMS)

Enter annual (1) ILO and Professional ILO assessment analysis reports, and (2) Action Plan Updates

LiveText Conduct ILO/Professional ILO and PLO assessments in LiveText for instant and detailed analytics as a basis for the program’s annual analysis report. These assessments will be kept for the long-term over years to look for strengths, gaps, and trends.

Assessment (Office of Educational Effectiveness - OEE)

This site has everything a program needs to know about program review, ILO/Professional ILO assessments, program review, distance and digital education, institutional research, and more.

Canvas LLU’s learning management system (LMS) for all courses: face-to-face, hybrid, and fully online. Although course assignment rubrics and grades can be kept in Canvas, it is not designed to be an assessment management system.

ProfessionalInstitutionalLearningOutcomes–AlternativeApproach4LLU has new freedom and responsibility to define, assess, and document learning in our own way. The following alternative approach to the LLU’s existing ILOs and assessment tools is one of the first freedoms for programs to meet their unique needs:

1. Transform the Institutional Learning Outcomes (ILOs) into Professional Institutional Learning Outcomes(Professional ILOs). Note: The Professional ILOs do not replace the regular Program Learning Outcomes (PLOs)that address the specific curriculum and skills of the program’s discipline or profession.

2. Develop definitions for the ILOs that are meaningful for the program’s discipline and level.3. Select or develop a rubric that accurately assesses student learning for the program’s new Professional ILO

definitions. Programs may continue to use the current LLU/AAC&U VALUE Rubrics or a rubric used within theprogram’s discipline/profession.

4. Alternatively, programs may choose to use clinical versions of the LLU/AAC&U rubrics that are currently beingdeveloped by the Learning Outcomes Committee. The clinical version of the Critical Thinking rubric should beavailable in February.

All programs can to choose whether to continue to use the regular ILOs or to move to the Professional ILO alternative assessment process.

4 A more complete resource on LLU’s Professional ILOs is posted here: http://home.llu.edu/academics/academic-resources/educational-effectiveness/llu-institutional-learning-outcomes

Academic Deans Council January 25, 2017 2

Academic Deans Council January 25, 2017 3

Academic Deans Council January 25, 2017 4

Academic Deans Council January 25, 2017 5

Academic Deans Council January 25, 2017 6

Academic Deans Council January 25, 2017 7

Academic Deans Council January 25, 2017 8

Academic Deans Council January 25, 2017 9

Academic Deans Council January 25, 2017 10

Academic Deans Council January 25, 2017 11

Academic Deans Council January 25, 2017 12

UpdatedOnlineCourseReclassificationProcessLoma Linda University • Division of Extended Education

In August, 2016, the Division of Extended Education (DoEE) developed an Online Course Reclassification Process. It included an initial review process at the school level that when completed would be sent to the DoEE for the final review and reclassification. This process placed a burden to the schools, so an alternative process has been developed in collaboration with Educational Technology Services (ETS). It is now active.

Below are the steps for online course instructors to initiate the upgrade process for their online courses that were designated as either “not aid eligible” or “correspondence” in the recent Online Course Audit. These online instructors should:

1. Send a request to ETS for a meeting with an instructional designer:https://llu.co1.qualtrics.com/SE/?SID=SV_eQ0F0y97VUvLZlP

a. A response from ETS will be sent to the instructor within 24 hours.

2. Work with ETS to set up an appointment with an instructional designer who will help theinstructor learn what needs to be done and how to move the course to the next classificationlevel. The instructor may need to meet with an instructional designer several times.

3. Complete the “Design and Development Reflection” survey (TBA) when the course has beensatisfactorily updated.

a. A second instructional designer will then review the course largely guided by thereflection survey responses.

b. The instructor and Academic Dean will be notified regarding the results of the course’sreclassification.

c. The course’s reclassification will be entered into the current course entry system forBanner.

4. Complete the “Implementation Reflection” (TBA) survey upon completion of teaching theredesigned course.

a. An instructional designer will then review/audit the completed course to confirm that itwas taught the way it was designed. If it was taught as designed, the updatedclassification will be retained. But if it wasn’t taught as designed, the course will bemoved back to the previous classification. It will need to go through the reclassificationprocess again.

b. The instructor and Academic Dean will be notified about the results of this final review.

This process will allow online courses to be reclassified before they have been taught but will still provide a final review upon completion of the course the next time it is taught. The final review will validate the instructor’s interactions and follow through on the new course design features and strategies.

Academic Deans Council January 25, 2017 13

Distance Education at Loma Linda University

Excellent online teaching promotes student success. It also helps LLU to meet federal regulations for

Title IV student financial aid eligibility.

Active Leaming. Active learning is the key for both online and face-to-face student success. Some

online instructors have asked why the federal government is focusing on distance education and not on

face-to-face education. The federal government is trying to ensure that online students receive a full

and complete learning experience equivalent to students in the face-to-face programs and courses.

Thus, what they require for distance education gives a good indication of what they believe is or

should be happening in the f2f classroom.

Required for Distance Education and Hybrid Courses. In brief, here are some of the things that are

required at LLU: • Regular and substantive interactions that include compelling discussions that draw in the

students• Weekly instruction by the instructor• Weekly requirement for student participation (attendance)• Clear and complete course syllabus• Meaningful and timely feedback• Respond to students' questions within 24 hours during the week• Instruction each week should meet LLU's credit hour policy and definition.• Include the Loma Linda experience through Mission Focused Learning (MFL)

Get to Know the Students. Online students are just as real and important as face-to-face students.

Instructors should take the time and effort to get to know them.

When the Instructor Can't Teach: Unplanned. If the instructor unexpectedly must miss a class session

due to extenuating circumstances including illness, accident, etc., they should at least let the students

know what has happened as quickly as possible and notify the program director. A qualified substitute

with the appropriate degree and expertise that meet accreditation standards should teach in the course

for duration of the instructor's absence.

When the Instructor Can't Teach: Planned. If the instructor knows before the course begins that they

will be absent one or more weeks-such as to attend a conference-and do not plan to interact with the

students during that time in Canvas, the course must be labeled ahead of time as a Hybrid

Correspondence course. However, if the instructor interacts with the students and asynchronously

runs the class while attending the conference, etc., the course must be labeled ahead of time as a Hybrid

Distance Education course.

Synchronous Courses. If a course is taught via Zoom, it should be noted in the course schedule ahead

of time that it is a synchronous online course. Otherwise, students may not be able to attend due to

schedule complications.

For more complete information, please review the LLU Distance Education Instructor Guide.

Academic Deans Council January 25, 2017 14

PROGRAM REVIEW RUBRIC Rubric for Assessing the Integration of Student Learning Assessment into Program Reviews

Criterion Initial Emerging Developed Highly Developed Required Elements of the Self-Study

Program faculty may be required to provide a list of program-level student learning outcomes.

Faculty are required to provide the program’s student learning outcomes and summarize annual assessment findings.

Faculty are required to provide the program’s student learning outcomes, annual assessment studies, findings, and resulting changes. They may be required to submit a plan for the next cycle of assessment studies.

Faculty are required to evaluate the program’s student learning outcomes, annual assessment findings, bench-marking results, subsequent changes, and evidence concerning the impact of these changes. They present a plan for the next cycle of assessment studies.

Process of Review

Internal and external reviewers do not address evidence concerning the quality of student learning in the program other than grades.

Internal and external reviewers address indirect and possibly direct evidence of student learning in the program; they do so at the descriptive level, rather than providing an evaluation.

Internal and external reviewers analyze direct and indirect evidence of student learning in the program and offer evaluative feedback and suggestions for improvement. They have sufficient expertise to evaluate program efforts. Departments use the feedback to improve their work.

Well-qualified internal and external reviewers evaluate the program’s learning outcomes, assessment plan, evidence, benchmarking results, and assessment impact. They give evaluative feedback and suggestions for improvement. The department uses the feedback to improve student learning.

Planning and Budgeting

The campus has not integrated program reviews into planning and budgeting processes.

The campus has attempted to integrate program reviews into planning and budgeting processes, but with limited success.

The campus generally integrates program reviews into planning and budgeting processes, but not through a formal process.

The campus systematically integrates program reviews into planning and budgeting processes, e.g., through negotiating formal action plans with mutually agreed-upon commitments.

Annual Feedback on Assessment Efforts

No individual or committee on campus provides feedback to departments on the quality of their outcomes, assessment plans, assessment studies, impact, etc.

An individual or committee occasionally provides feedback on the quality of outcomes, assessment plans, assessment studies, etc.

A well-qualified individual or committee provides annual feedback on the quality of outcomes, assessment plans, assessment studies, etc. Departments use the feedback to improve their work.

A well-qualified individual or committee provides annual feedback on the quality of outcomes, assessment plans, assessment studies, benchmarking results, and assessment impact. Departments effectively use the feedback to improve student learning. Follow-up activities enjoy institutional support

The Student Experience

Students are unaware of and uninvolved in program review.

Program review may include focus groups or conversations with students to follow up on results of surveys

The internal and external reviewers examine samples of student work, e.g., sample papers, portfolios, and capstone projects. Students may be invited to discuss what they learned and how they learned it.

Students are respected partners in the program review process. They may offer poster sessions on their work, demonstrate how they apply rubrics to self-assess, and/or provide their own evaluative feedback.

Academic Deans Council January 25, 2017 15

Guidelines for Using the Program Review Rubric For the fullest picture of an institution’s accomplishments, reviews of written materials should be augmented with interviews at the time of the visit.

Dimensions of the Rubric: 1. Self-Study Requirements. The campus should have explicit requirements for the program’s self-study, including an analysis of the program’s learning

outcomes and a review of the annual assessment studies conducted since the last program review. Faculty preparing the self-study can reflect on theaccumulating results and their impact, and plan for the next cycle of assessment studies. As much as possible, programs can benchmark findingsagainst similar programs on other campuses.

Questions: Does the campus require self-studies that include an analysis of the program’s learning outcomes, assessment studies, assessment results, benchmarking results, and assessment impact, including the impact of changes made in response to earlier studies? Does the campus require an updated assessment plan for the subsequent years before the next program review?

2. Self-Study Review. Internal reviewers (on-campus individuals) and external reviewers (off-campus individuals, usually disciplinary experts) evaluatethe program’s learning outcomes, assessment plan, assessment evidence, benchmarking results, and assessment impact; and they provide evaluativefeedback and suggestions for improvement.

Questions: Who reviews the self-studies? Do they have the training or expertise to provide effective feedback? Do they routinely evaluate the program’s learning outcomes, assessment plan, assessment evidence, benchmarking results, and assessment impact? Do they provide suggestions for improvement? Do departments effectively use this feedback to improve student learning?

3. Planning and Budgeting. Program reviews are not be pro forma exercises; they should be tied to planning and budgeting processes, with expectationsthat increased support will lead to increased effectiveness, such as improving student learning and retention rates.

Questions: Does the campus systematically integrate program reviews into planning and budgeting processes? Are expectations established for the impact of planned changes?

4. Annual Feedback on Assessment Efforts. Institutions often find considerable variation in the quality of assessment efforts across programs. Whileprogram reviews encourage departments to reflect on multi-year assessment results, some programs are likely to require more immediate feedback,usually based on a required annual assessment report. This feedback might be provided by an assessment director or committee, relevant dean orothers; and whoever has this responsibility should have the expertise to provide quality feedback.

Questions: Does someone or a committee have the responsibility for providing annual feedback on the assessment process? Does this person or team have the expertise to provide effective feedback? Does this person or team routinely provide feedback on the quality of outcomes, assessment plans, assessment studies, benchmarking results, and assessment impact? Do departments effectively use this feedback to improve student learning?

5. The Student Experience. Students have a unique perspective on a given program of study: they know better than anyone what it means to go throughit as a student. Program review can take advantage of that perspective and build it into the review.

Questions: Are students aware of the purpose and value of program review? Are they involved in preparations and the self-study? Do they have an opportunity to interact with internal or external reviewers, demonstrate and interpret their learning, and provide evaluative feedback?

Academic Deans Council January 25, 2017 16

Academic Deans Council January 25, 2017 17

Academic Deans Council January 25, 2017 18