We’re Accredited! Now What?

Preparing the First QA Report

Presented by: Aaron Glassman, D.Mgt.Kelly Whealan George, PhD Dixie Button, DBA

University Overview

• 3 campuses:

– Daytona Beach (~ 4500 UG & 500 grad)

– Prescott (~ 1500 students)

– World Wide (>150 locations, ~ 40 states, S. America, Europe, Asia and Middle East; ~26,000 FTE students)

• Emphasis is aviation & aerospace

• BS, MS and PhD programs

• ~ $17M in research

Personnel Changes

Accreditation granted in December 2013. Since then . . .

• Third Reorganization

• Third Dean

• Third CAO

• Chancellor, Combined CAO/Chancellor, back to Chancellor

• New Accreditation campus liaison to work w/University liaison

• New faculty with faculty roles

• ACBSP co-champion leaves. New co-champion appointed 5 months before QA report due

Personnel Changes

In the midst of personnel changes . . . .

• What is this thing called Notes and Conditions attached to the accreditation?

• What is this thing called a Quality Assurance (QA) report?

Design Teamwork for Success

ACBSP Committee to address upcoming QA report issues

• Monthly teleconferences

• Knowledge management (Sharepoint)

• Developed a road map

• Attempt to intersect QA requirements with all other program requirements (SACS, QEP….)

• Spreads knowledge base around the College

Conditions and Notes

Notes & Conditions attached to ACBSP accreditation

• Okay . . . But what does this mean?

• The initial accreditation letter says “Conditions and notes indicate that either the Standard or Criteria is not fully met, and should be viewed as an opportunity to move your program to a higher level of excellence.”

• First thought . . . Is satisfaction of Conditions & Notes due by the first QA report?

• No, just need to show that we are addressing them.

Conditions and Notes

ACBSP Process Book - ConditionsMost standards are met, but there are some deficiencies in fully meeting all of the ACBSP standards.

ACBSP Process Book - NotesIn compliance with ACBSP standards, but not some of the criteria

. . . Notes represent concerns of the Board of Commissioners and until those concerns are dealt with by the institution and accepted by ACBSP, the progress on improving the process to remove the notes should be included in the quality assurance report submitted to ACBSP every two years.

Conditions and Notes

Conditions must be removed by 9/15/2017 and progress on notes must be reported in future Quality Assurance Reports the first of which is due September 15, 2015

• Note on Standard 2

• Condition on Standard 3

• Condition on Standard 4, Criterion 4.2, 4.3, and 4.4

• Note on Standard 5, Criterion 5.3.2.b

• Note on Standard 6, Criterion 6.3.6 and 6.3.7

Conditions and Notes

Note on Standard 2

• Referenced input from faculty in the development of both long-term and short-term strategic goals and objectives with measures, action plans, and timelines.

• Result: Removed

Condition on Standard 3

• Need feedback from stakeholders of the business programs other than students.

• Result: Condition reduced to a Note

Conditions and Notes

Condition on Standard 4, Criterion 4.2, 4.3, and 4.4

• Show evidence of changes/improvements resulting from internal assessments

• Need to better document and analyze trend data as well as use comparative data to continuously improve student learning outcomes.

• Result: Conditions reduced to a Note

Conditions and Notes

Note on Standard 5, Criterion 5.3.2.b

• Show progress on plan to ensure 40% of undergraduate credit hours taught by academically qualified faculty within two years.

• Result: Ongoing, Note maintained

Note on Standard 6, Criterion 6.3.6 and 6.3.7

• Document results & improvements resulting from newly implemented dashboard

• Result: Notes removed

Shifting Priorities

• QEP (Undergraduate Research)

• Research

• Externally Funded Research

• Promotion and Tenure

• Cross-campus collaboration

Publicly Available Information

ACBSP Requirement

• For Demonstrating Excellence in Baccalaureate/Graduate degree Business Programs the business unit must routinely provide reliable information to the public on its performance, including student achievement, such as assessment results.

Publicly Available Information

What to include in publicly available information?

• Formed a committee, developed policy, execution ongoing. (http://worldwide.erau.edu/degrees-programs/colleges/business/index.html)

Publicly Available InformationBiggest Challenges

Sources of Information

• ACBSP requirements from acbsp.org • Review of other ACBSP accredited colleges, exp. North

Carolina Central University and Gardner-Webb University

Publicly Available InformationBiggest Challenges (Cont.)

What will be displayed

• Student Satisfaction Survey• Worldwide Alumni Survey• Graduation and Retention Rates for the College of Business• External Assessment Data (Peregrine), to fulfill the

requirement of publishing student learning outcome assessment results

How we assess programs…..

• Courses have activities

• Activities are mapped to Learning Outcomes

• Learning Outcomes are mapped to Program Outcomes

• Program Outcomes Rubrics Embedded in LMS

• Learning Analytics Data Extracted from LMS

Tips, Tricks, and Traps

Tip #1 – Do not map all activities even though it can be done.

• Only assess activities that have grade significance (students are less likely to skip the activity and grades are more normally distributed).

• Avoid mapping discussion board posts and other “route” weekly assignments as interrater reliability is usually poor even with a rubric.

Tips, Tricks, and Traps

Tip #2 – Provide clear directions to faculty on using rubrics with LO assessments embedded.

• Faculty may unintentionally override the rubric to give the student a “0” or “100” score which means LO data is not collected.

• Faculty need to understand that attaining a LO does not reflect the students activity score. (e.g., student can earn a 75 in a paper for improper APA formatting and other non-content deductions but still achieve the mapped LO’s at the level of mastery)

Tips, Tricks, and Traps

Trap #1 – We only think we are assessing programs.

• We are actually assessing course activities and the sum of the activities across multiple courses is how we gauge program health.

• Students from different programs may take the same course.

Tips, Tricks, and Traps

Trap #2 – Binary/Administrative grades pollute the data.

• Students who receive zero scores for not submitting assignments skew learning outcome attainment data (averages) to the negative.

• Activities that are pass/fail (100pts) skew the outcome attainment data to the positive.

• These errors are hard to account for.

Tips, Tricks, and Traps

Trick #1 – Use multiple data streams to assess programs.

• Direct Measure – Student Scores on Mapped Activity, Capstone, External Assessment, Artifact Grade/Regrade, etc. across multiple sections of multiple classes.

• Indirect Measure – Post-Course Feedback (Student), Post-Course Feedback (Faculty), Grade Distributions, Admissions vs. Graduation Rates

• Configure all systems with assessment in mind!

Lessons Learned

• Fight for continuity

• Prepare yearly report similar to the 5th year QEP summary

• Incentivize key people

• Development by attending ACBSP conference

Contacts

Dr. Kelly Whealan George

Director of Accreditation, Assessment, and

Research Advancement of IgniteDiscipline Chair, Social Sciences

and Economics

Dr. Dixie ButtonMBA-A Program ChairACBSP Co-Champion

Dr. Aaron Glassman

Department Chair, Technology Management

Department