We’re Accredited! Now What?
Preparing the First QA Report
Presented by: Aaron Glassman, D.Mgt.Kelly Whealan George, PhD Dixie Button, DBA
University Overview
• 3 campuses:
– Daytona Beach (~ 4500 UG & 500 grad)
– Prescott (~ 1500 students)
– World Wide (>150 locations, ~ 40 states, S. America, Europe, Asia and Middle East; ~26,000 FTE students)
• Emphasis is aviation & aerospace
• BS, MS and PhD programs
• ~ $17M in research
Personnel Changes
Accreditation granted in December 2013. Since then . . .
• Third Reorganization
• Third Dean
• Third CAO
• Chancellor, Combined CAO/Chancellor, back to Chancellor
• New Accreditation campus liaison to work w/University liaison
• New faculty with faculty roles
• ACBSP co-champion leaves. New co-champion appointed 5 months before QA report due
Personnel Changes
In the midst of personnel changes . . . .
• What is this thing called Notes and Conditions attached to the accreditation?
• What is this thing called a Quality Assurance (QA) report?
Design Teamwork for Success
ACBSP Committee to address upcoming QA report issues
• Monthly teleconferences
• Knowledge management (Sharepoint)
• Developed a road map
• Attempt to intersect QA requirements with all other program requirements (SACS, QEP….)
• Spreads knowledge base around the College
Conditions and Notes
Notes & Conditions attached to ACBSP accreditation
• Okay . . . But what does this mean?
• The initial accreditation letter says “Conditions and notes indicate that either the Standard or Criteria is not fully met, and should be viewed as an opportunity to move your program to a higher level of excellence.”
• First thought . . . Is satisfaction of Conditions & Notes due by the first QA report?
• No, just need to show that we are addressing them.
Conditions and Notes
ACBSP Process Book - ConditionsMost standards are met, but there are some deficiencies in fully meeting all of the ACBSP standards.
ACBSP Process Book - NotesIn compliance with ACBSP standards, but not some of the criteria
. . . Notes represent concerns of the Board of Commissioners and until those concerns are dealt with by the institution and accepted by ACBSP, the progress on improving the process to remove the notes should be included in the quality assurance report submitted to ACBSP every two years.
Conditions and Notes
Conditions must be removed by 9/15/2017 and progress on notes must be reported in future Quality Assurance Reports the first of which is due September 15, 2015
• Note on Standard 2
• Condition on Standard 3
• Condition on Standard 4, Criterion 4.2, 4.3, and 4.4
• Note on Standard 5, Criterion 5.3.2.b
• Note on Standard 6, Criterion 6.3.6 and 6.3.7
Conditions and Notes
Note on Standard 2
• Referenced input from faculty in the development of both long-term and short-term strategic goals and objectives with measures, action plans, and timelines.
• Result: Removed
Condition on Standard 3
• Need feedback from stakeholders of the business programs other than students.
• Result: Condition reduced to a Note
Conditions and Notes
Condition on Standard 4, Criterion 4.2, 4.3, and 4.4
• Show evidence of changes/improvements resulting from internal assessments
• Need to better document and analyze trend data as well as use comparative data to continuously improve student learning outcomes.
• Result: Conditions reduced to a Note
Conditions and Notes
Note on Standard 5, Criterion 5.3.2.b
• Show progress on plan to ensure 40% of undergraduate credit hours taught by academically qualified faculty within two years.
• Result: Ongoing, Note maintained
Note on Standard 6, Criterion 6.3.6 and 6.3.7
• Document results & improvements resulting from newly implemented dashboard
• Result: Notes removed
Shifting Priorities
• QEP (Undergraduate Research)
• Research
• Externally Funded Research
• Promotion and Tenure
• Cross-campus collaboration
Publicly Available Information
ACBSP Requirement
• For Demonstrating Excellence in Baccalaureate/Graduate degree Business Programs the business unit must routinely provide reliable information to the public on its performance, including student achievement, such as assessment results.
Publicly Available Information
What to include in publicly available information?
• Formed a committee, developed policy, execution ongoing. (http://worldwide.erau.edu/degrees-programs/colleges/business/index.html)
Publicly Available InformationBiggest Challenges
Sources of Information
• ACBSP requirements from acbsp.org • Review of other ACBSP accredited colleges, exp. North
Carolina Central University and Gardner-Webb University
Publicly Available InformationBiggest Challenges (Cont.)
What will be displayed
• Student Satisfaction Survey• Worldwide Alumni Survey• Graduation and Retention Rates for the College of Business• External Assessment Data (Peregrine), to fulfill the
requirement of publishing student learning outcome assessment results
How we assess programs…..
• Courses have activities
• Activities are mapped to Learning Outcomes
• Learning Outcomes are mapped to Program Outcomes
• Program Outcomes Rubrics Embedded in LMS
• Learning Analytics Data Extracted from LMS
Tips, Tricks, and Traps
Tip #1 – Do not map all activities even though it can be done.
• Only assess activities that have grade significance (students are less likely to skip the activity and grades are more normally distributed).
• Avoid mapping discussion board posts and other “route” weekly assignments as interrater reliability is usually poor even with a rubric.
Tips, Tricks, and Traps
Tip #2 – Provide clear directions to faculty on using rubrics with LO assessments embedded.
• Faculty may unintentionally override the rubric to give the student a “0” or “100” score which means LO data is not collected.
• Faculty need to understand that attaining a LO does not reflect the students activity score. (e.g., student can earn a 75 in a paper for improper APA formatting and other non-content deductions but still achieve the mapped LO’s at the level of mastery)
Tips, Tricks, and Traps
Trap #1 – We only think we are assessing programs.
• We are actually assessing course activities and the sum of the activities across multiple courses is how we gauge program health.
• Students from different programs may take the same course.
Tips, Tricks, and Traps
Trap #2 – Binary/Administrative grades pollute the data.
• Students who receive zero scores for not submitting assignments skew learning outcome attainment data (averages) to the negative.
• Activities that are pass/fail (100pts) skew the outcome attainment data to the positive.
• These errors are hard to account for.
Tips, Tricks, and Traps
Trick #1 – Use multiple data streams to assess programs.
• Direct Measure – Student Scores on Mapped Activity, Capstone, External Assessment, Artifact Grade/Regrade, etc. across multiple sections of multiple classes.
• Indirect Measure – Post-Course Feedback (Student), Post-Course Feedback (Faculty), Grade Distributions, Admissions vs. Graduation Rates
• Configure all systems with assessment in mind!
Lessons Learned
• Fight for continuity
• Prepare yearly report similar to the 5th year QEP summary
• Incentivize key people
• Development by attending ACBSP conference
Contacts
Dr. Kelly Whealan George
Director of Accreditation, Assessment, and
Research Advancement of IgniteDiscipline Chair, Social Sciences
and Economics
Dr. Dixie ButtonMBA-A Program ChairACBSP Co-Champion
Dr. Aaron Glassman
Department Chair, Technology Management
Department
Top Related