WEBINAR FOR

ACBSP

FIRST-TIME EVALUATORSPresented Spring 2013

ACCREDITATION COUNCIL FOR BUSINESS SCHOOLS

AND PROGRAMS

Purpose of this webinar:

Provide additional information for our first time evaluators.

Provide additional information on how to write strengths and opportunities for improvement comments.

Provide an opportunity for you ask any questions you may have.

Agenda:• Welcome and Introductions• Role of the evaluator• Resources available to site visit team• Scheduling of Consensus Calls and your role

during the call• Travel arrangements and travel policies• Communication with institutional contact• Site Visit Schedule• Format for final comments to be included in

the feedback report• Review and submission of feedback report• Submission of expense reports and other

forms • Any other topics

ACBSP STAFF CONTACTS:Steve Parscale

Director of Accreditation

913-339-9356,

Mobile: 913-526-5285

sparscale@acbsp.org

Diana Hallerud

Assistant Director of Accreditation

913-339-9356,

Mobile: 816-699-9964

dianahallerud@acbsp.org

ACBSP STAFF CONTACTS:Ron DeYoung, Executive Liaison for the Baccalaureate/Graduate Degree Board of Commissioners, DEYOUNG@nwmissouri.edu Mobile: 816-695-9585

Larry Zachrich, Executive Liaison for the Associate Degree Board of Commisisoners, lzachrich@acbsp.org Mobile: 419-966-7091

ACBSP STAFF CONTACTS:

Maria Hallerud, Accreditation Manager, mhallerud@acbsp.org Office: 913-339-9356

Andrea (Andy) Ernst, Accreditation Administrative Assistant aernst@acbsp.org Office: 913-339-9356

Primary Role of the Evaluator:

• Evaluate and validate the information provided in the self-study to the standards and criteria.

• Provide feedback ready comments to the business unit regarding strengths and opportunities for improvement based on information from the self-study and observations during the site visit.

Travel arrangements:

Review the Travel Guidelines

Evaluators should plan on using their personal credit cards to pay for airfare and be reimbursed through ACBSP.

There are exceptions for international travel, when the institution may choose to make the flight arrangements for you or when you can use the ACBSP travel agent. You should always check with Steve Parscale or Diana Hallerud before contacting the ACBSP travel agent.

The institution will typically make the hotel reservation for the site visit team, but it varies by institution for the billing arrangements. The institution sometimes sets up a direct billing arrangement. Most institutions ask the site visit team to pay for the hotel room directly and seek reimbursement through ACBSP.

Travel expenses are processed on the 15th and the 30th of the month.

Submit your expense forms and receipts to Mary Riley via email at maryriley@acbsp.org, fax to 913-339-6226, or send snail mail.

Resources available to you, cont.

Files on the CD and notebook you received

Under Tab One:

Evaluator Training PowerPoint Slides – June 2011

Scoring Guidelines & Worksheet

Practical Guide for Evaluators (includes)

ACBSP Observation Card

ACBSP Site Visit Notes

Do’s & Dont’s for Evaluators

Site visit Issue Worksheet

Site Visit Questions

Site Visit Issue Strategy

Resources available to you, cont.

Files on the CD and

notebook you received

Under Tab Two:

Process Manuals

Under Tab Three:

Standards and Criteria

Under Tab Four:

Evaluator Workbook

Feedback Report

Executive Summary Template

Resources available to you, cont.

Files on the CD and notebook you received

Under Tab Five:

Travel Reimbursement Guidelines

Travel Reimbursement Form

Travel Expense Form

Evaluator Agreement Form

Evaluator Critique Form

Review Process:

• Review contents of package and verify that you have received items checked on the checklist.

• Read the entire self-study, including the overview, prior to the consensus call. The overview will provide important information regarding mission, vision, student demographics, strategic challenges, etc.

Reviewing the self-study, cont.

READING THE ENTIRE SELF-STUDY DOCUMENT IS IMPORTANT BECAUSE

a standard or specific criteria may not be addressed where you expect to see it. Don’t be surprised if you see student satisfaction results in Standard 4 rather than Standard 3 or Standard 4 items such as learning outcomes in Standard 3 or Standard 6.

the standards have many linkages and information can be found throughout the document. Some self-studies even have most of the results in the appendices rather than the self-study document.

the information may not be in the correct place, but the business unit can still meet the standard. It just requires additional effort on your part to locate the information.

Using post-its or different colored highlighters when you find information referencing the standards you are responsible for can be helpful when you begin preparing your comments.

Review Process:• Use the evaluator workbook and

begin writing observations prior to the consensus call.

• The team chair will notify each team member regarding the standards you will have primary responsibility for reviewing and providing strengths and OFI’s. The chair may ask you if you have a preference based on your expertise.

Contact with the Institutional Coordinator/Contact All communication with the institutional

coordinator or contact should be routed through your team chair.

Any requests for clarification or additional information should be forwarded to the team chair.

This way the institutional coordinator is not responding to duplicate requests for information. Your cooperation is appreciated.

The Consensus Call:

• Using the scoring worksheet, evaluate each of the standards using the best in class to major improvements needed criteria.

• The team chair will review site visit logistics and lead the discussion regarding observations after reviewing the self-study.

The Consensus Call continued.

• For baccalaureate/graduate degree site visits, the commissioner and the mentor are invited to participate in the call and answer questions.

• For associate degree site visits, the mentor is invited to participate in the consensus call.

The Consensus Call continued.

• The team will identify site visit issues and additional information needed to be supplied prior to or during the site visit.

• Discussion regarding meetings and the site visit agenda will be finalized.

Overall Scoring Summary Worksheet

Use these scoring dimensions to complete the table on the following page. Your individual review will be discussed during the consensus call. 

Individual Assessment Dimensions

Best in Class

Good to Excellent

Good

Major Improvements Needed

Poor

Scoring BandQualitative Quantitative

Poor * no systematic approach evident; anecdotal information Poor * no results or poor results in areas reported

* beginning of a systematic approach to the primary purposes of the item

* early stages of developing trends; some improvements and/or early good performance levels in a few areas

* early stages of a transition from reacting to problems to a general improvement orientation

* major gaps exist in deployment that would inhibit prog- ress in achieving the primary purposes of the item

* a sound, systematic approach, responsive to the pri- mary purposes of the itema fact-based improvement process in place in key areas; more emphasis is placed on improvement than on reaction to problems

no pattern of adverse trends and/or poor performance levels in areas of importance to the applicant’s key

no major gaps in deployment, though some areas or business requirementswork units may be in very early stages of deployment * some trends and/or current performance levels --

evaluated against relevant comparisons and/or good to very good relative performance levels

* a sound, systematic approach, responsive to the over- all purposes of the item

* current performance is good to excellent in most areas of importance to the applicant’s key business requirements

a fact-based improvement process is a key manage- ment tool; clear evidence of refinement and improved

* most improvement trends and/or performance levels are sustained

integration as a result of improvement cycles and anal- ysis

many to most trends and/or current performance levels -- evaluated against relevant comparisons and/or bench-

* approach is well-deployed, with no major gaps; deploy- ment may vary in some areas or work units

marks -- show areas of leadership and very good relative performance levels

* a sound, systematic approach, fully responsive to all the requirements of the item

current performance is excellent in most areas of importance to the applicant’s key business require-

a very strong, fact-based improvement process is a mentskey management tool; strong refinement and integra- tion -- backed by excellent analysis

* excellent improvement trends and/or sustained excellent performance levels in most areas

* approach is fully deployed without any significant weaknesses or gaps in any areas or work units

* strong evidence of industry and benchmark leadership demonstrated in many areas

*

*

*

*Best in Class Best in Class

* results not reported for many to most areas of impor- tance to the applicant’s key business requirements

*

*

Good

Good to Excellent

Good to Excellent

improvement trends and/or good performance levels reported for many to most areas of importance to the applicant’s key business requirements

Results

*

Dimensions Approach/Deployment Dimensions

Good

Major Improvements Needed

Major Improvements Needed

*

ASSOCIATE DEGREE SCORING SUMMARY WORKSHEETEvaluator Name: School Name:

SUMMARY OF EXAMINATION ITEMS Individual Assessment Dimension1 LEADERSHIP 2 STRATEGIC PLANNING 3 STUDENT, STAKEHOLDER, AND MARKET FOCUS

4 MEASUREMENT, ANALYSIS, AND KNOWLEDGE MANAGEMENT 4.1 Student Learning Outcomes Assessment 4.2 Program Evaluation 4.3 Student Assessment 5 FACULTY AND STAFF FOCUS 5.1 Human Resource Planning 5.2 Faculty Qualifications 5.3 Faculty Composition 5.4 Faculty Deployment 5.5 Faculty Load 5.6 Faculty Evaluation

5.7 Faculty and Staff Professional Development & Scholarly Activities 5.8 Faculty Instructional Development 5.9 Faculty Operational Policies, Procedures, & Practices6 PROCESS MANAGEMENT 6.1 Curriculum 6.2 Professional Component 6.3 General Education Component 6.4 Business Major Component6.5 Off-Campus Operations and Unique Items6.6 Minimum Grade Requirement6.7 Learning and Academic Resources6.8 Support Services6.9 Educational Innovation6.10 Articulation and Transfer Relationships

Scoring Summary worksheetBaccalaureate/Graduate Degree:

Examiner Name:School's Name:

SUMMARY OF EXAMINATION ITEMSIndividual Assessment

Dimension

1 LEADERSHIP

2 STRATEGIC PLANNING

3 STUDENT AND STAKEHOLDER FOCUS 4 MEASUREMENT AND ANALYSIS OF STUDENT LEARNING AND PERFORMANCE 4.1 Selection and Use of Information and Data

4.2 Selection and Use of Information Results4.3 Selection and Use of Comparative Information and Data4.4 Continuous Process Improvement5 FACULTY AND STAFF FOCUS 5.1 Human Resource Planning5.2 Employment5.3 Faculty Deployment5.4 Faculty Size and Load5.5 Faculty Evaluation5.6 Faculty and Staff Development5.7 Faculty Operational Procedures, Policies, and Practices5.8 Scholarly and Professional Activities6 EDUCATIONAL AND BUSINESS PROCESS MANAGEMENT 6.1 Education Design and Delivery 6.2 Management of Educational Support Service Processes and Business Operation Processes6.3 Enrollment Management

Typical Site Visit Schedule:

Review the typical schedule outlined in the Practical Guide for Evaluators.

Typically, team members arrive Sunday afternoon/early evening and have a team meeting to discuss site visit issues and finalize the schedule.

Monday is typically filled with meetings with administrators, faculty, students, advisory boards, assessment officer, etc.

Most teams complete all of the meetings on Tuesday morning and hold any follow-up meetings early afternoon. Tuesday afternoon and evening will be used to finish writing your portion of the report.

Your completed portion of the report should be given to the team chair to be used to prepare the executive summary and as a part of the exit interview.

CONTINUOUS IMPROVEMENT PROCESS

APPROACH

DEPLOYMENT OR IMPLEMENTATION

RESULTS

andLEARNING

IMPROVEMENT

Keep in mind:

When describing processes for the business school or programs, that the overall value of the results and improvements taking place is dependent on the approach (what they do) and the level and extent of deployment (how they do it).

Review Process after Consensus Call:

• Continue to refine comments for the feedback report prior to the site visit – share with the team chair.

• Ask for guidance when needed – there are no dumb questions and every chair has experienced a first visit – he or she will welcome any questions you have.

MOST IMPORTANT THING TO REMEMBER AS AN EVALUATOR:

A site visit that is conducted in an organized and professional manner followed by a feedback report that is professionally and clearly written will create a positive impression of what ACBSP represents as an accrediting body.

While you may travel to a location new to you, participating in a site visit is not a vacation and should not be considered as such.

AS AN EVALUATOR, DO… Remember that you serve as an ambassador for ACBSP. Most

of the faculty and staff you will be interacting with may not have an opportunity to attend regional or national conferences. The site visit experience will be what they base their perception of ACBSP on. Make it a positive one!

Come prepared for the site visit with materials read in advance. You are part of a team with your team members depending on you. You will work hard during the site visit and will find it immensely rewarding.

Say thank you and complement the institution for their hospitality and availability of materials. Many institutions provide resources and snacks over and above what is expected.

Make every attempt to stay on the established schedule and let the next person know in advance if the team is falling behind schedule.

Let the team chair contact the institutional contact if additional information or resource is needed. A little diligence on your part will result in a stronger report and assist the Board of Commissioners in making the accreditation decision.

AS AN EVALUATOR, DO…

Let the team chair serve as the primary point of contact in making the arrangements both in advance and during the site visit.

Contact the ACBSP office if you encounter any problems and are not sure how to proceed.

Remember that each institution has its own mission and operating processes and procedures. You’re only concerned with how the processes and procedures used serve the institution, not how they compare to your institution.

Remember that the size and location of the institution you are visiting will determine the type of lodging available during your stay. Institutions are asked to provide reasonable, clean, comfortable accommodations as close to the campus as possible.

AS AN EVALUATOR, DO NOT…

Market your services as a consultant or as a potential recruit at the institution. The evaluator’s role is to evaluate and validate the information in the self-study, not to offer personal assistance when a gap in a particular area is observed or to look for employment opportunities while on campus.

Under any circumstances, attempt to recruit faculty as either a student or employee from the institution you are visiting. Even if a faculty member voices an interest in leaving the institution and there is a position available at your institution, do not discuss this opening with the faculty member. Faculty members who express an interest in seeking additional degrees should be directed to ACBSP for a list of institutions available rather than to your specific institution.

Cancel appointments with administrators unless you can provide a valid reason for the cancellation. Usually the President or Vice-Presidents work their schedules around the site visit schedule and do not appreciate a sudden cancellation.

AS AN EVALUATOR, DO NOT…

Under any circumstances, make negative comments regarding the business faculty or administration to anyone during the site visit.

Criticize the content or quality of the self-study during the site visit. If a standard is not adequately addressed, let the institutional contact know what is missing and ask if the information can be derived from other sources.

Wait more than 10 days to submit the feedback report to ACBSP. Do not submit the report unless you proofread the report thoroughly before submitting to ACBSP. All team members should have the opportunity to read the final report prior to submitting the final report. Please make sure the comments are complete and accurately reflect your observations as an evaluator.

Writing Comments: THINGS TO REMEMBER:

Comments are based on the requirements of the standards and criteria, not personal opinions or how processes are followed at other institutions.

Comments should be clear and concise and should address the situation, how the business unit does (for strengths) or does not meet (for OFI’s) the standard, and explain the significance of the situation.

Comments should not be prescriptive unless you are using language directly from the standards and criteria.

Comments should be written in complete sentences and should use proper grammar and punctuation.

Avoid the use of slang or local expressions others may not be familiar with, especially when on international visits. Example: “pig in a poke”.

Writing a strength:

If the institution has a strong strategic plan that includes both short and long-term goals and

has a good approach,

has been deployed or implemented,

demonstrates learning,

and identifies improvements, your comment could state:

“The John Doe School of Business has a strategic plan that has identified both short and long-term goals. The plan is systematic with an approach that has been implemented for all disciplines, provides examples of learning and evidence of improvements the school has made. The strategic plan helps the school to plan for increasing enrollment and the replacement of faculty members as they retire.”

Writing a strength:

If the institution has an assessment instrument and provides evidence of a systematic process and examples of how the results are used to make improvements, you could provide a comment such as:

“The John Doe School of Business has administered the MFT test for all disciplines for five years. Each department documents their review of the pre and post results each semester during department meetings and makes improvements based on the results. For example, the accounting department added additional projects for preparing financial statements in the ACCT 3300 course.”

Writing an opportunity for improvement:Suppose the institution has identified:

increasing enrollment and a large number of faculty scheduled for retirement as strategic challenges.

All the strategic objectives in the strategic plan are for one to three years, and faculty are not consulted in the development of the objectives.

The opportunity for improvement could read,

“While the John Doe School of business has strategic objectives for up to three years, there was no evidence of planning beyond three years or involvement by faculty in the development of the objectives. Developing long term goals and involving more stakeholders such as faculty could provide resource planning to meet the increasing enrollment and replacement of faculty members scheduled to retire in the next five years.

Writing an Opportunity for Improvement:

Suppose the institution administers the Major Field Test (MFT) to all students as a pre and post test.

There is no evidence, either in the self-study or on-site, that the results are reviewed to indicate that there is learning or subsequent improvements are made. In addition, the school does not use the data for comparative purposes

Your opportunity for improvement comment could read:

“The John Doe School of Business provided three years of data for the Major Field Test (MFT). However, there is no evidence that the school is using this data to measure how their students are performing in each discipline and how the results compare to other institutions with a similar mission. Analyzing the results to identify program strengths and weaknesses could allow the school of business to make improvements where needed and provide an external source of comparative data.”

Submitting your final comments: Using your initial review, you will conduct interviews and

review documentation to validate the information provided in the self-study.

You may find that some initial strengths become OFI’s and some OFI’s will either be reduced and/or become a strength.

Wording for OFI’s should change from “it is not clear” to “there is no evidence” following the site visit.

Before submitting your comments to the chair, read them from the perspective of someone who did not write the self-study. Based on the comment, would the person understand the jist of the comment? Can they relate it to the standard or criteria you are referencing?

Ask your team members for help if you get stuck. They won’t mind and will be asking you to read a comment for clarity as well.

Final Steps:

On Tuesday evening, you will submit your final comments to the chair and confer with the team if you have questions.

Wednesday morning, you will participate in the exit interview and return home.

The chair should send the report for a final review to team members before submitting it to ACBSP. Respond with your comments promptly and let the chair know if you saw evidence of meeting a standard that a team member may have missed or if a comment is not clear to you.

Submit your expense report, receipts and evaluation form to ACBSP as soon as you return from the trip.

Questions?

While you will work hard, most evaluators find the site visit experience to be very rewarding and no matter how many visits they have participated in, they always learn something new that they can use to improve the processes at their own institution.

Thank you for participating in this webinar. If you have any suggestions for improvement, please feel free to contact me by phone or via email at dianahallerud@acbsp.org.

We hope you enjoy your first site-visit experience.