Preparing for ABET EAC Evaluation Visit r032916

Preparing for ABET EAC Evaluation Visit

March 30, 2015

2

DisclaimerThe information presented here represents the experience of the consultant and does not represent any endorsement by either the ABET Foundation or ABET, Inc..

3/29/2016

3

Topics• What/Who is ABET?• Context for Evaluation• Timeline & Terminology• PEV Responsibilities Before, During &

After Visit• Typical Visit Schedule• EAC Criteria & Suggestions• Questions3/29/2016

4

ABET’s Core Purpose

With ABET accreditation, students, employers, and the society we serve can be confident that a program meets the quality standards that produce graduates prepared to enter a global workforce

3/29/2016

5

Who Recognizes ABET?In the U.S.

• 35 Member and Associate Member Societies of ABET• Council for Higher Education Accreditation (CHEA)• State Boards for Engineering & Surveying Licensure & Registration

(over 55 jurisdictions)• U.S. Patent Office• U.S. Reserve Officers Training Corps• Council of Engineering Specialty Boards (CESB)• Board of Certified Safety Professionals (BCSP) • Accreditors in other disciplines• U.S. Trade Office• U.S. State Department• Employers (position announcements)

3/29/2016

ABET’s 35 Member Societies

http://www.aaees.org/

7

What Does ABET Accredit?• Academic program leading to specific

degree in a specific discipline• Misconceptions clarified:

– Not institutions– Not schools, colleges, or departments – Not facilities, courses, or faculty– Not graduates– Not degrees

3/29/2016

8

Objectives of ABET Accreditation

(1) Assure that graduates of an accredited program are adequately prepared to enter and continue the practice of engineering’

(2) Stimulate the improvement of engineering education;

(3) Encourage new and innovative approaches to engineering education and its assessment; and

(4) Identify accredited programs to the public.

3/29/2016

9

ABET Impact• Approximately 3,500 programs at

over 700 colleges and universities in 29 countries have received ABET accreditation.

• Approximately 85,000 students graduate from ABET-accredited programs each year, and millions of graduates have received degrees from ABET-accredited programs since 1932.3/29/2016

• Accredited programs by commission: ASAC: 81 CAC: 429 EAC: 2437 ETAC: 216

ABET Accreditation StatisticsAs of 1 October 2015 … 3,569 Programs

CommissionDomestic Non-Domestic

Programs Institutions Programs InstitutionsASAC 80 62 1 1CAC 377 296 52 35EAC 2071 424 366 76ETAC 581 204 59 12

11

How is ABET Structured?

3/29/2016

Organizational StructureVolunteer-Driven: 2,200+ Volunteers

100% of accreditation decisions are made by volunteers

Board of Directors• Elected by Board of

Delegates• Provides strategic

direction and plans• Appeals process

4 Commissions• ASAC, CAC,

EAC, ETAC• Make decisions

on accreditation status

• Implement accreditation policies

• Propose changes to criteria

Program Evaluators• Visit campuses• Evaluate individual

programs• Make initial

accreditation recommendations

• “Face of ABET”

Board of Delegates• Nominated by &

represent the member societies

• Decides policy and procedures

• Approves criteria

13

Proliferation of criteria

Need for innovation in programs

Prescriptiveness of criteria

Industry call for change – continuous improvement & preparation for professional practice

CATALYST FOR CHANGE(early 1990’s)

3/29/2016

14

The Paradigm Shift

Outcomes-based Accreditation

3/29/2016

15

New Philosophy• Institutions and Programs define mission and

objectives to meet the needs of their constituents – enables program differentiation

• Emphasis on outcomes – preparation for professional practice

• Programs demonstrate how criteria and educational objectives are being met

• Focus on continuous improvement and sense of urgency

3/29/2016

16

ISO 9001:2008• ABET is committed to total quality

management in is own operations and has obtained ISO 9001:2008 certification. A third party auditor has verified compliance with the criteria.– A focus on the customer– Organization-wide continuous improvement– Documented critical processes– Management commitment to a QMS.

3/29/2016

Quality Management System: ISO 9000:2008

17

Measurement, Analysis &

Improvement

ResourceAllocation

ManagementResponsibility

Product / ServiceRealizationRqmts Product /

Service


ImprovementResourcePlanning


Satisfaction

Customer Customer

Continual improvement of quality management system

3/29/2016

18

ABET ValueStudents and Parents

• Helps students select quality programs• Shows institution is committed to improving the

educational experience• Helps students prepare

to enter “the profession”• Enhances employment

opportunities• Establishes eligibility for

financial aid and scholarships

3/29/2016

19

ABET ValueInstitutions

• “Third-party” confirmation of quality of programs

• Prestige, recognition by “the profession”

• Attract the strongest students• Acceptability of transfer credits• Some external funding depends

on accreditation status

3/29/2016

20

ABET ValueFaculty

• Encourages “best practices” in education

• Structured mechanisms for self-improvement

• Institution is serious and committed to improving quality– Facilities, financial resources,

training, etc.

3/29/2016

21

ABET ValueIndustry

• Ensures educational requirements to enter “the profession” are met

• Aids industry in recruiting– Ensures “baseline” of

educational experience • Enhances mobility• Opportunity to help guide

the educational process– Program’s industrial advisory groups– Professional, technical societies

3/29/2016

Questions?

23

Readiness Review• Required of all programs at institutions with no prior ABET

experience.• Based on the Self-Study Report (SSR)and transcript of

program graduate.• Request for Readiness Review (RREv) due Oct 1• SSR +1 transcript per program due Nov 1• Reviewed by ABET HQ Accreditation staff, members of the

Commission ExCom or designees. (Nov – early Feb)• Provide recommendation to:

– Submit the RFE in the immediate upcoming accreditation review cycle, addressing the REv suggestions, if any;

– Postpone the RFE submission unless substantive changes in the Self-Study preparation and documentation are made; or

– Not submit the RFE in the immediate upcoming accreditation review cycle because it is likely to be rejected.

3/29/2016

Context for Evaluation• The Program Evaluator will perform an initial

evaluation BEFORE arriving on campus – Evaluation centers on the evidence provided that

supports achievement of each of the criterion– The SSR will be the primary evidence used in this

initial evaluation.• The SSR provides the first impression of program to the

PEV and the only impression for the Readiness Review.• The Program Evaluator will make adjustments to

his/her evaluation during the campus visit– Interviews, display materials and tours will provide

the additional evidence.

3/29/2016 24

25

The Accreditation Timeline

JanuaryInstitution requests

accreditation for engineering programs

February - MayInstitution preparesself-evaluation (Program Self-Study Report)

May - JulyTeam chairs assigned,dates set, team memberschosen

September - DecemberVisits take place, draft statements written and finalized following7-day response period

January - FebruaryDraft statements editedand preliminary statements sent to institutions

March - AprilInstitutions respond to draft statement and return to ABET w/i 30 days

May - JuneNecessary changes,if any, are made

JulyEAC meets to takefinal action

AugustInstitutions notifiedof this action

Year 1 Year 2

3/29/2016

Terminology: Strengths and Shortcomings

• Strength – stands above the norm• Concern – program currently satisfies criterion, policy, or procedure,

however potential exists for the situation to change such that the criterion, policy, procedure may not be satisfied– Working definition: criterion, policy, or procedure is fully met, but there is

potential for non-compliance in the near future (duration of accreditation)• Weakness – program lacks strength of compliance with criterion,

policy, or procedure– Working definition: policy, or procedure is met to some meaningful extent,

but compliance is insufficient to fully satisfy requirements• Deficiency – program does NOT satisfy the criterion, policy, or

procedure– Working definition: assigned to any criterion, policy, or procedure that is

totally or largely unmet

3/29/2016 26

Shortcomings vs. Accreditation Action for a General Review

ShortcomingResults of Evaluation

WeaknessNo Yes Yes ----

DeficiencyNo No No Yes

Type of Review Possible Actions

General NGR IR IV SCFollowing a SC

NGR IR IV SC or NA27

3/29/2016

Who is a Program Evaluator (PEV)?

• A volunteer (one of more than 2,200 dedicated technical professionals from academia, industry, and government)

• A member of one or more ABET member societies– IIE for Industrial Engineering– IEEE for Electrical Engineering

• May have academic or industry background• Selected by the member society to represent ABET on

program evaluations• Provide knowledge concerning professional practice,

professional preparation, and continuous improvement.• Work with a team of colleagues from other professional

societies to evaluate the requested programs at an institution• ABET experience may vary, but has extensive training

conducted by ABET and is evaluated after each visit using the ABET PEV Competency Model.

3/29/2016 28

http://www.abet.org/network-of-experts/expand-your-network/

http://www.abet.org/network-of-experts/expand-your-network/

How are PEVs Selected?• Using a Competency Model

– Technically Current– Effective Communicator– Professional– Interpersonally Skilled– Team-Oriented– Organized

• Assigned to visit team by member society; accepted by Team Chair and institution.

3/29/2016 29

Using a rubric found on ABET website (www.abet.org)

30

Who is the Team Chair?• A volunteer.• A member of the Engineering Accreditation Commission

(EAC) as a representative of an ABET member society• Nominated by the member society to represent ABET on

the EAC using a Team Chair Competency Model; approved by EAC and appointed by the Board of Delegates.

• Assigned to visits by the Engineering Accreditation Commission Executive Team.

• Provide knowledge concerning professional practice, professional preparation, and continuous improvement.

• Lead a team of colleagues from appropriate professional societies to evaluate the requested programs at an institution

• ABET experience may vary, but has extensive training conducted by ABET and is evaluated after each visit using the ABET Team Chair Competency Model.3/29/2016

Questions?

32

Team Chair Responsibilities• Coordinate logistics for the visit with the

institution.– Schedule– Lodging, meals

• Prepare and coordinate the Program Evaluators for the visit

• Lead team meetings before, during and after the visit to arrive at consensus decisions.

• Meet with institution leaders during the visit:– President, Provost, Registrar, CFO– Others as needed

3/29/2016

33

PEV Responsibilities Before the Visit

• PEV will complete initial evaluation compared to criteria:– Review the SSR– Complete transcript analysis– Review additional material provided by the

program• PEV will complete Pre-Visit Forms• PEV will draft a plan for the visit

3/29/2016

34

PEV Responsibilities Before the Visit: Review Self-Study

• Corrective actions taken after previous visit.

• All graduates meet graduation requirements

• Students meet minimum accreditation requirements.

• Students took all courses in the proper order (prerequisites/co-requisites).

• Identify potential program strengths and shortcomings compared to the accreditation policies, procedures, and criteria.

3/29/2016

35

PEV Responsibilities Before the Visit: Transcript Analysis

• ABET recognizes transcripts as the official record of student coursework

• The Program Evaluator will look for:– Does the transcript identify the name of the

degree in a way that clearly identifies the program as an EAC of ABET accredited program?

– Are courses counted toward the degree consistent with the published requirements of the program?

– Are prerequisites taken before each course that requires them?

– Are the number of transfer credits and number of course substitutions excessive?

3/29/2016

36

PEV Responsibilities Before the Visit: Review Additional Material• What material?

– Catalogs and other publications provided by the institution

– University, College, and program websites• The Program Evaluator will look for:

– Additional information not provided in the SSR– Consistency with the information provided in

the SSR– How the institution and program present

themselves to the public

3/29/2016

37

PEV Responsibilities During Visit: Examine Assessment Materials

• Program Evaluator will:– Verify the assessment plan for Student

Outcomes as described in the SSR– Evaluate assessment processes:

• Are assessment processes adequate to determine attainment of the Criteria?

• Are assessment processes robust enough to identify program shortcomings?

• Are assessment processes sustainable?• Will the assessment process lead to program

improvement?3/29/2016

38

PEV Responsibilities During Visit: Interviews

• Program Head– Program leadership– Program strengths and continuous improvement

• Faculty Members– Particular course questions– Teaching philosophy and activities to maintain

currency– Student advising and interaction– Role in the assessment process – Role in preparation for the visit– Quality and maintenance of facilities– Professional development– Institutional support3/29/2016

39

PEV Responsibilities During Visit: Interviews

• Students– Level of satisfaction/enthusiasm for the

program– Curriculum & course quality– Advising – academic and career– Adequacy of facilities

• Staff– Level of satisfaction/enthusiasm for program– Professional development– Adequacy of resources

3/29/2016

40

PEV Responsibilities During Visit: Review Facilities

• Labs/Design Studios focused on undergraduate curriculum.– Sufficient number and size– Appropriate coverage across the breadth of specializations

within the program– Appropriate equipment, in good repair– Appropriate student access outside scheduled lab times.– Appropriate technical and instructional support– Safe physical arrangement and appropriate safety practices

in place

3/29/2016

41


• Classrooms– Appropriate physical arrangement, support for educational

technologies, etc.– Not overcrowded

• Faculty offices– Sufficient size, privacy– Access to computing resources

3/29/2016

42


• Support facilities– Computing resources

• Sufficient number & access• Software• Support personnel

– Appropriate spaces for students to gather (not an explicit criterion but relates to several criteria)

– Appropriate shop with parts, repair facilities, etc. (as appropriate to the discipline)

• Plan for on-going facilities maintenance, repair and upgrade – PROCESS in place to ensure facilities remain up-to-date, support

PEO’s and are safe

3/29/2016

What is a Process?The ABET Criteria include the word process in several locations and

implies it in several others. A process is a series of sequenced activities that convert inputs

(materials, information/data, people, machines/equipment) into outputs to satisfy customer requirement/need.

43

ProcessMaterial Outputs

Customers

Info/Data

People

Machines/ Equipment

Therefore, whenever you see Process in the criteria, you need to specify the activities, activity sequence & timing and roles &

responsibilities that make up that process.

3/29/2016

44

PEV Responsibilities During Visit: Visit Support Areas

• Library• Adequate resources for faculty & students• Adequate hours

• Supporting departments (Mathematics, Physics, Chemistry, English, etc.)

• Advisors• Career support center• Cooperative education/Internship office• Etc.

The Team will share responsibilities for visiting support areas

3/29/2016

45

Typical Visit Agenda• Sunday

– Initial team meeting– Visit campus to evaluate materials and tour facilities– Team meeting

• Monday– Team meeting with President/Dean– Meet with program head, faculty, students, and staff– Visit supporting areas– Team meeting– Draft Exit Statement

3/29/2016

46

Typical Visit Agenda• Tuesday

– Complete interviews, facility tours, and material review– Debrief program head and Dean on strengths and

shortcomings– Team meeting to finalize evaluation

• Complete Visit Report and Exit Statement• Team review and preliminary recommendation

– Exit Meeting with President, Provost, Dean and designated guests.

• Each PEV will read statement of findings for their assigned program.

• Team will leave behind a Program Audit Form (PAF) summarizing findings for each program evaluated.

3/29/2016

47

Responsibilities After the Visit

• Team Chair will combine exit statements for each program into one Draft Statement.

• Team will complete online Team Chair and Peer PEV Performance Appraisal Forms

• Institution representatives requested to complete online Team Chair and PEV Performance Appraisal Forms.

• PEVs will review Draft Statement written by Team Chair.• PEVs will review Due Process materials provided by

institution and consult with Team Chair on Final Statement.

• PEVs will consult with Team Chair on recommended accreditation action.

3/29/2016

48

How Can You Help the PEV?• Provide clear, concise, consistent responses to Self-

Study questions; Use current version of the template.– Quality Not Quantity helps the PEV identify

appropriate evidence; use tables/graphs where appropriate

– ANSWER THE QUESTIONS!• Provide supporting documentation for each transcript

– Include Registrar accepted degree audit form/checklist for each transcript with description of waivers, substitutions, transfers

– Provide copies of earlier curricula and pre-requisite flowcharts if applicable to transcripts

– Do NOT include transcripts in body of SSR.• Organize / label display materials so it is easy to

locate materials• Support PEV and Team agenda / schedule3/29/2016

Questions?

EAC Criteria & How to Address Common Issues

Criteria are Quality Management System

51


Improvement

ResourceAllocation


Product / ServiceRealizationRqmts Product /

Service


ImprovementResourcePlanning


Satisfaction

Customer Customer

Continual improvement of quality management system

3/29/2016

52

Changes• Changes can occur before your visit in the

following documents:– Accreditation Policy & Procedures Manual– Self-Study Questionnaire– Criteria, including Program Criterion and Definitions– Interpretations

• Changes are posted on the ABET website and sent via ENEWS.

• Dean should attend Institutional Rep training and Luncheon at the July Commission Meeting preceding the visit.

3/29/2016

53

Self-Study Questionnaire• Self-Study Questionnaire

– Follow the template format as much as possible and include ALL tables in the template.

– Remove instructions from the document– May include additional tables and/or graphs as

needed to best document how the program meets the criteria.

• Tables/graphs often summarize information in less space (“Picture is worth a thousand words.”)

– Questions focus on describing processes:• Sequence of steps• Timing• Responsibility

3/29/2016

54

Criterion 1: Students• Admission, acceptance of credits from other

institutions• Advising regarding curricular & career

matters– Have and enforce registration procedures

regarding pre- and co-requisites, course substitutions

• Have and enforce procedures to ensure all graduates meet graduation requirements.– Degree audit– Supporting documentation on file

3/29/2016

55

Criterion 2The program must have:• Published PEO’s consistent with

mission, needs of constituents and the criteria.

• A documented and effective process, involving constituents, for the periodic review and revision of the PEO’s

3/29/2016

56

PEO Issues

• Contain Student Outcomes language• Focus on the program and not graduates.• Frequently too many• Language imprecise, e.g.,

– ‘are capable of’– ‘are equipped with’– ‘have the attitude and —’– ‘have good or a solid understanding of’

• Large number of constituents, many not involved in establishing the PEO’s, nor in subsequent reviews and revisions.

• ‘what graduates are expected to attain’ is much broader than ‘career and professional accomplishments’

3/29/2016

57

PEO Highlights1. The process needs to document and

demonstrate that the PEO’s are based on constituent needs which were determined by involving them in some manner.

List the needs and show linkage to PEOs2. They are also to be reviewed and revised as

needed.3. If you survey the alumni in order to capture

information about your graduates, you can potentially use the results for a continuous improvement action.

3/29/2016

58

Sample PEO Process Involving Constituents*

3/29/2016

Input Method Schedule Constituent Alumni survey Every three years Alumni 2-5 years out

Employer focus group Every two years during Career Fair

Employers (and recruiters); some are alumni

Senior exit interview Annually Students; retrospective discussion of PEOs and

their intended career paths Advisory Council discussions As needed—available

annually Industrial representatives,

employers, alumni Curriculum Committee

meetings Available as frequently as

needed Faculty and students

*From Upper State University mock self-study, ABET PEV training, 2011.

Not specific- may raise questions

59

Criterion 2 & 4 Language• Even though programs are no longer

required to assess and evaluate their PEO’s, they must still conform to the PEO definition and not appear to be Outcomes!

3/29/2016

60

PEO Issues• Do the published PEO’s meet the definition?

• Are they really broad statements that describe what the graduates are expected to attain within a few years?

• Can the program convince the team that the PEO’s are consistent with constituent needs?

• There is NO language that insists on constituent approval, however there must be involvement!

• Is there a documented and effective process, involving program constituencies, for the periodic review and revision of PEOs?

3/29/2016

61

Scenario A: Are these PEOs?Are they really broad statements that describe what graduates are expected to attain within a few years of graduation?Graduates of the program will have:• A solid understanding of the basic principles of

mathematics, science, and engineering and the technical competency to use the techniques, skills and modern tools for practice in engineering as well as for graduate education.

• The ability to work in a team and develop problem-solving skills that include oral and written communication skills to effectively communicate technical and professional information.

3/29/2016

No, they are not really PEOs, but rather reworded student outcomes

62

Scenario B: Are these PEOs?Are they really broad statements that describe what graduates are expected to attain within a few years of graduation?Graduates of the culinary engineering program are

expected within a few years of graduation to have:1. Established themselves as practicing professionals or

be engaged in advanced study in culinary engineering or a related area.

2. Demonstrated their ability to work successfully as a member of a professional team and function effectively as responsible professionals.

3/29/2016

Yes, they describe what graduates are expected to attain a few years after graduation

63

Criterion 2 FAQ’s• What if the PEO’s really sound like outcomes (instead of

objectives)?– If PEO’s are not PEO’s, there will be a Criterion 2

shortcoming. • What if PEO’s are ambiguous or reflect outcomes retooled

to apply after graduation?– Becomes a team judgment – do they meet the intent

of the Criterion?• What if there is no process for determining the needs of

the program’s constituents?– If the PEOs do not incorporate constituents’ needs,

there will be a Criterion 2 shortcoming.

3/29/2016

64

Criterion 3-Outcomes DefinitionsCurrent Definition: Student Outcomes describe

what students are expected to know and be able to do by the time of graduation. These relate to the skills, knowledge, and behaviors that students acquire as they progress through the program.

3/29/2016

65

Criterion 3: Student Outcomes

• The program must have documented student outcomes that prepare graduates to attain the program educational objectives.

3/29/2016

66

Criterion 3: Student Outcomes

• Student outcomes are defined as (a) – (k) for engineering plus any additional ones articulated by the program

• The program must demonstrate that the engineering criteria (a) – (k) are attained to some extent.– The assessment and evaluation process that periodically

documents and demonstrates the degree to which outcomes are attained is in Criterion 4.

• Student outcomes must foster attainment of the PEOs– Must describe the relationship between SOs and

PEOs in the SSR.2016-17 Student Outcomes

3/29/2016

67

Criterion 3:Student Outcomes

• The definition of student outcomes are (a) – (k) plus locally articulated ones– Some programs don’t have their student outcomes expressed as (a) – (k).

They may have identified their own set of outcomes. As long as the program demonstrates coverage of all elements of (a) – (k) in its own outcomes, this part of the criterion is met.

– If additional outcomes beyond (a)-(k) are identified, they MUST be assessed (Criterion 4)

– Assessment and evaluation of Student Outcomes is in Criterion 4.

Changes to Criterion3, in conjunction with changes to Criterion 5 are out for public comment.

• Reduces number of required Student Outcomes to 6, covering 5 categories.

• Earliest implementation, if approved, would be 2017-18 cycle, with a possible phase-in period.

• Check ABET website (www.abet.org) periodically for updates.

• Proposed Student Outcomes

3/29/2016

http://www.abet.org/

68

Criterion 4: Continuous Improvement

• The program must regularly use appropriate, documented processes for evaluating the extent to which the student outcomes are being attained. The results of these evaluations must be utilized as input for the continuous improvement of the program. Other available information may also be used to assist in the continuous improvement of the program.

3/29/2016

69

Criterion 4 Components• Criterion 4 essentially contains two

components: 1. Process(es) for assessment and evaluation

of the extent of attainment of each of the Student Outcomes, and

2. Actions taken to improve the program, regardless of how information/data obtained

This is a closed loop Corrective Action process.3/29/2016

70

Assessment

ABET defines effective assessment as:

“Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome being measured. Appropriate sampling methods may be used as part of an assessment process.”

3/29/2016

71

Criterion 4: Continuous Improvement

• The process of assessment and evaluation needs to demonstrate the degree to which student outcomes are attained, however …– There is NO language that says all student outcomes

must be attained to the same degree or be measured on a numerical scale

– There is NO language that says assessment must be done in every course, every student or every semester.

• Many of the student outcomes contain multiple aspects that may not be possible to assess with one instrument. Be sure to define each aspect and assess accordingly. (see example for SO (e) on upcoming slide).

3/29/2016

72

Student Outcomes Assessment: SSQ Text

1. A listing and description of the assessment processes used to gather the data upon which the evaluation of each student outcome is based. Examples of data collection processes: specific exam questions, student portfolios, internally developed assessment exams, senior project presentations, nationally-normed exams, oral exams, focus groups, industrial advisory committee meetings, or other processes that are relevant and appropriate to the program

2. The frequency with which these assessment processes are carried out

3. The expected level of attainment for each of the student outcomes3/29/2016

73

Student Outcomes Assessment: SSQ Text

4. Summaries of the results of the evaluation process and an analysis illustrating the extent to which each of the student outcomes is attained

5. How the results are documented and maintained

3/29/2016

74

Student Outcomes Assessment

• What is adequate data? – Does it all have to be objective/direct? (NO)– Can it be subjective? (Some of it may be;

nothing says it cannot)– Is the observation or conclusion of course

instructor adequate? (What was his or her basis for the observation?)

– Does evidence for each student outcome have to be in the form of work the student has produced? (No, however, the PEV & ultimately the team, needs to be convinced that outcome attainment has been demonstrated.)

3/29/2016

75

Student Outcome Assessment Issues

• Excessive number of student outcomes supported in a single course– All 11 in the major design experience is not

credible or sustainable• Course grades used as basis for assessment • Design of Experiments (Outcome b)

– Students never actually ‘design’ an experiment and then ‘run’ it to see if the design worked.

• Confusing course assessment with outcome assessment.

3/29/2016

76

Simple Student Outcome Assessment Process

• Major design experience for engineering programs:– a- ability to apply knowledge of math, science and

engineering– c- design a system, component, process– d- multi-disciplinary teams– e- formulate & solve engineering problems– g- communicate

• FE Exam for f – ethics • 5 or more outcomes remain that need to be

addressed3/29/2016

77

Sample SO Assessment Process*


3/29/2016

78

Sample SO Assessment Frequency*

3/29/2016


Student Outcome 2005 2006 2007 2008 2009 2010 a. an ability to identify, formulate, and solve

engineering problems X X

b. an ability to apply knowledge of mathematics, science, and engineering X X

c. an ability to use the techniques, skills, and modern engineering tools necessary for engineering practice.

X X

d. an ability to design and conduct experiments, as well as to analyze and interpret data X X

e. an ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

X X

f. an ability to function on multi-disciplinary teams X X

g. an understanding of professional and ethical responsibility X X

h. an ability to communicate effectively, both orally and in writing X X

i. the broad education necessary to understand the impact of engineering solutions in a global, economic, environmental, and societal context

X X

j. a recognition of the need for, and an ability to engage in life-long learning X X

k. a knowledge of contemporary issues X X l. a willingness to assume leadership roles and

responsibilities X X

Sample Assessment for Student Outcome e*

(an ability to identify, formulate and solve engineering problems)

Performance Indicators

Method(s) of Assessment

Where data are collected (summative)

Length of assessment cycle (yrs)

Year(s) of data

collectionTarget for

Performance

1) Problem statement shows understanding of the problem

Faculty assessment

of design problem

statement

EGR 40903 years 2007, 2010 90%

Senior Survey

On-line survey

2) Solution procedure and methods are defined.

Faculty assessment

of senior project plan

EGR 40903 years 2007, 2010 85%

Senior Survey

On-line survey

3) Problem solution is appropriate and within reasonable constraints

Faculty assessment

of senior design

solution

EGR 40903 years 2007, 2010 80%

Senior Survey

On-line survey 79

*From Upper State University mock self-study, ABET PEV training, 2011.3/29/2016

80

Sample Assessment Analysis & Evaluation for Student Outcome e*(an ability to identify, formulate and solve engineering

problems)

3/29/2016

Assessment Results (direct measures) 2005: For the summative assessment (end of program), the decision was made to focus on the faculty’s direct assessment for all indicators.


81

Continuous ImprovementCommon Issues

• Linkages between assessment and CI actions not documented.

• Loop not closed between assessment and actions taken to improve the program.– For every student outcome not attained,

a corresponding action should be identified (even if still in-progress)

– Sense of urgency lacking3/29/2016

82

Continuous ImprovementSample Table

When (AY)

Program Element Action

Why (Assess-

ment)Timing

Who Respons-

ibleResult

2012-13

Curriculum: ISE124: Introduction to Industrial and Systems Engineering

Focused on reading

comprehension and

assignment completion.

Increased the weight of the grade to help motivate and reward the students.

Direct Measure

not attained

Fall 2013 course offering

J. Smith, course

Coordinator

Direct Measure

improved by 20%; above

target for attainment

3/29/2016

Criterion 4 Continuous Improvement Closed Loop Process Control

3/29/2016 83

Suppliers Process Customers

Process Management

and Improvement

SupplierMeasures

CustomerFeedback

InputMeasures Output

MeasuresProcess

MeasuresProcessChanges

High SchoolsOther Programs at InstitutionOther Institutions

Student Performance on entrance exams

EmployersAlumniGraduate Programs Institution

Program Curriculum

Outcomes

Objectives

Course & Outcome Assessment/CI

Facilities, Faculty, Resources

Student Monitoring & Advising

84

Criterion 5:Curriculum2 Elements

1. Professional Component:a) 1 year combination of college level

mathematics and basic science (some with experimental experience) appropriate to the discipline.

Proposed Criterion 5 Definitionsb)1.5 years of engineering topics, consisting of

engineering sciences and engineering design appropriate to the field of study.

c) General education component that complements the technical content of the curriculum and is consistent with program and institution objectives.3/29/2016

85

Criterion 5:Curriculum2 Elements

2. Curriculum culminates in a major design experience based on the:

a) knowledge and skills acquired in earlier course work, and

b) incorporates appropriate engineering standards and realistic constraints.

Changes to Criterion 5, in conjunction with changes to Criterion 3 have been proposed and are out for public comment.

3/29/2016

86

Criterion 5:Curriculum Common Issues

• Split of an Engineering Course between Math/Basic Science and Engineering Topics categories

• Major Design Experience– Must be based on knowledge and skills acquired in earlier

coursework– And, incorporate engineering standards and multiple

realistic constraints (project reports should identify them and show use)

• Common courses across programs (i.e. statics, dynamics, circuits, engineering economy) categorized differently by different programs.

• Project management and computer programming courses categorized as engineering science or design – Should be categorized as ‘Other’

3/29/2016

87

• Suggestion: Include table for projects associated with summited transcripts: (ME sample)

• OR, include summary table in SSR for all projects completed previous course cycle.

• OR, ask students to include similar table in their project report.

Student #1 #2 #3-6

Project Title & Area Thermal Systems Mechanical Systems

Constraints Economic Environmental Sustainability Manufacturability Ethical Health and Safety Social Political Other

Standards ? ?

Standards & Constraints

3/29/2016

88

Criterion 6: Faculty • Sufficient number to:

– achieve program educational objectives and student outcomes,– deliver curriculum for students to graduate in a timely manner – achieve adequate levels of student-faculty interaction,– provide student advising and counseling, – Include university service, – provide time for professional development, and– interact with industrial and professional partners.

• Make sure description, Tables 6-1 and 6-2 AND faculty Vitae (in Appendix B) are consistent.

• Competent to cover all curricular areas of program.– Include a table of faculty by curricular area.

• Authority for creation, delivery, evaluation, modification and continuous improvement of the program.– Should align with the description of the CI process

3/29/2016

89

Criterion 7: Facilities• Adequate to support educational

objectives and student outcomes of the program.

• Fosters faculty-student interaction• Encourages professional development &

professional activities, and • Provides opportunities to use modern

engineering tools.• APPM II.G. 6.b.(1): Safe

3/29/2016

90

Criterion 8: Support• Sufficient to attract, retain, and provide for

continued professional development of faculty.• Sufficient to acquire, maintain, and operate

facilities & equipment appropriate for the program.

• Constructive leadership• Consider adding a table demonstrating budget

stability since previous evaluation/initiation of program. (table no longer required in SSQ)

3/29/2016

91

Program Criteria

• Program Criteria for almost all programs have 2 elements:1. Curriculum2. Faculty

Note: this is no longer identified as Criterion 9.

3/29/2016

92

Program Criteria Curriculum Aspects

• If add as Student Outcomes, MUST assess. – Sage advice: do NOT convert program criteria into

SOs.• Simply demonstrate how addressed in the

curriculum by providing specific examples.• May be impacted by proposed changes to

Criterion 3 and Criterion 5.

3/29/2016

93

Program Criteria: Key Curriculum Elements

• IE: integrated systems; analytical, computational, and experimental practices.

• EE: analyze & design complex electrical and electronic devices, software, and systems containing hardware and software components

• ME: work professionally in either thermal or mechanical systems

3/29/2016

94

Program Criteria: Key Faculty Elements

• IE: understand professional practice and maintain currency in their respective professional areas

• EE: none• ME: maintain currency in their

specialty area

3/29/2016

95

APPM Requirements• II.A.1 – represent the accreditation status of each program

accurately and without ambiguity.• II.A.6 – Each accredited program must be specifically

identified as “accredited by the _____ Accreditation Commission of ABET, http//www.abet.org.”

• II.A.6.a – Each ABET accredited program must publically state the program’s educational objectives and student outcomes.

• II.A.6.b - Each ABET accredited program must publically post annual student enrollment and graduation data per program.

• II.G.6.b – Examine facilities – to assure the instructional and learning environments are adequate and are safe for the intended purposes.

3/29/2016

What Questions Do You Have?

97

Appendix

3/29/2016

99

Student Outcomes• Engineering programs must demonstrate that their graduates have:

a) An ability to apply knowledge of mathematics, science and engineering appropriate to the discipline

b) An ability to design and conduct experiments, analyze and interpret datac) An ability to design a system, component, or process to meet desired

needsd) An ability to function on multi-disciplinary teamse) An ability to identify, formulate, and solve engineering problemsf) An understanding of professional and ethical responsibilityg) An ability to communicate effectivelyh) The broad education necessary to understand the impact of engineering

solutions in a societal contexti) A recognition of the need for, and an ability to engage in life-long learningj) A knowledge of contemporary issuesk) An ability to use the techniques, skills, and modern engineering tools

necessary for engineering practice3/29/2016

100

Proposed Student Outcomes

1) An ability to identify, formulate, and solve engineering problems by applying principles of engineering, science, and mathematics.

2) An ability to apply both analysis and synthesis in the engineering design process, resulting in designs that meet desired needs.

3) An ability to develop and conduct appropriate experimentation, analyze and

4) interpret data, and use engineering judgment to draw conclusions.5) An ability to communicate effectively with a range of audiences. 6) An ability to recognize ethical and professional responsibilities in

engineering situations and make informed judgments, which must consider the impact of engineering solutions in global, economic, environmental, and societal contexts.

7) An ability to recognize the ongoing need for additional knowledge and locate, evaluate, integrate, and apply this knowledge appropriately.

8) An ability to function effectively on teams that establish goals, plan tasks, meet deadlines, and analyze risk and uncertainty

3/29/2016

101

Proposed Criterion 5 Definitions• College-level Mathematics consists of mathematics above pre-

calculus level.• Basic Sciences consist of chemistry and physics, and other

biological, chemical, and physical sciences, including astronomy, biology, climatology, ecology, geology, meteorology, and oceanography.

• Engineering Science is based on mathematics and basic sciences but carry knowledge further toward creative application needed to solve engineering problems.

• Engineering Design is the process of devising a system, component, or process to meet desired needs, specifications, codes, and standards within constraints such as health and safety, cost, ethics, policy, sustainability, constructability, and manufacturability. It is an iterative, creative, decision-making process in which the basic sciences, mathematics, and the engineering sciences are applied to convert resources optimally into solutions.

• Teams consist of more than one person working toward a common goal and may include individuals of diverse backgrounds, skills, and perspectives.

3/29/2016

Preparing for ABET EAC Evaluation Visit r032916

Documents

Transcript of Preparing for ABET EAC Evaluation Visit r032916