GAIL WISANUNIVERSITY DIRECTOR OF ASSESSMENT

OFFICE OF INSTITUTIONAL EFFECTIVENESS AND ANALYSISGWISAN@FAU.EDU

X71006 (561) 297-1006

MARC RHORERASSISTANT DEAN

COLLEGE OF BUSINESSMRHORER@FAU.EDU

X70210

SEPTEMBER 2010

Assessment in ITOM and Beyond: Defining, Assessing, and Documenting

Student Learning Outcomes

Overview :

Describe the Nature and Purposes of Assessment

Relate Assessment in ITOM to Larger Context

Explain the uses of a Curriculum Matrix and how it relates to the Academic Learning Compact (ALC)

Writing better learning outcomes

Assessing and Documenting Student Learning Outcomes

Close the Assessment Loop: Identify the Do’s and Don’ts of how you will USE your assessment results

Assessment in FAU ITOM Programs: The Context

1. National: Department of Education and Public Opinion

2. Regional Context: SACS Accreditation3. Florida Context: Academic Learning Compacts

with required annual reports to the state BOG on continuous learning improvement

4. Institutional Context: Strategic Plan 5. The College of Business Context:o Core Curriculumo Business programs are accredited by

The Association to Advance Collegiate Schools of Business (AACSB International)

Assessment in FAU ITOM Programs: The SACS Accreditor Context

The Commission on Colleges of the Southern Association of Colleges and Schools (SACS) is the recognized regional accrediting body in the eleven U.S. Southern states (Alabama, Florida, Georgia, Kentucky, Louisiana, Mississippi, North Carolina, South Carolina, Tennessee, Texas and Virginia) and in Latin America for those institutions of higher education that award associate, baccalaureate, master's or doctoral degrees. It is charged with carrying out the accreditation process.

SACS Accreditation (2012-2013): Need 3 years of good data on learning outcomes assessment and continuous improvement.

o 2009-2010o 2010-2011o 2011-2012

Assessment in FAU ITOM Programs: The SACS Accreditor Context

1. Accreditation standards requires all institutions of higher learning to establish college level learning goals

2. Assess student achievement against those goals

3. Provide evidence of student achievement

4. Use results of assessment to improve student learning outcomes

Assessment in ITOM and College of Business:

The AACSB Context

Accreditation standards requires that:

“The school specifies learning goals and demonstrates achievement of learning goals for key management-specific, and/or appropriate discipline-specific knowledge and skills that its student achieve in each undergraduate degree program.”

Assessment in ITOM and College of Business:

The AACSB Context

Accreditation standards requires that:

“…the school specifies learning goals and demonstrates achievement of learning goals in each specialized master’s degree program.”

The AACSB Context

AACSB, at the undergraduate level, is concerned about “degree programs”

Academic Learning Compact (ALC) required by Florida for each degree program and could be aligned with AACSB /COB learning goals. (ALCs will have some unique content and skills learning goals for degree programs.)

The COB learning goals (AACSB) for the BBA/BS program are universal across all majors

The AACSB Context

BBA/BS Learning Goals (approved 2006) – our graduates will: Demonstrate functional knowledge specified in the

college core Understand the impact of global economic and

multicultural issues as they related to business Demonstrate the ability to utilize computing or

telecommunications technologies in business practice and making business decisions

Demonstrate communication skills appropriate for a business professional

The AACSB Context

BBA/BS Learning Goals (approved 2006) – our graduates will: Demonstrate analytical critical thinking Understand ethical and legal implications of business

decisions and practices Understand managerial and legal aspects of

conducting business in a diverse environment

The AACSB Context

Perhaps FAU COB learning goals need slight revision or combining for conciseness – examples: 2 goals with similar/same skill of computer technology 2 goals with similar/same issue of understanding legal

aspects 2 goals with similar/same issue of multiculturalism and

diversity

Several learning goals (AACSB) overlap Academic Learning Compacts (BOG) – recommend simplification and reduce duplication of effort

The AACSB Context

Example of concise learning goals from the undergraduate degree program (Virginia Tech – Pamplin COB) Students will be effective communicators Students will be analytical problem solvers Students will be proficient with a wide range of

relevant business-related technology Students will have business skills that include an

international emphasis

The AACSB Context

AACSB emphasizes assurance of learning (AOL) now, as much as faculty sufficiency

Key areas of emphasis for AACSB in AOL Actively engaged participants Faculty involvement and understanding of the process Assessment of learning, not faculty methods or teaching Meaningful discussion of improvement among stakeholders,

NOT collection of data and the process of assessment methods Alignment of learning goals, what faculty teach, and what

students experience/learn Group work cannot be used for AOL assessment unless specific

contribution/authorship is attributed to each student

The AACSB Context

The essence of AACSB Assurance of Learning in 3 questions What are students supposed to learn in your program? How do you know that they are learning these? What can you do to improve learning?

If these cannot be answered, there will be problems with the reaffirmation of accreditation

The AACSB Context

Assurance of Learning research and Relationship to Faculty Sufficiency AACSB values and recognizes research in AOL Research in student learning in business programs

(AOL) is applicable and valued by AACSB Publications from assessment and AOL are applicable

to attainment / maintenance of AQ status for terminally qualified (PHD) faculty

Journal outlets / examples

What Is Student Learning Assessment?

Assessment is a process in the learning cycle in which faculty measure and document the degree to which students are attaining defined (should be valued and important) learning outcomes

Assessment is a student learning-centered and faculty-guided approach to improving student learning in the classroom and in degree programs

Defining Assessment:The Three Steps of Assessment

1. Articulate the program/course goals for student learning

Learning Outcomes: Clear Measurable Expected Outcomes When completing the program, students will be able to ….

2. Systematically gather evidence about whether goals are being met.

Where are the learning opportunities in the program? Measuring outcomes: gathering, analyzing, & interpreting

data

3. Use Information for Improving Student Learning Outcomes.

Defining Assessment:The Three Questions to Ask

1. What are the program student learning outcome goals? When completing the program, What should students know? What

should students be able to do? What skills should they have?

2. How will the program/course gather evidence about whether students have achieved the expected learning outcomes?

In what courses should students have learned the skill? How was the skill measured? Was it measured at different times in the program?

How will the program/class gathering, analyzing, & interpret results?

3. How will the program/course Use the Information gathered to Improving Student Learning Outcomes?

What Assessment Isn’t

Assessment is not just giving gradesAssessment is not an evaluation of

facultyAssessment is not an infringement of

academic freedomAssessment is not an invasion of

student privacyAssessment is not punitive

The Value of Assessment: What It Does Best…

“Much of the value of assessment comes from the systematic way it makes educators question, discuss, share and observe.”

Palomba and Banta (1999). Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass.

The Purposes of Assessment: Why Assess Student Learning?

To provide evidence of institutional effectivenessTo demonstrate the effectiveness of ITOM, College

of Business and FAU courses/programs to all stakeholders (internal & external)

To know your students’ weakness and strengths and use those results to improve overall quality of teaching and learning within your program

To provide critical feedback to students and facultyTo provide students with clear statements about

how they can expect to improve skills and abilities

Types of Assessment: Indirect

Learning is subtle, often supporting direct methodsGrades not based on explicit criteriaCourse evaluationsStudent hours spent on active learning and/or

service learningFocus group interviews with studentsEmployer or alumni surveysStudent perception surveysEmployer and internship supervisor ratings

Types of Assessment: Direct (Preferred)

Learning is tangible and visibleCourse assignmentsExaminationsClassroom Assessment Techniques (CATs)Term papers and reportsResearch projectsCase study analysisPortfolios – body of workCapstone projectsLicensure examsANALYTIC Rubrics Used to score

performance

Assess…Student achievement against those goals or outcomes

Use…Results…To improve teaching and learning The

Assessment Cycle

Provides Feedback

for Continuous Improveme

nt

Implement…

Provide Opportunities forstudents to achievethose goals

Plan…Establish Learning GoalsLearning Outcomes

So, The Cycle Continues

Why Aren’t Grades Sufficient?

Grades (A’s, B’s & C’s) in and of themselves do not give you enough information about students’ strengths and weaknesses

Increasing emphasis is now being put on students: thinking critically acquiring life-long learning and business professional

skills acquiring social values

Assessment can build on to the grading process, but grades alone do not provide the kind of feedback we want

What are key Student Learning Outcomes in FAU’s MIS B.B.A/B.S. Degrees and how are

they assessed ??

Critical Thinking in MIS Man. 4720: Students will demonstrate

analytical critical thinking through their ability to apply relevant business models and/or strategic concepts and tools in a research project on a firm and/or industry. (ALC)

Fin. 3403: Students will apply financial math and valuation principles to value securities, capital projects and other assets. Students will interpret financial information and use it in decision making. (ALC)

Some Key Student Learning Outcomes in ITOM Programs and their Assessment

Written Communication in ITOM

ENC 3213:Demonstrate Communication Skills appropriate to a professional (basic level) (ALC)

ISM 4133 Adv. Design & Anal.: ALC (more advanced )

Oral Communication in ITOM ISM 4133: The presenter effectively communicates the

organizations’ problem(s), solution(s), and methodologies used to obtain solution(s). (Outcome stated in rubric provided in ALC.)

Writing Better Learning Outcomes

DO DON’T•Better: Students will locate information, evaluating its validity and appropriateness for a project.

•Too Vague: Students will demonstrate information literacy skills. •Too Specific: Students will be able to use a specific institutional databases to demonstrate information literacy.

•Better: Students will: 1.) accurately describe and analyze two major management approaches , and 2.) evaluate each perspectives strengths and weaknesses.

•Too Vague: Students will demonstrate critical thinking skills. •Too Specific: Students will be able to describe the X management perspective.

•Better: Students will write clearly organized position papers, providing financial and spreadsheet support.

•Too Vague: Students will write proficiently. •Too Specific: Students will write a five page paper with no more than 3 grammatical errors.

Writing Better Learning Outcomes

DO DON’T•Better: Students will systematically analyze and solve problems, advocate and defend their views, and refute opposing views.

•Too Vague: Students will solve problems. •Too Specific: Students will find a solution to problem x.

•Better: Student teams will analyze and evaluate case studies about ethical business dilemmas.

•Too Vague: Students will demonstrate ethics. •Too Specific: Students will value the x point of view of ethical business??? (arguable?)

•Better: Students will be able to identify the major periods in 19th and 20th century European art, describing the art styles and their cultural context.

•Too Vague: Students will appreciate art. •Too Specific: Students will be able to identify 7 of 8 artists from their paintings.

What is a Rubric and Why Use One?

A Systematic Scoring Guideline for evaluating performance (e.g., case study analysis, research paper, business plan, book review, exam) via descriptions of criteria for different levels of achievement

Rubrics provide structure for consistent and unbiased grading

What are the 2 Main Types of Rubrics and their Use?

Holistic Rubrics provide a single score based upon multiple factors for each level of performance.

Analytic Rubrics provide multiple scores based upon distinct achievement criteria/factors from which they can then create a single score.

What are Holistic Rubrics?

Holistic Rubrics provide a single score for a whole performance (e.g., research paper or case study analysis is given one score) : Excellent, Good, Fair, Poor or

Exceeds standards, Meets standards, Below standards

Each Grading Level using MANY CRITERIA so you cannot identify specific strengths/weaknesses

Holistic Rubric’s Value and Purposes:

Easy to GradeQuick Picture of PerformanceSingle Dimension is Sufficient for GradingHelpful tool for calibrating grading

standards across multiple classesNot useful for improving student learning

outcomes standards across multiple classes

What are Analytic Rubrics? Why Use?

Analytic Rubric provide specific feedback on multiple criteria.

Value: Performance (e.g., research paper or case study analysis) is assigned many scores on key outcome criteria so that student and class outcomes can be assessed as to strengths and weaknesses.(See ITOM rubric in ALC.)

Assessment Grading uses MANY Specific CRITERIA which can then be used to calculate (automatically if desired) total grade.

Analytic Rubric’s Value and Purposes:

Detailed feedback to learners, professors, and programs

More detailed scoring can be used to guide improvement

Rubric, when provided to students, can help them identify key factors in their performance and thus guide achievement

Useful feedback to faculty and the ITOM Program on students’ strengths and weaknesses on specific criteria

Assessment results can be used to modify learning opportunities in the curriculum and improve the program and document continuous improvement cycle in assessment.

Association of American Colleges and Universities (AACU): VALUE Project

VALUE: Valid Assessment of Learning in Undergraduate Education

Valid data needed to guide planning, learning, teaching, and improvement

Good Practice in assessment requires multiple assessments over time http://www.aacu.org/value/rubrics

Developed Tested Rubrics Critical Thinking Problem Solving Written Communication Oral Communication Others

Keep It Simple

Utilize the same student work (artifact) for various assessments of a learning outcome

Example: A Case Study Analytic and company or industry

problem solving paper on improving profits

Multiple outcomes assessed using this paper Critical Thinking Written Communication Team Work Oral Communication

Curriculum Outcomes Matrix: What is it?

Curriculum mapping is a method to align instruction with desired goals and program outcomes. It can also be used to explore what is taught and how.

The map or matrix: Documents what is taught and when Reveals gaps in the curriculum Helps design an assessment plan

http://manoa.hawaii.edu/assessment/howto/mapping.htm

Curriculum Outcomes Matrix: Benefits

Why do it? BenefitsImproves communication among faculty Improves program coherence Increases the likelihood that students achieve

program-level outcomes Encourages reflective practice

http://manoa.hawaii.edu/assessment/howto/mapping.htm

Assessment In Multiple Courses

Introduce the learning outcome skill (I)

Reinforce the learning outcome skill (R)

Master the learning outcome skill (M)

Department matrix : More Benefits

Process more important than product

Thinking in skills, not only content knowledge

Tool for constructing a degree program building in redundancy

Plays to strengths of individual faculty Each person need not cover all skills in a single course

Community and Communication within department

Working with matrix Requires mutual intellectual respect Requires willingness to find out what colleagues are

doing and why Requires opening oneself up to similar scrutiny

e.g. “Why do you think students need to understand spreadsheet applications?”

Helps Further Develop Sharing, Discussion and Community within the Department

Additional Skills: Learning Outcomes for Matrix

Information Literacy skillsStatistical analysis skills (broken down in another

document to identify strengths and weaknesses)Oral Presentation skillProjection, Simulation and computer modeling

skillsSpecific language skillsMicrosoft office skillsProblem Solving skills: Analysis, evaluation,

creative solutions

Declarative Research Technical Written Oral TeamApply/Analyze/

EvaluateSolve problems/

Design/CreateMgmt & Organizational Behavior

MAN 3025 I, A* R

Marketing Management MAR 3023 I, A*

Financia l Management FIN 3403 R, A* R, A* R, A* R, A*

Writing for Management ENC 3213 I I, A* R R

Quantitative Methods in Admin

QMB 3600 I, A* R R, A* R, A*

Management Information Systems

ISM 3011 I, A* I I, A* R R I I

Bus iness Law I BUL 4421 R, A* R, A* R, A* R, A*

Operations Management MAN 3506 I, A* R, A* R, A* R, A* R, A*

Global Strategy and Pol icy MAN 4720 R, * R R R R R, A* R, A*

3 credi ts of Upper-Level Economics1 International Perspective CourseIntro to Computer Sys tems and Software Development

ISM 3230 I I I I I

Bus iness Data Communications

ISM 4220 I R

Database Management Systems

ISM 4212 R R R R

Advanced Systems Analys is & Des ign

ISM 4133 M, A*** A*** M, A*** M, A*** M, A***M, A*** M, A*** M, A***

Introduction to Bus iness Intel l igence

ISM 3116 I, A** I, A** I A** A**

Data Mining and Data Warehous ing

ISM 4117 I, A** R, A** R A** A**

Advanced Bus iness Intel l igence

ISM 4403 M R, A** R R R R M, A** M, A**

Information Systems Securi ty ISM 4320 A** A** A** A**

Securi ty Management ISM 4323 A** A** A** A**

Computer Forens ics ISM 4324 A** A** A** A** A** A**

A - assessment (* - ALC, ** - AP, *** - ALC & AP)

College of Bussiness and Info Tech & Op Management: Learning Outcomes and Curriculum Matrix

Name CourseContent Communication Critical thinking

Additional Goals

Colle

ge o

f B

usin

ess

Core

(B

BA

or

BS)

ITO

M C

ore

- M

anag

emen

t In

form

ation

Sys

tem

s

I - introduced; R - reinforced/practiced; M- mastered

The Outcomes Matrix and Assessment

Observations of student success and difficulty in the capstone course or other core courses

Use the skills/learning outcome categories on the matrix to frame specific questions to a class in exit interviews/surveys Formative assessment in various courses

How Can Assessment Results be Used to Improve Student Learning Outcomes for program graduates.

TimetableTimetable

When you will start collecting the assessment data?

How often you will collect it?

When will you plan to report on the results?

Reporting on the Results

Now, What Are You Going To Do With All These Good Data? Collaborate with your departmental colleagues on

the findings Determine how the results can be used to increase

students’ learning and success Put assessment results into action Re-assess to measure the effectiveness of your

course/program modifications

Explaining your use of results

DO DON’T•DO focus on making specific improvements based on faculty consensus.

•DON’T focus on simply planning for improvements or making improvements without faculty feedback.

•DO address specific program improvements that will impact student learning.

•DON’T address assessment improvement plans such as revising the rubric.

•DO use concrete ideas. •DON’T write vague ideas.

•DO state strategies that are sustainable and feasible.

•DON’T use strategies that are impossible to complete in one year considering your resources.

•DO use strategies that can improve the curriculum and also help improve student learning outcomes (e.g. solve problems).

•DON’T focus only on improving the curriculum.

Suggestions

Review and Revise Academic Learning CompactCoordinate Assessment Plans with ALCSet priorities as a department“Hand off” specific learning outcomes

assessment to course/professor best suited to task but share and use results

Consider using Analytic RubricsBe clear as to how assessment results will be

used to revise pedagogy, curriculum matrix, and teaching/learning strategies in program/courses

Conclusions?

Align Assessment Activities to make them UsefulDon’t duplicate work: Align the workCreate An Outcomes Curriculum MatrixAlign assessment for AACSB, ALC, and FAU

Assessment DatabaseDocument and Report ResultsIdentify Specific ways in which assessment results

will be sharedIdentify Specif ways to USE data to improve

learning

The Flow of the Assurance of Learning - AACSB

ASSESSMENT IS A TOOL TO BRING

About better teaching and learning!

www.fau.edu/iea/assessmentgwisan@fau.edu

Questions?

Appendix

ITOM ALC on remaining slides

B.B.A. & B.S. Management Information SystemsCollege of Business

Dep’t. of Information Technology And Operations Management

Rubric