Using Simulation, 360-degree Feedback, a Balanced ... 2016 Drexel 2016 Assessment Conference...
Transcript of Using Simulation, 360-degree Feedback, a Balanced ... 2016 Drexel 2016 Assessment Conference...
Using Simulation, 360-degree Feedback, a Balanced Scorecard, and AARs to Assess Individual and
Team Performance in Different Delivery Formats
© James. V. Caruso, 2016
Drexel 2016 Assessment Conference September 8, 2016
Presented by:Drexel University, LeBow College of Business
Jim Caruso, MBA, CPA, MHRD, CPLP
2
ObjectivesDuring this session, you will learn more about:• How individual, team, and business performance can be assessed and
developed using an integrated simulation experience, and the tools used• How business acumen and leadership content can be integrated and taught• What roles the instructor can perform, how to scaffold content, and how to
develop critical thinking given the different delivery formats• Ways to apply best practices on how to integrate simulation, multi-discipline
content, instructional design resources, and various assessment tools in your setting
After completing this session, you will be able to:• Use a 360-degree feedback tool (Capsim’s TeamMATE) to assess individual and
team performance in your courses• Use After Action Reviews (AARs) in your courses to enable individuals and
teams to analyze, reflect on, and improve their performance
© James. V. Caruso, 2016
3
Agenda1. Introduction2. Integrated and Experiential – Corporate Contextualization3. Case Example – Successfully using a Total Enterprise Simulation
with a Graduate Management Classa. Contextb. Business Acumen Contentc. Leadership / Management Contentd. Assessment Tools
4. Delivery formatsa. 11 weeks Face-to-Faceb. 4-day Residencyc. 11 weeks Online
5. Questions
© James. V. Caruso, 2016
2. Integrated and Experiential –
Corporate Contextualization
4© James. V. Caruso, 2016
5
Integrating Content, Simulation, & Resources• Content – prework; scaffolding; order of content and
simulation; flipped classroom• Company-specific information – terminology, models
• Finance / Strategy / Marketing / Sales• Production / Operations• Leadership / Management / Communication /
Presentation Skills
• Simulation – total enterprise level, scenario-based• Delivery mode
© James. V. Caruso, 2016
6
Integrating Content, Simulation, & Resources (cont’d)
• Project-based learning, after action review (AAR), reflection, gap analysis
• Roles - instruct / facilitate / coach / consult• Considerations - Create an overwhelming scenario /
discovery learning / ‘sink or swim’
• Critical thinking• Sensitivity analysis• Engagement - Accountability & Ownership
© James. V. Caruso, 2016
7
Integrating Content and Simulation Possibilities
• Finance - Total Enterprise Simulation – Level 1, 2, 3
• Business Acumen (finance, strategy, marketing) - Total
Enterprise Simulation – Level 2, 3
• Business Acumen and Leadership - Total Enterprise Simulation
• Level 3, 4, 5 (with executive coaching)
• Leadership – Scenario (Influence, Change, Conflict, Strategy)
• Project Leadership – Scenario
• Business Acumen and Project Leadership – Scenario
• Sales – Scenario
© James. V. Caruso, 2016
Total Enterprise Simulation – Complexity Levels
8
Level 1 (59) – single-player; individual or team play
Level 2 (17) – multi-player; smaller # of products/markets
Level 3 (73) – multi-player; larger # of products/markets• E.g., Capsim – Foundation; Capstone
Level 4 (23) – multi-player; sub-divided team & decisions; negotiated transactions w/ other teams; business ecosystem - make/buy/sell decisions
• Senior executives (Executive Education); EMBA
Level 5 (7) – Level 4 + customized; more complex negotiations; executive coaching
© James. V. Caruso, 2016
3. Case Example –
Successfully using a Total Enterprise Simulation with a
Graduate Management Class –
a. Context
9© James. V. Caruso, 2016
Core Intellectual and Practical Skills
Information Literacy; Communication; Critical Thinking; Self-Directed Learning; Ethical Reasoning; Technology Use
INFORMATION LITERACY - Possess the skills and knowledge to access, evaluate and use information effectively, competently, and creatively
Experiential and Applied Learning
LEADERSHIP - Develop a vision, translate that vision into shared goals, and effectively work with others to achieve these goals
Incorporates University’s Student Learning Priorities
© James. V. Caruso, 201618
Integrating Content, Simulation, & Resources (continuously linking)
• Graduate MBA class – Management
• Business Acumen – finance; accounting; strategy; marketing; operations
• Leadership / Management – team, conflict; feedback, communication; reflection
– TeamMATE - Team evaluation tool
• Total enterprise simulation – Capsim – Foundation– 2 Practice Rounds / 8 Competition Rounds
• Assignments
• Real-world examples
Managing the Total Enterprise
© James. V. Caruso, 201611
Total Enterprise Simulation - Characteristics
12
• Industry Overview
• Company
• Goals, strategies, & performance metrics
• Products / segments
• Key decisions – decision-making process
• Tradeoffs
• Sensitivity analysis
• Rounds
• Level
© James. V. Caruso, 2016
1. Strategy formulation / refresh
2. Planning & analysis (industry & company)
3. Make decisions (tactical)
4. After Action Review (AAR) – Part A / Group reflection
5. Individual reflection
6. Results debrief
7. After Action Review (AAR) – Part B / Group reflection
Simulation Process for each Round- Rounds – Practice (2), Competition (8)
13© James. V. Caruso, 2016
© James. V. Caruso, 2016
Total Enterprise Sim Example: Capsim – Foundation
3. Case Study –
Successfully using a Total Enterprise Simulation –
b. Business Acumen Content
15© James. V. Caruso, 2016
Finance and Accounting
Financial statements/analysis, ratio analysis, pro-forma financial statements; margin maximization
Strategy
Total enterprise impact; top management functions & responsibilities; competitive strategies; strategic planning & implementation
Marketing
Segmentation & targeting; customer loyalty; pricing
Operations
Production & operations - forecasting, capacity planning & inventory control; quality
Business Acumen
16© James. V. Caruso, 2016
3. Case Study –
Successfully using a Total Enterprise Simulation –
c. Leadership / Management Content
17© James. V. Caruso, 2016
• Team – stages, structure/roles
– processes - decision-making, communication, meetings
• Managing conflict
• Giving and receiving feedback
• Developing and using influence
• Communication styles
• Communication & presentation skills
• Alignment
Reflection (team & individual) - self-awareness
• TeamMATE – individual/team development/assessment tool
• After Action Review (AAR)
Leadership / Management
18© James. V. Caruso, 2016
3. Case Study –
Successfully using a Total Enterprise Simulation –
d. Assessment Tools
19© James. V. Caruso, 2016
20
Assessment Tools
© James. V. Caruso, 2016
Capsim
• Prework – Quiz, Rehearsal Tutorial
• Course – Success Measures; Balanced Scorecard; Analyst
Report; Star Summary Report
• Overall team performance
• TeamMATE – 360-degee feedback (individual & team)
Course
• Pre-course survey
• Assignments (8) – graded
• After Action Review (AAR)
21© James. V. Caruso, 2016
# Assignment (I = Individual; T = Team) I / T Points
1 Pre-Course & Pre-Simulation Preparation I 20
Team Participation Information (Qualtrics Survey) I 0
2 Preliminary Goals & Strategy T 15
3 Success-Performance Metrics T 10
4 Decision Reports (3) T 30
5 Decision Making Analysis Tools T 10
6 Board Presentation Material T 10
Board Presentation Delivery T 10
Board Presentation (content, delivery, visuals, responses) I 25
7 Lessons Learned and to be Learned I 30
Company Performance T 25
TeamMATE Evaluation (Capsim) I 15
Total Points 200
8 Team Contribution-Evaluation I +/-25
Summative Assessment & Grading Tools (11-week)
3. Case Study –
Successfully using a Total Enterprise Simulation –
d. Assessment Tools:
Success Measures
22© James. V. Caruso, 2016
A. Financial Management GoalsSuccess Performance Metrics - Success Measures
23
1. Cumulative Profits
2. Average $Market Share
3. Avg. ROS4. Avg. Asset Turnover (ATO; T/O)5. Avg. ROA6. Avg. ROE7. Ending Stock Price8. Ending Market Capitalization
DuPont Profitability Model
© James. V. Caruso, 2016
• Financial – Stock Price, Profits, Leverage
• Internal Business Processes – Contribution Margin, Plant Utilization, Working Capital Days, Stock-out Costs, Inventory Carrying Costs
• Customer – Customer Buying Criteria, Customer Awareness, Customer Accessibility, Product Count, SG&A Expense
• Learning & Growth – Employee Turnover rate, Employee Productivity, TQM Material Reduction, TQM R&D Reduction, TQM Admin Cost Reduction, TQM Demand Increase
Balanced Scorecard – Strategic Alignment
24© James. V. Caruso, 2016
Balanced Scorecard
25© James Caruso, 2016
Team Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8 Recap Total
Pos-sible 82 89 89 100 100 100 100 100 240 1000
An-drews 51 62 59 74 75 75 80 77 209 762Bald-win 52 41 46 59 62 50 57 61 147 575
Ches-ter 38 35 30 59 63 73 73 73 164 610
Digby 55 52 51 47 56 48 45 49 98 501
Erie 53 49 53 37 38 40 42 49 97 458
Ferris 61 52 45 45 47 43 46 43 94 476
Analyst Report – Year 8 & Cumulative
26© James Caruso, 2016
Team Mar-gins
Prof-its
Emer-gency Loan
Wor-king Cap-ital
Mar-ket
Share
Fore-cas-ting
Cust.Sat.
Pro-duc-tivity
Finl. Struc-ture
Wealth Crea-tion
Total-
Year 8
Cumul. Total
An-drews 100 100 100 50 100 100 100 92.4 0 100 842 6,474
Bald-win
66.7 100 100 50 80 100 100 63 0 100 760 5,331
Ches-ter 100 100 100 100 60 100 66.7 70.7 0 84.2 782 5,189
Digby 83.3 38.3 100 50 20 100 0 61.9 100 63.5 617 5,226
Erie 100 61.7 100 100 40 100 0 93.9 100 56.6 752 5,181
Ferris 36.7 19.7 100 50 0 100 0 49.7 100 45.7 502 4,736
Star Summary Report – Cumulative (8 years)
27© James Caruso, 2015
Team Profit Margin Emerg.Loan
Inventory Stock Price
Cumul.Total
+Contrib. Margin> 30%
$0
Met 95% of demand; Inv. Days
< 90
Increase from prior
year
Andrews 8 7 8 8 8 39Baldwin 8 6 6 0 8 28
Chester 8 6 8 4 8 34Digby 8 7 8 5 6 34
Erie 8 7 7 6 6 34
Ferris 8 5 8 5 5 31
3. Case Study –
Successfully using a Total Enterprise Simulation –
d. Assessment Tools:
After Action Review (AAR)
28© James. V. Caruso, 2016
1. Where are we now?
– Current state, ‘as is’
2. Where are we going?
– Desired future state, ‘to be’, vision, goal; business issue
3. How do we get there?
– Gap; Recommendations, decisions
Gap Analysis
29© James. V. Caruso, 2016
1. What did we intend to do?
Planned | Forecasted | Budgeted results
2. What actually happened and why?
Actual results
3. What do we need to change/improve next time?
After Action Review (AAR)
30© James. V. Caruso, 2016
After Action Review (after each decision round)
31
1 (low)
2 3 4 5 (high)
1. Do I understand our strategy?
2. Do I agree with the actions taken to implement our strategy?
3. Do I feel that my input is valued and respected?
4. Are we leveraging the diversity of the group (experience, functional background (supporting other’s development areas), communication style)?
5. Do I feel that my team has effective decision-making processes?
Remember - self-deception leads to self-destruction
© James. V. Caruso, 2016
3. Case Study –
Successfully using a Total Enterprise Simulation –
d. Assessment Tools:
TeamMATE
32© James. V. Caruso, 2016
• Developmental and performance-based; 360 degree feedback
• Information individuals can use to diagnose their own behaviors and overall team functionality in real time to allow for corrective, developmental action
– Individual-level Teamwork Behavior dimensions: the 4 P’s of Individual Performance - Performance … (1) Preparation; (2) Execution; (3) Monitoring; & (4) Adjustment
– Team-level Process dimensions: The 5 C’s of Team Dynamics –Confidence, Coordination, Cohesion, Cooperation, & Conflict
• Information assessors can use in evaluating individual team behavior and team performance
• The ability to correlate team dynamics scores with financial and operational results in the simulation over time, for analysis on the impact of effective team behaviors
TeamMATE Evaluation Tool – Parameters and Use
33© James. V. Caruso, 2016
Receiving Feedback (360-degree feedback)
34
UnrecognizedWeakness
(“blind spot”)
RecognizedStrength
RecognizedWeakness
Others’ (peers’) rating of you
Your rating of self
Low (1)
Low (1)
High (5)
High (5)
UnrecognizedStrength
Which box do you focus on? Also compare to Team Average
© James. V. Caruso, 2016
4. Different Delivery Formats
a. 11 Weeks Face-to-Face
b. 4-day Face-to-Face Residency
c. 11 weeks online
35© James. V. Caruso, 2016
11 Weeks (face-to-face and online)
• MBA - part-time
• MS
– Finance, Accounting, Business Analytics
– Electrical and Computer Engineering, Chemical Engineering; Television Management
• Joint – MBA and
– B.S. – Business Administration
– MS - Finance / Television Management
– Ph.D. - Electrical and Computer Engineering
• Post-masters certificate
• University employees
4-day Residency
• Incoming MBA students (first term)
– full-time
– part-time
– online
Student Profile
36© James. V. Caruso, 2016
Course Component SummaryCONTENT SIMULATION ASSIGNMENTS & ASSESSMENTS
After Action Review - (AAR – T&I)
LECTUREDEBRIEF
DECISIONS RESULTS AARAssignment
I = IndividualT = Team
360-degree Feedback: TeamMATE (T&I; 3)
Lessons Learned (I)
Prework (I) - Finance, Survey
Strategy / Team (T)
Success Metrics (T)
Presentations – Simulation (T & I)
Prestn. – Decision Making Tools (T)
Functional Analysis - 3 (T)
Team Contribution (I)
TeamMATE
Intros, Course & Simulation Overview - Strategy
Presentation Skills
Forecasting (Ops)
Success Measures/ Metrics
Team-conflict, feedback, influence; Commcn. styles
F/S Analysis
Lessons Learned
Real World Examples
Course Summary
Portfolio Mgmt., Marketing Segmentation
Prework – F/S
Debriefs (2)
Practice Rounds (2)
Competitive Rounds (8)
Results (2)
Results (8)
Debriefs (8)
Prework – Sim (I) – Quiz…
Company Performance (T)
Communication Style (I)
Ethics
4. Different Delivery Formats
a. 11 weeks Face-to-Face
38© James. V. Caruso, 2016
Sample Weekly Process – Thu. – Thu.Thu. - Class Fri. Sat. & Sun.
Mon. Tue. Wed. Thu - Class.
Decisions
Lecture Headlines & Results
Debrief
After Action Review – Team –
w/o Results
360-degree Feedback: TeamMATE
(I – Decisions)
Lecture
After Action Review – Indiv.
– w/ Results
360-degree Feedback: TeamMATE (T-Debrief)
360-degree Feedback:
TeamMATE (I –Reflection)
After Action Review – Team – w/ Results / Team Planning
MeetingAssignment (T) Due
Individual Analysis
Curriculum Sequence – 11 weeks face-to-facePrework Week 1 Week 2 Week 3 Week 4 Week 5
Week 6 Week 7 Week 8 Week 9 Week 10 Week 11
PR - 1
PR - 1
Lecture – 1Course,
Sim, Strategy
Results– PR1
AAR - 1
Lessons Learned
(I)
TeamMATE - 1
7- Presents.
5 - Forecasting
4 - Metrics
3 - Team
2 - Finance
8–Lessons Learn
9
10 – Course Summary
6 - Portfolio
Debrief - 1
PR - 2
AAR (T&I)
Finance, Survey, Sim (I)
Strategy / Team (T)
CR-7
PR - 2
65
4
3
2
LECTURE
8
DEBRIEF DECISIONS RESULTS
23
Decisions-1
AAR
AssignmentI = Individual
T = Team
Metrics (T)
Presentations –Simulation (T&I)
Presetn. – Decision Making Tools (T)
32
8
7
Results–CR-1 CR-2
CR-6CR-5CR-4
CR-3
CR-8
6
54
3
2
54
6
Functnl. Analysis (T) Functnl. Analysis (T)Functnl. Analysis (T)
Team Contribu -
tion (I)
TeamMATE (T&I)
4. Different Delivery Formats
b. 4-day face-to-face residency
41© James. V. Caruso, 2016
Curriculum Sequence – 4-Day ResidencyPrework DAY 1 DAY 2 DAY 3 DAY 4 Postwork
CK 4-day slides -CONTENT
Lunch Lunch Lunch Lunch
Lecture –Course,
Sim, Team, Strategy
AAR (T&I)
Lessons Learned
(I)
TeamMATE - 1
Presentatns.
Portfolio
Finance, Metrics
Forecasting
Debrief - 1
Strategy / Team (T)
CR-7
AAR
LECTURE
8
DEBRIEF DECISIONS RESULTS
2
TeamMATE (T&I)
Decisions -1
AARAssignment
I = IndividualT = Team
Metrics (T)
Presenta-tion
Delivery –
Simulation (T&I)
3
2 8
7
Results–CR-1
CR - 2
CR-6
CR-5
CR-4
CR-3
CR-8
6
5
4
3
25
4
6
Functnl. Analysis (T)
Functnl. Analysis (T)
Course Summary
Presentation Preparation – Simulation
(T)
Team Contribu-
tion (I)
Finance, Survey, Sim (I)
4. Different Delivery Formats
c. 11 weeks online
43© James. V. Caruso, 2016
• Blackboard Collaborate
– Synchronous webinar (recordable)
– Team meetings
• Recorded lectures – asynchronous
– Tips – # of min., content only, no instructions or time references => reusable
–Mediasite (studio)
–Active Presenter
• Discussion Board
BlackBoard Course Tools (Online)
44© James. V. Caruso, 2016
Sample Weekly Process – Thu. – Thu.Thu. Fri. Sat. & Sun.
Mon. Tue. & Wed. Thu.
Decisions DueRecorded Lecture
Headlines & Results
Recorded Debrief
After Action Review – Team –
w/o Results
360-degree Feedback: TeamMATE
(I – Decisions)
After Action Review – Indiv.
– w/ Results
360-degree Feedback: TeamMATE (T-Debrief)
360-degree Feedback:
TeamMATE (I –Reflection)
After Action Review – Team –w/ Results / Team Planning
Meeting
Assignment (T) Due
Individual Analysis
Live Webinar
Curriculum Sequence – 11 weeks OnlinePrework Week 1 Week 2 Week 3 Week 4 Week 5
Week 6 Week 7 Week 8 Week 9 Week 10 Week 11
PR – 1 Due
PR - 1Lecture 1
–Course,
Sim, Strategy
Results– PR1
AAR - 2
Assignments (I)–Lessons Learned
Team Contribution 2
TeamMATE - 1
7- Presents.
5 - Forecasting
4 - Metrics3 - Team
2 - Finance
Debrief 1PR – 2 Due
AAR - 1
Finance, Survey, Sim (I)
Strategy / Team (T)
CR-7
PR - 2
AAR - 7
AAR - 6
AAR - 5AAR - 4
AAR - 3
RECORDED LECTURE
REC. DEBRIEF DECISIONS
RESULTS
TM 2TM 3
TeamMATE (T&I)
Decisions-1
AAR (T&I)Assignment
I = IndividualT = Team
Team Cont 1
Presenta-tions –
Simulation (T&I)
Presetn. – Decision Making Tools (T)
3 Due2 Due
8 Due
7 Due
Results–CR-1
CR-2
CR-6
CR-5
CR-3
CR-8
6
5
3
5 Due
6 Due
Functnl. Analysis (T) Functnl. Analysis (T)
Functnl. Analysis (T)
Webinar
Webinar
WEBINAR (rec.)
Webinar
6 - Portfolio
CR-4
4
4 Due
Webinar
Discussion Board
Discussion Board
PR - 2
Webinar
2
Metrics (T)
Residency (4-days)
• No practice rounds (team)
• Less time for content and reflection
• Immersion; different learning objectives/outcomes
• 2 TeamMATE evaluations
Online (11-weeks)
• Time needed up front to get students up to speed on sim understanding & navigation
• Discussion Board – reflection, real-life examples, simulation questions
• Team Collaboration – team & instructor communications / meetings
• Similar online group-work challenges
• Potentially less exercises / assignments
Face-to-Face (11-weeks)
• Option to use recorded lectures - reinforcement
• Flipped classroom – lectures, simulation decisions, round debriefs, real-world examples, exercises
Differences and Opportunities in 3 Formats
47© James. V. Caruso, 2016