Assessing Motivation and Capacity to Argue in a Gamified Seminar Setting
-
Upload
technological-ecosystems-for-enhancing-multiculturality -
Category
Education
-
view
31 -
download
0
description
Transcript of Assessing Motivation and Capacity to Argue in a Gamified Seminar Setting
Assessing Motivation and Capacity to Arguein Gamified Seminar Setting
Shigeki Ohira† Kousuke Kawanishi‡ Katashi Nagao‡
†Information Technology Center, Nagoya University
‡Graduate School of Information Science, Nagoya University
E-mail: [email protected]
10/2/2014 TEEM2014
Technological Ecosystems for Enhancing Multiculturality (TEEM’14)Track 4. Gamification Ecosystems
Introduction
• Focus on Gamification
“The use of game design elements in non-game contexts”
– Successful cases primarily in business sector
– Application to e-learning education
• Require considerable advance preparation in real-world classroom situation
10/2/2014 TEEM2014
Events in classroom
Recording
Game mechanisms
Mapping
Critically important!for
- providing continuous support- measuring effects
Purpose and Approach
• Importance of discussion in laboratory seminars
– Understand a wide range of academic topics
– Express own opinions and brainstorm with others
→ Motivation to participate in discussion is not so high…
10/2/2014 TEEM2014
Sustain and improve motivation to participate in seminarsusing gamification techniques
Purpose
…
…
Student(Presenter)Teacher
Students(Participants)
• Recording semantic structure of seminars+ Gamification framework= Gamified Discussion (GD) System
Integrate gamification framework into seminar environmentApproach
Discussion Recorder
• Capturing video data with multiple cameras and microphones
• Presenter uses a browser-based interface to operate slides
• Each participant uses a special device to input metadata
• Secretary uses a browser-based interface to input statements
• Discussion structures and metadata are semi-automatically recorded
10/2/2014 TEEM2014
Gamification Framework
• Seven game design elements
10/2/2014 TEEM2014
Goal
Goal
Goal
Goal
Goal
Goal
Goal
End
Start
Difficulty adjustment:Adjust degree of difficulty and complexity of goals
Goal
GoalVisualization:Show a changing capacity to argue in real-time and permit users to track status
Rules:Enable participants in a discussion to evaluate one another
Design:Provide rewards and a sense of accomplishment
Social:Promote competition and cooperation among users
Goals:Step-by-step goal setting
Tutorial:Familiarize novices with how to use the system
Self
Others
Argumentative skills:LowMotivation: Low
Argumentative skills: HighMotivation: High
Capacity to Argue Classification and Goal Setting
• Five “capacity to argue” into 83 argumentative skills
– 10 reasoning skills
– 28 analytical skills
– 11 compositional skills
– 26 communication skills
– 8 system use skills
• Goal setting
– Seminar participants are able to set or modify goals as required using a tablet device
10/2/2014 TEEM2014
More difficult and complex toward the top
Higherskill
Lower skill
Argumentative skill graph
Real-time Evaluation of Argumentative Skills
• Statement evaluation interface
– Goal attainment level : five-point
– Statement quality : “+1” or “+2”
– Feedback to the Speaker
10/2/2014 TEEM2014
Participants
Participate in seminar with one tablet device per person
Screen
Other participantsevaluate speaker’s goal
4 53
5 5
Utterance Goal: “Valid criticism”
Average 4.4
Presenter
Feedback
Provide a sense of accomplishment
Level, Exp.
Ranking
• Rank– 10 levels of capacity to argue
– 10 most recent seminars
• Passion! Ranking– Last 3 seminars
– Commend student’s tenacity
– Strongly motivate new students and beginners
10/2/2014 TEEM2014
• Latest Ranking– High score award
– Show on shared displayat the end of each seminar
Show top 3 ranking to prevent users from losing motivation
Loud voice
Talk with a smile
With gestures
Rank 1st
Rank 2nd
Rank 3rd
Visualizing Capacity to Argue
• GD MyPage
– Top, Skill, Status, History, Ability, Item, and Badge tab
– Verify own progress and set next goals
– Badges are earned by pursuing recommended actions
10/2/2014 TEEM2014
Verify their own progress in comparison with others by referring to a radar chart
Set next goals while browsing argumentative skill graph
Radar chart of capacity to argue
Badge“Access to MyPage”, “Achieve 10 goals”, “Get ranking 1st”, etc.
Status tab:
Argumentative skill graph
Skill tab:
Evaluation Experiment
• Participants and experimental period
– 11 students and 3 faculties
– November 14, 2012 to December 25, 2013 (62 seminars)
• Assessment measure
– Motivation
• Number of statements
– Capacity to argue
• Speaking skills : quantity and quality of statements (3 criteria)
• Listening skills : whether he/she is listening closely (5 criteria)
10/2/2014 TEEM2014
11 12 1 2 3 4 5 6 7 8 9 10 11 12
1st period 2nd period 3rd period
Goal setting andmutual evaluation
Mutual evaluation of statement quality and the concepts of Exp. and Levels
Rewards
New school year
Improvements of GD
Summervacation
Springvacation
Details of Participants of Students
Student A M2 M2 M2 M2 M2 GraduationStudent B M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student C M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student D M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student E M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student F M1 M1 M1 M1 M1 M2 M2 M2 M2 M2 M2 M2 M2 M2Student G B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student H B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student I B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student J B4 B4 B4 B4 B4 M1 M1 M1 M1 M1 M1 M1 M1 M1Student K B4 B4 B4 B4 B4 GraduationStudent L
Prior to joining laboratoryB4 B4 B4 B4 B4 B4 B4 B4 B4
Student M B4 B4 B4 B4 B4 B4 B4 B4 B4
10/2/2014 TEEM2014
11 12 1 2 3 4 5 6 7 8 9 10 11 12
1st period 2nd period 3rd periodNew school year
Total 11 students
(M2 : 1, M1 : 5, B4 : 5)Total 11 students
(M2 : 5, M1 : 4, B4 : 2)
B4 : undergraduate seniorM1 : first year graduate studentM2 : second year graduate student
Evaluating Motivation
• Comparison of the number of statements• Before GD (Apr. 1, 2010 to Nov. 13, 2012; 135 seminars)
• After GD (Nov. 14, 2012 to Dec. 25, 2013; 62 seminars)
• Average number of statements per hour of all students
• Increasing tendency
– Before and after introduced GD system
– Implementation in 2nd period and 3rd period
– Questionnaire: all of the students felt improvement of motivation10/2/2014 TEEM2014
1.07
1.40
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
Significantly differentt-test (α=0.05)
1.010.86
1.110.98
1.22
1.73
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
第1期 第2期 第3期
GD導入前
GD導入後
Before GD After GD
Before GD
After GD
1st period 2nd period 3rd period
rewards- Evaluation of statement quality- Concepts of exp. and levels
Evaluating Motivation (cont.)
• Number of B4 statements of each school year
• The number of B4 statements in school year 2013 is greatly increased as compared to other years
• GD contributes to increase motivation of students unfamiliar with seminars
10/2/2014 TEEM2014
0
0.2
0.4
0.6
0.8
1
1.2
1.4
2010 2011 2012 2013
Average number of statements per hour
45% up
(After GD)
Evaluating Capacity to Argue
• Graph of capacity to argue
– Capacity to argue improved in all grades
• B4 students maybe concentrate on listening to the other students while unfamiliar with discussions
• Questionnaire: GD system and mutual evaluation scheme contributes to better “capacity to argue”
10/2/2014 TEEM2014
0
10
20
30
40
50
60
0 10 20 30 40 50 60
List
enin
g sk
ill
Speaking skill
1st period
2nd period
3rd period
B4
M1
M2
Summary and Future Work
• Purpose and approach
– Sustain and improve motivation to participate in seminar
• Incorporate gamification framework into seminar setting
• Analyze one year’s seminar records
• Results
– Improvement of student’s motivation and capacity to argue
• Future work
– Continued operation and assessment of our system
• What happens when removing gamification framework?
– Detailed analysis and refinement in each game element
• What elements contribute to attain educational goals?
– Expansion of application to other research activities
• How smoothly everything flows?10/2/2014 TEEM2014
Gracias por su atención
Ask your question
Preliminary Slides SLIDES
10/2/2014 TEEM2014
Game elements introduced to GD
10/2/2014 TEEM2014
Game element Main mechanism introducing the element
Goals Acquire segmentalized capacity to argueLevelsRankAcquire badgesMissions
Visualization Immediate evaluation feedbackVisualization of capacity to argue on MyPage
Rules Step-by-step learning capacity to argueMutual evaluation of goals
Design AbilityItemProfile image
Social Compare capacity to argueRanking
Tutorial Tutorial seminar
DRIP Cycle
• A cycle of four phases representing knowledge activities focused on discussion
10/2/2014 TEEM2014
Investigation
Recording new knowledgebased on past discussion
Preparation
Creating of presentationmaterials using records
Reflection
Arranging ofdiscussion content
Discussion
Creating of reusablediscussion content
Discussion Browser
10/2/2014 TEEM2014
Video View
Statement View
Layered Seek Bar http://dm.nagao.nuie.nagoya-u.ac.jp
Slide View
Evaluating Capacity to Argue
• Scoring method of speaking skills
– Harmonic mean between two scores for quantity and quality of statements (3 criteria: S1 to S3)
• Quantity score “𝑄𝑡𝑦”:
– S1 : number of statements 𝑁 0,… , 𝑡ℎ𝑒 𝑙𝑎𝑟𝑔𝑒𝑠𝑡 𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑎 𝑠𝑒𝑚𝑖𝑛𝑎𝑟
𝑄𝑡𝑦 = 𝑃(𝑁)
• Quality score “𝑄𝑙𝑡”:
– S2 : evaluated goal score 𝐺 (1,… , 5) and learning difficulty 𝐷 (1, … , 100)
– S3 : evaluated quality score 𝑄 0 𝑜𝑟 1 𝑜𝑟 2
𝑄𝑙𝑡 =𝑆2 + 𝑆3
2=𝑃 𝐺 ∙ 𝐷 + 𝑃(𝑄)
2
– Speaking skill score: 𝑆𝑠𝑐𝑜𝑟𝑒 =2∙𝑄𝑡𝑦∙𝑄𝑙𝑡
𝑄𝑡𝑦+𝑄𝑙𝑡
10/2/2014 TEEM2014
𝑃 𝑎 : function 𝑃 converts 𝑎 to perfect score of 100
Learning difficulty of goal is determined by pre-questionnaire
Evaluating Capacity to Argue (cont.)
• Scoring method of listening skills
– Whether he/she is listening closely when another person is talking (5 criteria: L1 to L5)
– L1 : Press agreement/marking button with special device
– L2 : Assess quality of statement as 1 or 2 points
– L3 : Make a follow-up statement in response to a statement
– L4 : Evaluation timing in response to a goal (0, …, 100)
» Within 5 seconds after the end of statement
– L5 : Reliability of evaluation score in response to a goal (0, …, 100)
» Comparison with evaluation by faculties
» Goal score and quality score for each statement
a) All statements 𝑠𝑎 within discussion segment includes actions L1 to L3 … perfect score
b) For all other statements 𝑠𝑏, add L4 and L5 scores at the ratio of 1:1 and convert to 100
– Listening skill score:
10/2/2014 TEEM2014
- if 𝑠𝑖 ∈ 𝑠𝑎 then 𝑃(𝑠𝑖)=100
- else if 𝑠𝑖 ∈ 𝑠𝑏 then 𝑃 𝑠𝑖 =𝐿4+𝐿5
2
𝐿𝑠𝑐𝑜𝑟𝑒 =1
𝑁𝑎𝑙𝑙
𝑖=1
𝑁𝑎𝑙𝑙
𝑃(𝑠𝑖)
Motivation based on Rewards
• Intrinsic motivation
– Derived from ambition and curiosity of human
– Gamification framework
• Goals, Visualization, Rules, Social
– What someone wants to do, what one wants to become
• Extrinsic motivation
– Derived from rewards and punishments
– Gamification framework
• Design: Experimental point, Level, Bonus point, Rank, Ability, Item, Coin, Badge
– Fun and pleasure
10/2/2014 TEEM2014
Player Type : Interest Graph [Bartle 96]
10/2/2014 TEEM2014
WorldPlayers
Acting
Interacting
AchieversKillers
ExplorersSocializers
Earn Points, Level upCompetition
Discovery of new oneCommunication
Tutorial
• Handling of structural remote control
– Pointing on screen, reservation of statement, etc.
– Mutual evaluation of goals
10/2/2014 TEEM2014
Item of tutorial:“Underline appropriate places”
Current goal score
Total records
• Number of seminars: 96
• Number of statements: 6,812
• Number of assessments: 44,979
• Number of badges earned: 1,012
• Number of goal achievement: 769
10/2/2014 TEEM2014
Metadata about Discussions
• Statement segment
– Start and end times of statements
– Each participant uses a special deviceto declare start/end of his/her statement
• Discussion segment
– Division discussion into units of topicby type of statement
• Start-up statement: To begin a new topic
• Follow-up statement : To continue previous topic
– Input type by changing direction of the device when user begins utterance
– Each discussion segment has tree structurewhich begins the start utterance
• Attitude towards statement
– Agreement/Disagreement, Marking
10/2/2014 TEEM2014
Start
Follow
Start
Follow
FollowFollow
Approach
• Recording semantic structure of seminars+ Gamification framework= Gamified Discussion (GD) System
10/2/2014 TEEM2014
Integrate gamification framework into seminar environment
Game design elements
Meeting
Active participation
Enjoy game elements along with the improvement of capacity to argue
Goal of fulfillment
Attractive interface
Assessmentand
feedback
Evaluating Capacity to Argue (exactly)
• Scoring method of speaking skill
– Harmonic mean between two scores for quantity and quality of statements
• Quantity score “𝑄𝑡𝑦” :
– S1 : number of statements 𝑁 0,… , 𝑡ℎ𝑒 𝑙𝑎𝑟𝑔𝑒𝑠𝑡 𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑎 𝑠𝑒𝑚𝑖𝑛𝑎𝑟
𝑄𝑡𝑦 = 𝑃 𝑁 = 100 ∙𝑁
𝑁𝑚𝑎𝑥
• Quality score “𝑄𝑙𝑡” :
– S2 : evaluated goal score 𝐺 (1,… , 5) and learning difficulty 𝐷 (1, … , 100)
– S3 : evaluated quality score 𝑄 0, 1, 2
𝑄𝑙𝑡 =𝑆2+𝑆3
2=𝑃 𝐺∙𝐷 +𝑃(𝑄)
2
= 0.5 ∙ 100 ∙1
𝑁
𝑖=1
𝑁𝐺(𝑖) ∙ 𝐷(𝑖)
500+ 0.5 ∙ 100 ∙
1
𝑁
𝑖=1
𝑁𝑄(𝑖)
2
– Speaking skill score: 𝑆𝑠𝑐𝑜𝑟𝑒 =2∙𝑄𝑡𝑦∙𝑄𝑙𝑡
𝑄𝑡𝑦+𝑄𝑙𝑡10/2/2014 TEEM2014
𝑃 𝑎 : function 𝑃 converts 𝑎 to perfect score of 100
Learning difficulty of goal is determined by pre-questionnaire
Real-time Evaluation of Argumentative Skills
• Statement evaluation interface
– Goal attainment level : five-point
– Statement quality : “+1” or “+2”
– Feedback to the Speaker
10/2/2014 TEEM2014
Participants
Participate in seminar with one tablet device per person
Screen
Other participantsevaluate speaker’s goal
4 53
5 5
Utterance Goal: “Valid criticism”
Average 4.4
Presenter
Feedback
Provide a sense of accomplishment
Level, Exp.
Reward commensurate with the number of statements
during a seminar
Mission
MISSION2 state. bonus → coin+3003 state. bonus → exp+604 state. bonus → coin+1500
Discussion Room
10/2/2014 TEEM2014
サーバ室
dms01dms02
ss02
DM-VIS
DM-MAIN
書記PC
発表PC
canopus
NICE
サーバ・PC
スイッチングハブ
無線LAN AP
情報コンセント
研究室内LAN (1系)
Wiiポインタ用LAN (3系)
gw
Bluet01-05
IB南395
Gamification 11 elements
10/2/2014 TEEM2014
1 Step-by-step great goal
2 Tutorial
3 Autonomy and independence
4 Rules
5 Direct feedback
6 Design and production
7 Difficulty control
8 Competition
9 Cooperation
10 Visualization of the situation
11 Title and growth
http://gamificationlab.jp