Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2...

15
TARDIS Training young Adult’s Regulation of emotions and Development of social Interaction Skills FP7-ICT-2011-7 Deliverable D5.3 System Architecture Game Process and User Modelling Due date of deliverable: 31/8/2014 Actual submission date: 30/9/2014 Partner responsible: DFKI Author: Patrick Gebhard ([email protected]) Name of participants: UPMC, DFKI, IOE, IT, UAU, UU, Charamel Classification: PU Grant Agreement Number: 288578 Contract Start Date: November 1 st , 2011 Duration: 36 Months Project Coordinator: UPMC Partners: UPMC, DFKI, IOE, IT, MLVOE, UAU, UU, Charamel. Project website address http://tardis-project.eu/

Transcript of Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2...

Page 1: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

TARDIS Training young Adult’s Regulation of emotions and Development of social Interaction

Skills

FP7-ICT-2011-7

Deliverable D5.3

System Architecture

Game Process and User Modelling

Due date of deliverable: 31/8/2014

Actual submission date: 30/9/2014

Partner responsible: DFKI

Author: Patrick Gebhard ([email protected])

Name of participants: UPMC, DFKI, IOE, IT, UAU, UU, Charamel

Classification: PU Grant Agreement Number: 288578 Contract Start Date: November 1st, 2011 Duration: 36 Months Project Coordinator: UPMC Partners: UPMC, DFKI, IOE, IT, MLVOE, UAU, UU, Charamel. Project website address http://tardis-project.eu/

Page 2: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

2

Table of Content

I INTRODUCTION .......................................................................................................................... 3

II TARDIS GAME – A JOB INTERVIEW SIMULATION .......................................................... 3 II.1 SUPPORTED LANGUAGES ..................................................................................................................... 4 II.2 INTERVIEW ............................................................................................................................................. 4 II.3 VIRTUAL RECRUITER ............................................................................................................................ 6 II.4 PLAYER WELCOME SCREEN – USER MODEL .................................................................................... 7 II.5 JOB AREAS .............................................................................................................................................. 8 II.6 GAME DIFFICULTY ................................................................................................................................ 8 II.7 ENVIRONMENT AND CAMERA ............................................................................................................. 8 II.8 TUTORIAL PHASE .................................................................................................................................. 9

III REWARD MECHANICS ............................................................................................................ 9 III.1 CREDIT POINTS FOR CORRECT SOCIAL CUES ............................................................................... 10 III.2 GAME LEVELS .................................................................................................................................... 11

IV TRACKING SOCIAL CUES ......................................................................................................12

V SUMMARY ..................................................................................................................................13

VI LITERATURE ............................................................................................................................14

Page 3: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

3

I Introduction This document describes the player progress within the TARDIS job simulation game with interactive

virtual recruiters addressing user model, gameplay, challenges, and reward system. In addition, it

reports on the employment of tracking a user’s social cues, which plays a central role for modeling the

reactions and feedback of the virtual job recruiter during the game.

The TARDIS game design discussion included the target group considering both the educative needs,

and the market segmentation (see Deliverable D 5.2). In this context, the opportunities and the

limitations of each of the used components have been elaborated. This guided us in shaping the vision

for the final TARDIS game. Therefore, we included expert knowledge of all stakeholders (e.g. both the

training centers and the NEETs targeted).

In general, the document describes the overall concepts and each part of the TARDIS job simulation

game from a user’s perspective.

II TARDIS Game – A Job Interview Simulation One goal of the TARDIS project is provide youngsters with a simulation game in which they can

interactively discover a simulated job interview for training purposes. In the TARDIS job interview

game a player is encouraged to conduct an interview with a Virtual Recruiter, who assesses the socio-

emotional behaviors of the player through passive input devices. The recruiter system interprets user

behaviors and reacts to these appropriately conveying responses fitting to the situation. The objective

of the game is that it be used as a tool to aid youth inclusion practitioners for use particularly by

youngsters who wish to improve their confidence and social-skills.

Figure 1: TARDIS Office for Simulated Job Interviews.

The target audience is defined to be between the ages of 18-25, but studies have shown that younger

people could play the game too. The TARDIS training game, a tool for supporting youngsters at risk of

exclusion to gain the social skills pertinent to job interviews.

Page 4: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

4

For the design of this game we have started from the question: What are the benefits for the

youngsters? When comparing our approach to typical job interview training material, such as books,

web coaching sessions, and videos, two aspects are striking: the physical setup and the interaction

experience. The physical setup imitates a real interview situation, and the interactivity elicits emotions

through dialog action; in any case, more intense as by the mentioned training material. The core benefit

is having experienced a simulated job interview that comes close to a real job interview, but in a save

environment. Directly related to the goal having a progressing effect (e.g. better cope with exposing

situations of oneself), we modeled 1) the overall game process, 2) the interaction structure and content,

3) the feedback on how a user performs, and 4) the emotional impact from the virtual recruiter.

In order to avoid the uncanny valley effect on our characters, the TARDIS game relies on the

commercially available and well-recognized Charamel (http://www.charamel.com/) avatar engine to

render the virtual characters. The animations (e.g. gestures, idle movements, and physiological aspects,

like eye gaze, or blinking) of Charamel's characters appears natural is well accepted in many

commercial applications. In addition, we extended the naturalness of the character, by directly

controlling gaze behavior, breathing appearance, head movement on the skeleton level; see the study

results of [Gebhard et al. 14]. And, following the outcome of this study, we designed the interview

character to show an understanding overall behavior, since the situation itself is inducing a lot of stress.

The game environment is designed to resemble a typical office and is inhabited by a virtual character

that plays the role of an interviewer, a game master, and a coach.

II.1 Supported Languages

The game is localized in the three languages: English, French, and German. Each utterance of the

virtual recruiter is synthesized with the current Nuance TTS system. Since both, female and male

recruiter voices are used, totally six TTS voices are supported.

Figure 2: Social Cue Cards with credit points.

II.2 Interview

The game itself is designed to roughly resemble an actual job interview but also uses game concepts to

make the experience more pleasant for users. The overall duration is designed to be approximately 10-

15 minutes. The difficulty of the levels is realized through different specifcations (see Section II.6).

Page 5: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

5

Game Modes

The TARDIS Job Simulation game has two modes that addresses different overall training goals:

1. Challenge. This is the main mode. It is designed to confront youngsters with the social challenges

that come along with a job interview.

2. Exploration. This mode is designed to help youngsters to get acquainted to the challenges of a job

interview and the game mechanics. Having the mission to perform really badly might help to

overcome possible restraints.

In the exploration mode the displayed scores reflect the level of inappropriate behavior – defined by the

social cue cards. For example, for the Welcome phase, one appropriate social cue is to show a friendly

behavior (smile). If the user does not smile at all, s/he’ll get a point for that.

Figure 3: Game Mechanics explained: users are asked to smile. The game gives feedback, if it is detected correctly.

Page 6: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

6

Social Cue Cards To guide youngsters in the interview game and to give them some level of control, social cue cards (see

Figure 2) are used. These cards, similar to cards of classic board games, give hints about appropriate

behaviors for a specific phase of an interview. And, on a conceptual level, they describe what is

expected from a user. Namely, they hint which social cues the user should perform in the next

interview phase.

During a play-through, the character will guide the user through three distinct interview phases: Welcome, Company

Welcome, Company Presentation, and Strengths and Weaknesses. Each phase is composed out of a series of questions

series of questions and answers (dialog games, or turns). After each question, the character will wait for the user to give

for the user to give an answer before proceeding to the next question. Prior to each phase, the character will show the

will show the user a social cue card (see

Figure 2) and instruct her or him to read it carefully. The hinted social cues address gaze behavior,

gestures, posture, and linguistic features. The character will wait until the user finishes reading the card

and says proceed or next.

After each phase, the virtual character switches the role from the job interviewer to a personal coach.

The character will go over the user's social cue performance during that phase, praising good behavior

Page 7: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

7

and encouraging a user to perform possible omitted social cues in a nextround. The feedback is

designed to address the possibility that cues are not detected correctly. Once the last phase is

completed, the virtual character will sum up the entire interview. During the game, the social cues on

the game cards are also visualized on the screen using dynamically colored icons. Every time the user

behaves according to one of the tips, as a reward, the corresponding icon will be highlighted (see

Figure 3).

II.3 Virtual Recruiter

The employed Virtual Recruiters are realized based upon the TARDIS academic framework using all

modules (see Deliverable D 5.2). The game provides two virtual characters: one male and one female.

This addresses the findings about common gender stereotypes that seem to be transferable to for virtual

characters too (e.g. rapport) [Zanbaka et al. 06, Karacora et al. 12].

Figure 4: Female and male virtual job recruiter in the TARDIS game.

The non-verbal behavior of both recruiters is defined by letting the character 1) show narrow gestures

close to the body, 2) show facial expressions that can be related to positive emotions (e.g. joy,

admiration, happy-for), 3) using shorter pauses (in comparison to the demanding character), and 4)

show a friendly head and gaze behavior. On the verbal level, explanations and questions show

appreciation for the user and contain many politeness phrases.

To control the amount of stress by linguistic variation, our work starts from the politeness theory

developed by Brown and Levinson [Brown and Levinson 87]. According to the politeness theory, all

social actors have face wants: the desire for positive face (being approved of by others) and the desire

for negative face (being unimpeded by others). Many conversational exchanges between people, (e.g.,

others, requests, commands) potentially threaten positive face, negative face, or both. To avoid this,

speakers employ various types of face threat mitigation strategies to reduce the impact on face.

Strategies identified by Brown and Levinson include positive politeness (emphasizing approval of the

interlocutor), negative politeness (emphasizing the interlocutor’s freedom of action, e.g., via a

suggestion) and off-record statements (indirect statements that imply that an action is needed).

The character model applies a variety of strategies to mitigate potential face threats, such as convey

interest (”I am eager to know more about ...”), claim in-group member- ship (”Let us now talk about

...”) or give option not to act (”Would you like to tell me ...”).

Page 8: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

8

Furthermore, the level of stress is modulated by the character’s backchannel behavior. Backchannels

are brief verbal or nonverbal cues listeners provide during a conversation to indicate the speaker that

they are still following the conversation. Previous work [Gratch et al. 07] shows that the simulation of

backchannel cues helps increase the level of rapport between a virtual character and a human. To create

a pleasant atmosphere for the user, the understanding character signals comprehension by head nods

and brief verbal utterances, such as ”Ok” or ”I see”. The user is also encouraged through the use of

positive feedback, such as smiles. In contrast, the demanding character does not provide any

backchannel cues to indicate engagement in the interaction. Additionally, it tries to unsettle the user by

showing negative feedback, such as frowning.

In addition to the character’s backchannel behavior, we use different types of conversational gestures

and key posture features for realizing the two personalities. We rely on the work of Ravenet, Ochs, and

Pelachaud who have collected a corpus of a virtual characters non-verbal behavior that a character

should display to convey particular interpersonal attitudes [Ravenet, Ochs, and Pelachaud 13]. The

corpus contains a comprehensive overview on non-verbal and verbal behavior rules, which are

congruent with the related literature we use. For this work, we focus on the rules for friendly gestures

and posture feature (e.g. tilt of the head on a side with no gaze not averted). To further decrease a

user’s stress level, the recruiter shows a friendly gaze behavior that is supposed to be perceived as

unobtrusive [Fukayama et al. 02].

Figure 5: Player Welcome Screen and related User Model.

II.4 Player Welcome Screen – User Model

The Player Welcome Screen let’s a user enter personal information. All personal information will not

be stored locally on the computer with regard to respect the privacy of the TARDIS game users. The

Welcome Screen is in fact an interface to parts of the user model the game has (see Figure 5).

The user model holds information in three areas:

1. Player General Info. Consists of the name and the gender information.

Page 9: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

9

2. Player Experience. Consist of a compilation of skills of the TARDIS target group youngsters that

are related to the modeled job areas (see next section).

3. Player Profile. Hold information about self-assessment of a player’s personality, ranging from shy

to confident.

The Player General Info is used by the game to address the user respectfully, e.g. users are addressed

with her/his noted name and syntactically correct gender flexion on the word level.

II.5 Job Areas

The final TARDIS game supports three different job areas among a user might chose in the Player

Welcome Screen. Those areas have been defined by asking job agency experts, teachers, and job

marked analysis:

1. Assistant in Call-Centers.

2. Postal Delivery Service.

3. Nursing staff (hospitals).

The overall game flow is not changed but different job related questions are asked and during the

Company Presentation phase other content is used.

II.6 Game Difficulty

Three aspects influence the overall difficulty of the game.

1. Selected Recruiter. Players might get uncomfortable when selecting a recruiter of the opposite

gender [Zanbaka et al. 06, Karacora et al. 12].

2. Chosen Player Experience. Users define the queried personal experiences in terms of strength and

weaknesses. They can be adjusted by a graphical interface that appears prior to the game play (see

Figure 6, top, left). The more extreme an item is configured (e.g. no knowledge in Math, expert

knowledge in Spanish) is higher the chance that this is addressed during the interview.

3. Chosen Player Profile. A confident selection lets recruiter have some demanding non-verbal

social cues; a shy selection selects understanding non-verbal behavior, drifty and cocky let the

system randomly chose a recruiter type but with a bias related to shy or confident.

The overall game flow is not changed by any of these values.

II.7 Environment and Camera

The world that the virtual recruiter inhabits is a typical office environment. As mentioned previously, it

is designed as a seated office with a desk.

Page 10: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

10

Figure 6: Player experience and virtual recruiter standing in 3D office and sitting at the desk.

The character can stand or sit (see Figure 6). While a user starts to play the game the character first

stands and invites a user to sit. Then the character sits, supporting the impression that the official

interview will start. After game, the virtual recruiter will stand up again saying goodbye, signaling that

the interview is over. Every posture transition, 1) standing to sitting, and 2) sitting to standing triggers a

camera movement. In the first case, the camera is zooms towards the virtual recruiter, showing at the

end of the transition the upper body. In the second case, the camera is zooms away from the virtual

recruiter, ending with showing the whole scene.

II.8 Tutorial Phase

At the beginning of the game, the character introduces itself and presents the game mechanics to the

user in a tutorial phase, which is common to modern computer games. The purpose is to make users

acquainted to the capabilities of sensor system, and linked reactions of the virtual recruiter and the

system. This addresses transparency and the functionality of the system.

III Reward Mechanics In general, the TARDIS game is designed for youngsters to play without any support with a youth

worker. To provide youngsters a guided experience the tasks they have to perform in the game are

defined by social cue cards (see Figure 2).

While playing the game, the youngster is asked to adapt to specific social task situations, which are

related to the game phase (see Figure 6, left bottom). The welcome phase is related to the social task of

presenting one self. The subsequent phase of company presentation is related to the task of listening

carefully, and the last phase strength and weaknesses is related to conversation about the youngster’s

profile. For each phase different social cues are required. Experts, e.g. social workers and job

recruiters, identified those in several TARDIS related workshops and evaluation studies, see

Deliverable D 6.1 and D 6.2.

Page 11: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

11

Figure 7: Interview phase related social cues and credit points.

Figure 7 gives an overview about all credit points a user can achieve during an interview. In total 17

credit points are possible. Since the last phase is difficult per se, credit points get doubled, if all cues

are performed correctly during that phase.

III.1 Social Cue Credit Points

During the virtual interview, the user will receive rewards for a range of properties concurrent with

appropriate behaviour in job interviews. These credits will contribute to an overall ‘score’ upon the

conclusion of the interview. Completing a level will occur once a specified score has been achieved.

Once they have completed one level, they can attempt a job interview with a more challenging level.

The repeated exposure to these various interviews should make the idea less daunting when it comes to

a real-life situation. Also, the achievement of a successful job interview, should improve confidence

with each success.

Every tip of the current social cue card, that a user performs correctly, will grant one point. The points

will sum up over the course of the entire game and are meant to give the user an additional incentive

for showing adequate behavior during the interview. For example, if the user uses an appropriate

amount of smiling during answers to the questions of the recruiter, the social cue gets highlighted and

the total credit counter gets increased. Some of the cues have to be performed (or not performed)

during a whole interview phase, e.g. do not freeze up (see Figure 7).

According to the social task, the user is asked questions by the virtual recruiter or to listen carefully

while she is talking. The game gives direct feedback of using required social cues in a fitting or non-

fitting way.

Page 12: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

12

During the game, these cues care displayed directly on the game screen supporting the youngster which

cues are required (see Figure 8).

The detection of the clues relies on the nonverbal behavior analyzer NovA tool that enables the

TARDIS framework to detect non-verbal cues, and voice quality aspects in real-time using the MS

Kinect sensor, see section Tracking Social Cues below, and Deliverable D 2.3.

Figure 8: Reward for holding eye contact.

III.2 Game Levels

The game levels are designed to address different necessary social qualities of employees. Such are:

friendliness, politeness, open-mindedness, being interested and committed, and giving the impression

of being competent.

Welcome

The Welcome phase is an important phase in job interviews since it comes with the first impression of

a possible job candidate. The (implicit, since this is not be verbalized) goal of that phase is to give the

impression of being polite, friendly and open-minded. The candidate is asked to present herself/himself

with the following social cues: 1) smile, 2) hold eye contact, 3) use open gestures, 4) speak loudly, and

5) don’t freeze up. During the game, these cues are displayed directly on the game screen supporting

the youngster which cues are required (see Figure 7, top).

According to the social task, the user is asked questions by the virtual recruiter or to listen carefully

while she is talking. The game gives direct feedback of using required social cues in a fitting or non-

fitting way.

Company Presentation

The Company Presentation phase is designed to test the qualities of candidates to listen and to show

interest. The candidate is asked to present herself/himself, with the following social cues: 1) don’t cross

arms, 2) hold eye contact, 3) lean front, 4) listen carefully. As in the Welcome phase, these cues are

displayed directly on the game screen (see Figure 7, center).

Page 13: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

13

Strengths & Weakness

The Strengths & Weakness phase is important with regard to a candidate’s own presentation of job

related skills. It is designed to test the qualities of candidates being committed, and giving the

impression of being competent. The candidate is asked to present herself/himself with the following

social cues: 1) give long answers, 2) don’t freeze up, 3) give long answers (at least more than 8 words),

and 4) talk lively (with a vivid voice). As in the Welcome phase, these cues are displayed directly on

the game screen (see Figure 7, center). If all cues are performed correctly, the credit points are doubled

to give an incentive to master the tough phase (see Figure 7, bottom).

Figure 9: Non-verbal behavior Analyzer tool.

IV Tracking Social Cues The NovA tool (see Figure 9) serves to analyze the learner’s social cues in real-time when interacting

with a virtual recruiter in a simulated job interview. In the context of the TARDIS job simulation game,

the SSI framework is used to analyze various social signals (see Deliverable D 2.3). SSI provides an

interface to a large diversity of sensing devices as well as a variety of tools for the real time recording

and pre-processing human behavior data. The following cues are detected in real-time:

• Body and Facial Features: Postures, gestures, head gaze, smiles, motion energy, overall activation

• Audio Features: Voice activity, intensity, loudness, pitch, audio energy, duration, pulses, periods,

unvoiced frames, voice breaks, jitter, shimmer, harmonicity, speech rate

Besides enabling the system to react to the user in real time, these cues also give us a glimpse into the

user’s state of mind during the interview, allowing us to observe the impact of the virtual character’s

actions on the user.

Providing such an environment is highly desirable from the point of view of improving practice, since

it enables a repeatable experience that can be modulated to suit the individual needs of the learner. It

may also mitigate negative side effects resulting from real-life settings, in particular, the stress

associated with engaging in unfamiliar interactions with others. In this context, the recognition rate of

our social cue detection for body language and voice activity detection has been evaluated and

Page 14: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

14

achieved a mean recognition rate of 88% [Baur et al. 13]. Various studies have been conducted using

real job seeking youngsters and trainer practitioners [Porayska-Pomsta et al. 13] [Gebhard et al. 14]

where NovA was used to debrief users of the TARDIS system. Ongoing user experience evaluations

[Porayska-Pomsta et al. 14] have shown so far that user’s self reports about their behavior

characteristics correspond to NovA’s calculated outputs. Feedback from youngsters and trainers

revealed that self-reflection with the help of audiovisual recordings, enhanced by objective labeling of

behavior is better accepted, as it is seen more fair than subjective feedback from trainers. Another

aspect we’ve learned from these studies is that a simple user interface is more suitable for such a target

audience. We therefore also created a simpler version of the UI, showing only the video recordings,

labels of behavioral cues and general outputs from the user state detection for such purposes. The data

of mentioned studies has been used to shape the design of the graphical user interface, the interaction

design, the choice in social cues and the preliminary Dynamic Bayesian networks (DBNs) for detecting

user states.

V Summary This document presents the final version of the TARDIS Job Interview Training Game, focusing on the

game progress, player incentive, and user model. The game software is based on modules of the

TARDIS academic research platform. The commercial Charamel virtual character rendering engine

and the commercial Nuance TTS system extends them. We call the resulting software framework the

TARDIS commercial platform. This system has fewer demands on the required hardware and could be

the starting point for a deployable commercial job interview training game.

In the TARDIS Job Interview Training Game a player conducts an interview with a Virtual Recruiter,

who assesses the socio-emotional behaviours of the player through the MS Kinect sensor. Upon the

interpretation of the player behaviours, the recruiter reacts to these appropriately, and conveys

responses fitting to typical phases of job interviews. The overall objective of the game is that it be used

as a tool to aid youth inclusion practitioners for use particularly by youngsters who wish to improve

their confidence and social-skills. The target audience for the TARDIS game are youngsters looking to

become more included in the work force. For the most part, participants will be between the ages of 18-

25, but it is possible that they may be younger or older than this range.

Using the SSI (social signal interpretation) framework, the TARDIS game perceices non-verbal

behaviour of the player with the help of the Kinect Sensor. It allow the system to recognize various

non-verbal behaviours (social cues) such as gestures, postures, head gaze, facial expressions, voice

activity and other vocal features (pitch, energy, loudness). Feedback from the virtual recruiter will be

conveyed in natural behaviours and appropriate reactions to the user’s actions and body language

throughout the game.

Conducted evaluation lab studies and studies in relevant schools in Germany have shown a massive

interesst in this kind of training software. In addition, the results of the studies show a steeper leraning

curve when compared to traditional job interview training material.

Page 15: Deliverable D5 - LIMSI · FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3 2 ... The TARDIS game design discussion included the target group considering both the

Grant agreement no 288578

FP7-ICT2011-7 – Collaborative Project TARDIS – Deliverable D 5.3

15

VI Literature [Baur et al. 1 ] Baur, T., Damian, I., ebhard, P., Porayska-Pomsta, ., and Andr e, . ( 1 ). A job

interview simulation: Social cue-based interaction with a virtual character. In Proc. SocialCom 2013,

pages 220–227. IEEE.

[Brown and Levinson 87] P. Brown and S. C. Levinson. Politeness — Some universals in language

usage. Cambridge University Press, Cambridge, 1987.

[Fukayama et al. 02] A. Fukayama, T. Ohno, N. Mukawa, M. Sawaki, and N. Hagita. Messages

embedded in gaze of interface agents — impression management with agent’s gaze. In Proceedings of

the SI CHI Conference on Human Factors in Computing Systems, CHI ’ , pages 41–48, New York,

NY, USA, 2002. ACM.

[Gebhard et al. 14] P. Gebhard, T. Baur, I. Damian, G. Mehlmann, J. Wagner, and . Andr e.

Exploring interaction strategies for virtual characters to induce stress in simulated job interviews. In

Proc. of the 2014 international conference on Autonomous agents and multi-agent systems, pages 661–

668. International Foundation for Autonomous Agents and Multiagent Systems, 2014.

[Gratch et al. 07] J. Gratch, N. Wang, J. Gerten, E. Fast, and R. Duffy. Creating rapport with virtual

agents. In C. Pelachaud, J.-C. Martin, E. Andr é, G. Chollet, K. Karpouzis, and D. Pelé, editors,

Intelligent Virtual Agents, 7th International Conference, IVA 2007, Paris, France, September 17-19, ,

Proceedings, volume 4722 of Lecture Notes in Computer Science, pages 125–138. Springer, 2007.

[Karacora et al. 12] B. aracora, . Dehghani, N. r amer-Mertens, and J. Gratch. The influence of

virtual agents’ gender and rapport on enhancing math performance. In Proc. of the 4th Annual

Meeting of the Cognitive Science Society (CogSci), Sapporo, Japan, Aug. 2012.

[Porayska-Pomsta et al. 2013] Porayska-Pomsta, K., Anderson, ., Damian, I., Baur, T., Andr e, .,

Bernar- dini, S., and Rizzo, P. ( 1 ). odelling users’ affect in job interviews: Technological demo.

In User Mod- eling, Adaptation, and Personalization, volume 7899 of LNCS, pages 353–355. Springer

Berlin Heidelberg.

[Porayska-Pomsta et al. 2014] Porayska-Pomsta, K., Rizzo, P., Damian, I., Baur, T., Andre, E.,

Sabouret, N., Jones, H., Anderson, K., & Chryssafidou, E. (2014). Who's afraid of job interviews?

Definitely a question for user modelling. 411-422 . Paper presented at The 22nd Conference on User

Modelling, Adaptation and Personalization, Aalborg, Denmark.

[Ravenet, Ochs and Pelachaud 13] B. Ravenet, M. Ochs, and C. Pelachaud. From a User-created

Corpus of Virtual Agent’s Non-verbal Behavior to a Computational Model of Interpersonal Attitudes,

volume 8108, pages 263–274. Springer Berlin Heidelberg, 2013.

[Zanbaka et al. 06] C. Zanbaka, P. Goolkasian, and L. Hodges. Can a virtual cat persuade you?: The

role of gender and realism in speaker persuasiveness. In Proc. of the SIGCHI Conference on Human

Factors in Computing Systems, CHI ’ 6, pages 115 –1162, New York, NY, USA, 2006. ACM.