Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of...

Post on 14-Dec-2015

224 views 9 download

Transcript of Complementing Embodied Conversational Agents by Implementing Compliments Measuring the Effect of...

Complementing Embodied Conversational Agents by Implementing ComplimentsMeasuring the Effect of Compliments in Embodied Conversational Agents

Group 23 Marieke Agterbos | Tim van Bremen | Carel Jansen | Lisa Oud | Justin Post

Research Question

To what extent do compliments have an impact on the

user experience of embodied conversational agents?

Questions

1. Are the ECA’s compliments recognized as compliments? (Part

I)

2. Experience with vs. without compliments (Part II)

A. Perceived liking of the agent

B. Perceived level of comfort

C. Perceived information quality

Background theory

Making machines more human (Picard, 2000)

Minimize gap between real life contact and digital contact

Lifelike ECA (Louwerse et al., 2009)

Is sensitive to moods and sentiments

Politeness and variation in language (Behnam & A. Niloufar, 2011)

Add value and believability

Function of Compliments (Manes en Wolfson, 1981)

Goodwill and solidarity

Three types of personal compliments (Jucker, 2009)

Implicit, explicit or indirect

Behavioral realism might be more important than photorealism (Groom, 2009)

Generating positive response

More productive conversation

Pre-test

Are the ECA’s compliments recognized as compliments? Participants read a transcript of an ECA conversation

Questionnaire on different parts of the conversation:

1. Bot: ‘Hello, how are you?’2. User: ‘Great, and you?’3. Bot: ‘Good, so am I.’4. User: ‘Good.’5. Bot: ‘What was the name of the lead guitar player of the band called “A band of Gypsies?”’6. User: ‘Jimi Hendrix.’7. Bot: ‘That is correct.’8. Bot: ‘Someone told me you are very bright.’

Pre-test results

‘Yes, this is a significant difference (p<0.01)’

84 participants (53, 31)

Pre-test results

 

 Explicit Implicit Indirect

Non-contextual

‘You are such a nice person!’

‘People like you make this world a better

place.’

‘Someone told me you are very bright.’

‘I wish everyone was as nice as you.’

‘It seems like I’m having a conversation with an intellectual.’

‘My colleague told me you have a great

personality.’

Contextual

‘You are smarter than most respondents!’

‘You seem to know your instruments!’

‘According to our system, you score better than most

people!’

‘You answered really fast!’

‘It seems like you know your politics!’

‘According to our database, you know a

lot about music!’

Experiment Design

Measured variables Liking

Comfort

Information Quality

Contextual Non-contextual

Based on input data Based on ‘nice-to-hear’statements

Related to conversational content Unrelated to conversational content

‘You seem to know your instruments!’

‘People like you make this world a better place.’

Experiment

53 Participants

Conversation with bot Quiz setting

Questionnaire Perceived liking

Perceived level of comfort

Information quality

“How many strings does a guitar have?”“Who is the prime minister of the Netherlands?”

Results

Effects

Results H1. Contextual compliments have a positive effect on the user experience of an ECA.

H1-a. perceived liking of the agent.

H1-b. perceived level of comfort.

H1-c. perceived quality of information.

H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.

H2-a. perceived liking of the agent.

H2-b. perceived level of comfort.

H2-c. perceived quality of information.

H3. Contextual compliments have a more positive effect than non-contextual compliments.

H3-a. perceived liking of the agent.

H3-b. perceived quality of information.

H3-c. perceived level of comfort.

Results H1. Contextual compliments have a positive effect on non-contextual

compliments on the user experience of an ECA.

H2. Non-Contextual compliments have a positive effect on the user experience of an ECA.

H3. Contextual compliments have a more positive effect than non-contextual compliments.

H4. Compliments have a positive effect on the user experience of an ECA.

H4-a. perceived liking of the agent.

H4-b. perceived quality of information.

H4-c. perceived level of comfort.

Conclusion

Compliments have a positive effect on the user experience of an ECA.

However, not on likingness.

The type of compliment matters

Non-contextual > contextual

Discussion

Pros:

Pretest, kind of compliments accounted for.

Pretest in line with experiment outcome.

All experiments under supervision of a test leader.

Improvements:

Sample too small

Language, not native language of all subjects

Variable mount of compliments given

Compliments when correctly answered

Definition of comfort, persuasiveness, likeliness might differ individually

Questions?

Measuring - Liking

‘I would like to be friends with her’

‘If I was upset, she would be able to cheer me

up.’

‘She is a bad listener’

Rated on Likert scale 1-7

Measuring - Level of comfort

During the conversation, I felt:

Comfortable

Involved

Open

Relaxed

Warm

Rated on Likert scale 1-7

Measuring - Information quality

The extent to which the information was:

Correct

Helpful

Persuasive

Insightful

Relevant

Rated on Likert scale 1-7

Context

Controlled environment

Participants from same population