1 Daniel Avrahami - Thesis Proposal Who, What, and When: Supporting Interpersonal Communication over...

1 Daniel Avrahami - Thesis Proposal

Who, What, and When:Supporting Interpersonal

Communication over Instant

Messaging Daniel Avrahami

Committee:Scott Hudson (Chair)

Eric HorvitzRobert KrautAlon Lavie


Illustration

• John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.


Illustration

• John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.

Since John is pressed for time, he decides to ignore all incoming messages until after he’s done, leaving Anne unable to finish her task.


Illustration (cont)

Consider now if we were able to:• Accurately predict, based on his activity, that John was

not likely to respond to Anne’s message for some time • Predict, based on past communication patterns, that

Anne and John are co-workers

Such models could be used, for example, to increase the salience of the alert, indicating to John that Anne’s message may deserve his immediate attention


Research goals

• The two main goals of my proposed thesis work are to provide a better understanding of factors affecting IM interaction in its context, and to use this understanding for the creation of predictive statistical models and tools that support IM communication.

• In order to achieve these goals my research will use three complementary steps:


Research goals

• Create accurate models that predict responsiveness to incoming IM, and investigate the factors affecting responsiveness (when)

• Investigate the effect of interpersonal relationships on IM interaction, and create statistical models that use this knowledge to predict relationships (who)

• Use basic properties of human dialogue to provide support for balancing of responsiveness and performance (what)


Background

• Instant Messaging, or IM, is one of the most popular communication mediums today

• No longer a medium only for social communication– 12 billion instant messages are sent each day.

Nearly 1 billion messages are exchanged by 28 million business users [IDC Market Analysis’05]

• Useful in many ways: from quick questions and clarifications, coordination and scheduling, to discussions of complex work [Bradner’99; Nardi’00; Handel’02; Herbsleb’02; Isaacs’02]


Background

• Some characteristics of IM:– Sending messages is “cheap”– People can choose when/whether to

respond– Asynchrony means that people can (and

do) multitask [Nardi’00, Isaacs’02]– Can tell whether a receiver is present

• But…


Background

• Especially in the workplace, means that messages may often arrive at inconvenient times

• Presence is not enough

• Unsuccessful communication can have a negative effect on both sender and receiver– Can disrupt the receiver’s work– Can leave the sender waiting for information– True not only for IM

10

Daniel Avrahami - Thesis Proposal

When:Predicting responsiveness

to IM

11


Background

Want to answer the following question:

• If an instant message were to arrive right now, would the user respond to it? In how long?

12


How can such models help?

sender receiver

intercept alert mask enhance

awareness

message

13


sender


message

receiver


14


sender


message

receiver


15


sender


message

receiver


16


sender


awareness

receiver intercept alert mask enhance

shhhh

17


sender


awareness

receiver

intercept alert mask enhance (carefully)

not now

18


Related work

• Interruptions and disruptions– [Gillie’89 , Cutrell’01 , Hudson’02 , Dabbish’04]

• Interruptibility and cost of interruption– [Horvitz’99 , Horvitz’03, Hudson’03 , Begole’04,

Horvitz’04, Fogarty’05, Iqbal’06]

• Models of presence– [Horvitz’02, Begole’03]

• Responsiveness to Email– [Horvitz’02, Tyler’03]

19


Data collection

20


Data collection

• Created a plugin for Trillian Pro (written in C)– Non-intrusive collection of IM and desktop events

– Why this setup?

21


Data collection (cont)

• What gets collected:

– IM Events• Messages, status changes, etc.

– Desktop Events• Applications, events, etc.

• Each participants records for at least 4 weeks

22


Data collection (cont.)

• Privacy of data– Masking messages

• for example, the message:“This is my secret number: 1234 :-)” was recorded as “AAAA AA AA AAAAAA AAAAAA: DDDD :-)”.

• Temporary masking

– Alerting buddies

– Hashing buddy-names

23


Participants

• 16 participants

– Researchers: 6 full-time employees at an industrial research lab (mean age=40.33)

– Interns: 2 summer interns at the industrial research lab (mean age=34.5)

– Students: 8 Masters students (mean age=24.5)

24


Participants

• Nearly 5200 hours recorded• Over 90,000 messages• Over 400 buddies• 4 participants provided full text

• On average, participants exchanged a message every: 8.1, 2.2, 3.1 minutes

25


Responsiveness

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour

10 min5 min2 min1 min30 sec

26


Responsiveness

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour


50%

27


Responsiveness

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour


92%50%

28


Defining “IM Sessions”

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour


session

92%

29


Defining “Session Initiation Attempts”

0 50 100 150 200 250 300 350 400 450 500

Message Number

Day

Hour


session

used two subsets: 5 minutes (similar to Isaacs’02) and 10 minutes

30


Features

• For every message:– Features describing IM state. including:

• Day of week• Hour• Is the Message-Window open• Buddy status (e.g., “Away”)• Buddy status duration• Time since msg to buddy• Time since msg from another buddy• Any msg from other in the last 5 minutes• log(time since msg with any buddy)• Is an SIA-5

31


Features (cont.)

• For every message:– Features describing desktop state (following

Horvitz et al. Fogarty et al. and others). including:• Application in focus• Application in focus duration• Previous application in focus• Previous application in focus duration• Most used application in past m minutes• Duration for most used application in past m minutes• Number of application switches in past m minutes• Amount of keyboard activity in past m minutes• Amount of mouse activity in past m minutes• Mouse movement distance in past m minutes

32


What are we predicting?

• “Seconds Until Response”– computed, for every incoming message

from a buddy, by noting the time it took until a message was sent to the same buddy

• Examined five responsiveness thresholds– 30 seconds, 1, 2, 5, and 10 minutes

33


Modeling method

• Weka ML toolkit• Features selected using a wrapper-

based selection technique• AdaBoosting on Decision-Tree models• 10-fold cross-validation

– 10 trials: train on 90%, test on 10%– Next we report combined accuracy

34


79.883.8

87.089.4 90.1

0

10

20

30

40

50

60

70

80

90

100

30sec 1min 2min 5min 10min

Predict response within

% A

ccu

rate

Results (full feature-set models)

All significantly better than the prior probability (p<.001)

35


Results (buddy-independent models)

• Previous models used information about the buddy (e.g., time since messaging that buddy)

• Can predict different responsiveness for different buddies– But what if you wanted just one level of

responsiveness?

• Built models that did not use any buddy-related features

36


79.882.5

87.0 89.4 89.3

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

Use

r C

entr

ic

0

10

20

30

40

50

60

70

80

90

100

30sec 1min 2min 5min 10min

Predict response within

% A

ccu

rate

Results (buddy-independent models)

all significantly better than the prior probability (p<.001) BUT not sig. diff. from previous set

37


Not using content is good and bad

• Pros:– Privacy preserving– Easy

• Cons:– Misinterpreting quick responses– Misinterpreting messages that do not need

responses– > Can lead to some (machine) learning

errors

38


Planned work: Content

• Need to be able to answer the following question (when a incoming message arrives):

Is this message the beginning of a new session or part of the previous session?

• I plan to develop a method for answering this question automatically using content analysis

39



40



• Evaluation plans:

– Obtain manual coding of a random subset of segment boundaries identified as correct or incorrect

– Use inter-coder agreement to establish a performance benchmark

– Compare to accuracy of automated approach

41


Planned work: Probabilities

• Create a probabilistic model predicting the likelihood of the arrival of the next message in a session– Given evidence about the session so far– What is the probability of

• Another message within time T ?• Another message within time >= T ?

• Useful in identifying ends of sessions– Good for load balancing (help-desk

scenarios)

42


Planned work: Understanding

• Showed the successful creation of statistical models that predict responsiveness

• A better understanding is needed of the connection between responsiveness and a person’s communication- and work-context

43


Planned work: Understanding

• I plan to investigate in detail the contribution of specific features and the interactions between those features to responsiveness

– Preliminary results suggest, for example, that indications of work fragmentation significantly affect responsiveness

44


Who:Relationships and

communication patterns

45


Relationships and IM communication

• People use IM for both work and social communication

• Availability might depend on relationship

• Wanted to investigate the effect of relationship on basic communication patterns

46


Background

• Relationship type has significant effects on communication, including the quality, purpose and perceived value [Duck’91]

• Cues, such as tempo, pauses, speech rates and the frequency of turns, affect the way in which conversation partners perceive each other [Feldstein’94]

• Frequency affects communication [FTF:Whittaker’94, IM:Isaacs’02]

47


Buddy Coder

• Co-worker (Senior)• Co-worker (Peer)• Co-worker (Junior)• Co-worker (Other)• Friend• Family• Spouse• Significant Other• Acquaintance• Friend & Co-worker• Self• Bot• [Unknown/Unused]

48



Buddy Coder

49



Buddy Coder

50



Buddy Coder

51



Buddy Coder

52


Relationships distribution

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

Researchers Interns Students

Work

Mix

Social

Other

53


Session-level measures

54



# Time Message Text

1 17:42:45 B: Hey [Participant’s name]

2 17:42:56 B: what time does your group get in the AM?

3 17:42:57 P: hey

4 17:43:01 P: usually around 10

5 17:43:25 B: ok

6 17:43:38 B: i want to start circulating the card in the AM

7 17:43:58 P: ok, good idea

8 17:44:02 P: that's for coordinating this

9 17:44:13 B: no problem

10 17:44:27 P: thanks :-)

11 17:44:35 P: sorry bout the typo

12 17:44:38 B: is ok

55



# Time Message Text

1 17:42:45 B: Hey [Participant’s name]

2 17:42:56 B: what time does your group get in the AM?

3 17:42:57 P: hey

4 17:43:01 P: usually around 10

5 17:43:25 B: ok

6 17:43:38 B: i want to start circulating the card in the AM

7 17:43:58 P: ok, good idea

8 17:44:02 P: that's for coordinating this

9 17:44:13 B: no problem

10 17:44:27 P: thanks :-)

11 17:44:35 P: sorry bout the typo

12 17:44:38 B: is ok

Variable Value

Group Student

Relationship Work

Duration 1.88 minutes

Message Count 12

Turn Count 7

Character Count 232

Messages per Minute 6.4

Messages per Turn 1.71

Characters per Message 19.3

Seconds Until First Reply 1 seconds

Minimum Gap (between turns) 1 seconds

Maximum Gap (between turns) 24 seconds

Average Gap (between turns) 12.2 seconds

Time of Day 5:44 pm

56


The effect of relationships

• Used a repeated-measures ANOVA

– Relationship Category (Work, Mix, Social) and Group (Researchers, Interns, Students) were repeated

– Participants and BuddyID modeled as random effects

– Participants nested in Group– BuddyID nested first in Participants, then in Group

– N = 3297

57


Results

58


Summary of Results

• Sessions with Social contacts were longer and with more messages BUT at a significantly slower pace– Maybe giving less attention to these

sessions?

• Sessions with Work contacts were at a faster pace with longer messages– Grounding? Complex concepts?

59


Results: Session length

• Significant effect on Session Duration (p<.001)– Social significantly longer sessions than

both Mix and Work (Work and Mix n.s.)

• Similar effects for– Number of Turns

Number of MessagesNumber of Characters• Duration correlated at >.85

0

1

2

3

4

5

6

7

8

w ork mix social

Relationship

Du

rati

on

(m

inu

tes)

60


Results: Messaging rate

• Significant effect on Messaging Rate (p<.01)– Social significantly slower than Mix (p=.003)– Social marginally slower than Work (p=.078)

• Maximum-Gap (p<.05)Social longer than Work(p=.013)

0

1

2

3

4

5

6

7

w ork mix social

Relationship

Mes

sag

es-p

er-M

inu

te

61


Results: Length of messages

• Significant effect on Message Length (Characters-per-Message) (p<.001)– Work significantly longer than both Social

(p<.001) and Mix (p=.002)

0

5

10

15

20

25

30

35

40

45

w ork mix social

Relationship

Ch

arac

ters

-per

-Mes

sag

e

62


Predicting relationships

63



• How can it be used?– Augmenting IM systems

• Indicators of unavailability• Differential alerts

– Shared with other mediums• E.g. Email

– Provide organizational overview

64


0 (Work)

0 (Work)

1 (Social)

1 (Social)

0 (Work)b1i

b1

b1

b2

b2

0 (Work)

0 (Work)

1 (Social)

0 (Work)

1 (Social)

1 (Social)

0 (Work)

0 (Work)

0 (Work)

< v1,v2,...,vn >

< v1,v2,...,vn >

< v1,v2,...,vn >

< v1,v2,...,vn >

< v1,v2,...,vn >i

i

i

i

SN Buddy

b1i 3 .333 Yes

2 .5 Nob2i

SN nBuddy

ActualSessionVariables


Cross-validation with 16 models (omitting one participant each time)

Nominal Logistic Regression

65


Models performance

• Results from pairs with 2 sessions or more (78% of the data)

Classified as

Work Social

Work40.9%(83)

5.9%(12)

Social14.8%(30)

38.4%(78)

Accuracy: 79.3%

66


Models performance

• Results from pairs with 2 sessions or more (78% of the data)

• Both significantly better than the prior probability

Classified as

Work Social

Work40.9%(83)

5.9%(12)

Social14.8%(30)

38.4%(78)

Accuracy: 79.3%

Classified as

Work Mix Social

Work25.3%(74)

5.1%(15)

2.0%(6)

Mix8.2%(24)

14.7%(43)

7.8%(23)

Social9.6%(28)

17.1%(50)

10.2%(30)

Overall Accuracy: 50.2%Work vs. Rest: 75.1%Social vs. Rest: 63.5%

67


Additional work

• Find a way to distinguish Mix from Social

• Examine the effects of additional aspects of relationships on communication– The effects of physical distance– The effects of buddy-familiarity– The effects of task criticality

68


What:Using content to balance

responsiveness and performance

69


Responsiveness / performance tradeoff

• Users often multitask when using instant messaging [Nardi’00, Isaacs’02, Voida’02]

• Users often have to choose between– Staying on task and being responsive to IM

buddies

• Current solutions typically force users to choose one or the other:– Update ‘away’ messages– Turn off IM client

70


•Quick response - “do you have the figures I need?”

•Leisurely response - “check out www.cnn.com”

•Politely deferred - “ru busy?”

•No response - “going to meeting. ttyl”

Expectations for responsiveness

71


The approach: QnA

• Users ignore, to the best of their ability, the alerts of incoming messages – Transitioning (internally) to being unavailable

• By observing the content of messages, QnA automatically highlights incoming messages that may deserve their attention– In particular, potential questions and answers

72


Demo

• (Good luck with the demo)

73


Why questions and answers?

• A question and an answer form an ‘Adjacency pair’ (Schegloff & Sacks’73)

• From “Arenas of Language Use” by Clark (1992)“Given a first pair part, a second pair part is conditionally relevant, that is, relevant and expectable, as the next utterance.

Once A has asked the question, it is relevant and expectable for B to answer in the next turn.” (p. 157)

74


• QnA listens to incoming and outgoing messages– when an outgoing messages is sent

• if it is a question– remember that expecting a response

– when an incoming messages arrives• if it is a question and/or we are expecting an

answer– wait x seconds to see if user attends to the message– if did not attend then show QnA notification

How does it work?

75


is_a_question?

• Match to list of questions that can be ‘politely deferred’– (are|r) (you|u) there– busy?

• Go through list of rules and look for match– (?|/) at end of sentence– what (is|are|r|were|does|do|did|should|can)– did(|n’t|nt) (i|u|you|he|she|they|we)– (are|r) (you|u)– huh

76


Delaying notifications

• First determine whether the user is engaged in the conversation by listening to these events:– Opens the message window– Clicks on the message window– Types in the message window– Message window was in focus when message

arrived

• Multiple questions from same buddy within same delay period will only get one alert

77


Issues

• Determining that a message contains a question or an answer can be difficult– interleaved conversations– many short messages that comprise a single

turn– loose grammar and spelling

• Gives buddies a way to increase the salience of their messages. what if they abuse it?

78


QnA summary

• QnA: A tool that allows users to stay on task, but still seem responsive to buddies who expect it

• Allows users to transition between work modes– Sits quietly in the background when the

user attends to messages– Only notifies when the user ignores

messages

79


Future work

• Collect feedback from users– A few users who have used QnA for over 2

years now– But would like more users

• Please download QnA from my homepage

• Improve question identification

• Implement ‘ignore list’

80


Conclusions

81


Conclusions

• I have presented planned and completed work on analysis and generation of predictive modeling in support of interpersonal communication over IM:– Completed work along with additional planned

work on predictions of responsiveness to IM– Completed work on analysis and predictions of

interpersonal relationships and their effect on communication

– Completed work on the use of basic properties of human dialogue to allow users to balance responsiveness and performance

82


Contributions

• This work’s contribution to the HCI field will span both theoretical and applied aspects. – From a theoretical point of view, this work will

provide insights into the factors that influence interpersonal communication patterns and responsiveness.

– At the applied level, this work will provide predictive statistical models that can be used in many applications.

– Finally, this work promotes the creation of tools that use knowledge and predictive models generated from naturally occurring interaction.

83


Acknowledgements

• No acknowledgements until my defense!

• Thank you Thi!

• Good luck to Darren, James, Jeff and Laura

84


Thank you.

This work was funded in part by the National Science Foundation under grants IIS 0121560 and IIS

0325351 andthis material is based upon work supported by the Defense

Advanced Research Projects Agency (DARPA) under Contract No. NBCHD030010

1 Daniel Avrahami - Thesis Proposal Who, What, and When: Supporting Interpersonal Communication over...

Documents

Transcript of 1 Daniel Avrahami - Thesis Proposal Who, What, and When: Supporting Interpersonal Communication over...