Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen...

16
Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho , Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007

description

Overview: Corpus Study  Study: How well we can predict leadership  Textual and “network” features What are the most predictive features  Large and special collection of s CSPACE corpus (aka GSIA corpus) Leaders (presidents of workgroups) were previously determined

Transcript of Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen...

Page 1: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Predicting Leadership Roles in Email Workgroups

Vitor R. Carvalho, Wen Wu and William W. CohenCarnegie Mellon University

CEAS-2007, Aug 2nd 2007

Page 2: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Motivation

Link to leadership studies in hci Importance of email Leadership at a distance,

communication media, etc.[Butler et al, 2002; Fussell et al, 2001]

Page 3: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Overview: Corpus Study

Study: How well we can predict leadership

Textual and “network” features What are the most predictive features

Large and special collection of emails CSPACE corpus (aka GSIA corpus) Leaders (presidents of workgroups) were

previously determined

Page 4: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

CSPACE corpus 15,000 messages from 277 students Emails associated with a semester-long project (14

weeks) of Carnegie Mellon MBA students To simulate companies competing for market share

and profit, students were divided in 50 companies/teams (4 to 6 students/team)

Very real: student grades largely based (70%) on company financial performance and external board review

Most communication happened inside group Very rich in task negotiation. Presidents assigned in the beginning of the game. Presidents selected other team members through a

round-robin draft.

Page 5: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Email dataset: network

= President…

Team A Team B

Team C Team D

Page 6: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Evidence from email header: network features

Broadcast messages Sent to all other team

members

Non-Broadcast messages Not sent all team

members

Page 7: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Evidence from email header: network features

Page 8: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Textual Features: Email Acts

Cohen et al., EMNLP-04: classification of email content in terms of having “email speech acts”

Examples: Deliver, Request, Commit, Propose, Meeting, etc.

Ciranda: Java toolkit available online

Page 9: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Textual Features: Email Acts

•We also used the associated ranking features, i.e., first (_1), second (_2), last (_last) and one but last (_butlast)•96 features total

Page 10: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Experiments 1

10-fold cross-validation using SVM with linear kernel

Page 11: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Experiments 2:

Restricting to a single president per group, using All Features: 96% of accuracy F1-measure of 0.882. It correctly predicts the president in 30 out of

34 groups (minimum of 20 messages)

Page 12: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Analysis: Feature Selection with 2 test

Page 13: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Analysis: Feature Selection with 2 test

Page 14: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Analysis Overall, results are interesting

Suggests some types of evidence are correlated with leadership

How “natural” is this dataset? Semester long, 14 weeks No language restriction (flames, arguments, cheering, arrangements,

gossip, etc.) Grade depended on the team financial performance

Leadership & the choice of leader Laboratory setting of work mitigates the reliability of the

conclusions How much self-selection is going on in choosing "President" of

company? Are students behaving according to their expectations of how executives behave?

Presidents were selected on a popular vote: the class as a whole (277 students) elected the 50 presidents.

Page 15: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Related Work

Community structure and leadership roles in email archives of organizations.

[Leuski, 2004][Tyler et al, 2003.]

Very different datasets, methods, validations, etc.

Page 16: Predicting Leadership Roles in Email Workgroups Vitor R. Carvalho, Wen Wu and William W. Cohen Carnegie Mellon University CEAS-2007, Aug 2 nd 2007.

Thank you.