Email Classification
Results forFolder Classificationon Enron Dataset
Overall Goals
To help users managelarge volumes of email.
…by helping them to sorttheir email into folders.
Immediate Goals
To establish an credible test corpus
To create baseline results for email classification
To analyze possible future techniques
The “Enron” Corpus
Previous email classification experiments have used “toy” collections.
Enron emails are collected from actual business users.
Made public through legal proceedings.
The Enron Corpus
158 users 200,399 emails Average of 757 emails per user
Messages per User
1
10
100
1000
10000
100000
0 20 40 60 80 100 120 140Users
# o
f Mes
sage
s
Enron Data Analysis
Most users do use folders to classify their email. Some users with many emails still have few folders. Users with more emails tend to have more email in
each folder.
Correlation of Folders and Messages
020406080
100120140160180200
1 10 100 1000 10000 100000# of messages
# o
f fol
ders
Representation From To, CC Subject Body Date/Time? Thread? Attachments? etc…?
Approaches Using a bag-of-words
email data “bag of words” SVM classificationdecision
Approaches Using separate SVMs for each section
email data
SVMs
classificationdecision
LLSF
Approach Data was split in half, chronologically.
A “flat” approach was used. (not hierarchical)
An SVM was trained for each folder for each user for each field.
The SVM for each folder was trained using all of the emails for that user.
Combination weights were found with a regression for each folder.
Thresholding was performed for optimal F1 score, using the “scut” method.
“Enron” Results Analysis
Obviously some data fields are more useful than others. Unsurprisingly, the “To, CC” data is the least useful. Body is the most useful field, followed closely by sender. Using all fields works better than using any particular field alone. Linearly combining fields works better than bag-of-words approach. Because it’s SVM, the linear weights are not directly interpretable.
F1 Scores for Enron Dataset
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
Avg. Micro Avg. F1 Avg. Macro Avg. F1
FromSubjectBodyTo, CCAllLinear Comb.
Enron Results Analysis
F1 classification score is unrelated to the number of emails a user has.
Message Count vs. Micro Average F1
0
0.2
0.4
0.6
0.8
1
1.2
1 10 100 1000 10000 100000messages
Message Count vs. Macro Average F1
0
0.2
0.4
0.6
0.8
1
1.2
1 10 100 1000 10000 100000messages
Enron Results Analysis
F1 score is somewhat correlated with the number of folders a user has.
Emails are much harder to classify for users with many folders.
Folder Count vs. Macro Average F1
0
0.2
0.4
0.6
0.8
1
1.2
1 10 100 1000folders
Folder Count vs. Micro Average F1
0
0.2
0.4
0.6
0.8
1
1.2
1 10 100 1000folders
Enron Thread Analysis
200,399 messages 101,786 threads 30,091 non-trivial threads 61.63% messages are in non-trivial threads Average of 4.1 messages/thread Median of 2 messages/threadthread size: 2 3 4 5 6 7 8 9 10 (10-20] (20-30] (30-40] (40-50] 51+# of threads 16736 4782 3049 1282 879 903 378 214 178 1260 209 79 54 88
Enron Thread Analysis Largest threads are most potentially useful.
But, the largest threads are the least common.
Threads are also redundant with other kinds of evidence.Since threads are detected by subject and sender, much of the thread information is redundant. Also, emails in the same thread tend to have similar bodies.
Largest thread in the Enron corpus is 1124 copies of the same message…all in the “Deleted Items” folder for a particular user!
thread size: 2 3 4 5 6 7 8 9 10 (10-20] (20-30] (30-40] (40-50] 51+# of threads 16736 4782 3049 1282 879 903 378 214 178 1260 209 79 54 88
Top Related