November 10, 2006 I2B2 - Smoker Status Challenge 1

Determining Smoker Status using Supervised and Unsupervised Learning with Lexical Features

Ted PedersenUniversity of Minnesota, Duluth

tpederse@d.umn.eduhttp://www.d.umn.edu/~tpederse

November 10, 2006 I2B2 - Smoker Status Challenge 2

Approaches

• Smoking Status as Text Classification– supervised learning– lexical features– techniques used to good effect in word sense

disambiguation

• Smoking Status as Text Clustering– unsupervised learning– lexical features – techniques used to good effect in word sense

discrimination

November 10, 2006 I2B2 - Smoker Status Challenge 3

Objectives

• How well do WSD techniques generalize to related but different problems?– smoking status as "meaning" of record??– not quite the same problem…

• How well do WSD features generalize? – bag of words, unigrams– bigrams– collocations

• How well do learning algorithms generalize?– supervised and unsupervised

November 10, 2006 I2B2 - Smoker Status Challenge 4

Experimental VariationsSupervised Learning

• Learning Algorithm – naïve Bayesian classifier– J48 decision tree– support vector machine (SMO)

• Feature Sets (also used in unsupervised)– unigrams, bigrams, trigrams– various frequency and measure of association cutoffs– Stop List of 472 words

• 392 function words• 80 words that occurred in more than half the records

November 10, 2006 I2B2 - Smoker Status Challenge 5

Decision Tree

• J48 most accurate when using unigram features that occurred 5 or more times in the training data– over 3,600 unigrams as candidate features– decision tree has 47 nodes and 24 leaves– accuracy of 82% (327/401)

November 10, 2006 I2B2 - Smoker Status Challenge 6

Decision Treeunigrams : 5 or more times

November 10, 2006 I2B2 - Smoker Status Challenge 7

82% accuracy (327/398)10-fold cross validation on train

a b c d e <-- classified as

20 5 1 7 3 | a = PAST-SMOKER

8 46 3 8 1 | b = NON-SMOKER

8 2 240 2 0 | c = UNKNOWN

7 5 1 21 1 | d = CURR-SMOKER

1 3 1 4 0 | e = SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 8

Manual Inspection

• From the decision tree learned from the 3,600 features, we decided to use the following in a second experiment:– cigarette, drinks, quit, smoke, smoked,

smoker, smokes, smoking, tobacco

November 10, 2006 I2B2 - Smoker Status Challenge 9

9-feature Decision Treeselected from unigram tree

• quit = 0• | smoking = 0• | | smoker = 0• | | | tobacco = 0• | | | | smoke = 0• | | | | | drinks = 0• | | | | | | cigarette = 0• | | | | | | | smoked = 0: UNKNOWN (253.0/3.0)• | | | | | | | smoked = 1: PAST-SMOKER (2.0/1.0)• | | | | | | cigarette = 1: NON-SMOKER (3.0/1.0)• | | | | | drinks = 1: NON-SMOKER (6.0/3.0)• | | | | smoke = 1: NON-SMOKER (16.0)• | | | tobacco = 1• | | | | smokes = 0: NON-SMOKER (39.0/7.0)• | | | | smokes = 1: CURRENT-SMOKER (2.0)• | | smoker = 1: CURRENT-SMOKER (11.0/5.0)• | smoking = 1: CURRENT-SMOKER (42.0/22.0)• quit = 1: PAST-SMOKER (24.0/4.0)

November 10, 2006 I2B2 - Smoker Status Challenge 10

9-feature Decision Tree87% accuracy (345/398)

10 fold cross validation on train

a b c d e <-- classified as

20 5 1 10 0 | a = PAST-SMOKER

0 51 2 13 0 | b = NON-SMOKER

0 1 250 1 0 | c = UNKNOWN

5 4 2 24 0 | d = CURR-SMOKER

0 3 1 5 0 | e = SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 11

9-feature Decision Tree 82% accuracy (85/104)

evaluation data

a b c d e <-- classified as

62 0 1 0 0 | a = UNKNOWN

1 10 1 0 4 | b = NON-SMOKER

0 2 4 0 5 | c = PAST-SMOKER

0 0 0 0 3 | d = SMOKER

0 1 1 0 9 | e = CURR-SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 12

9-feature Decision Tree90% accuracy (94/104)

evaluation data

a b f <-- classified as

62 0 1 | a = UNKNOWN

1 10 5 | b = NON-SMOKER

0 3 22 | f = ALL-SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 13

Unsupervised Experiments

• Bigram Features– allow up to 5 intervening words– occur 2 or more times in training data– limit to those that include "smok" --> 96 features– social smoking, pack smoking, smoking alcohol,

smoking family, smoke drink, cigarette smoking, allergies smoking, allergies smoked, smoking quit, quit smoking, smoker drinks, former smoker, social smoke, denies smoking, habits smoking, ...

November 10, 2006 I2B2 - Smoker Status Challenge 14

Unsupervised Context Representations

• 2nd order Context Representations– Latent Semantic Analysis, native SenseClusters– each record represented by a vector that is the

average of vectors that represent the individual features :

• LSA– each bigram is replaced by a vector showing the

records in which it occurs

• native SenseClusters– each word is replaced by a vector showing the

second words it occurs with as a bigram

November 10, 2006 I2B2 - Smoker Status Challenge 15

Unsupervised Clustering

• Once vectors for all records are created, they are clustered using a partitional method similar to k-means

• The number of clusters is automatically discovered using the PK2 measure, which compares successive values of clustering criterion function

• assign clusters to categories based on distribution in training data – unknown, non-smoker, past-smoker,

current-smoker, smoker

November 10, 2006 I2B2 - Smoker Status Challenge 16

SenseClusters69% accuracy (72/104)

evaluation data a b c d e <-- classified as

63 0 0 0 0 | a = UNKNOWN

10 0 0 0 6 | b = NON-SMOKER

2 1 0 0 8 | c = PAST-SMOKER

1 0 0 0 2 | d = SMOKER

2 0 0 0 9 | e = CURR-SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 17

SenseClusters79% accuracy (82/104)

evaluation data

a b f <-- classified as

63 0 0 | a = UNKNOWN

10 0 6 | b = NON-SMOKER

5 1 19 | f = ALL-SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 18

Latent Semantic Analysis68% accuracy (71/104)

evaluation data

a b c d e <-- classified as

63 0 0 0 0 | a = UNKNOWN

10 0 0 0 6 | b = NON-SMOKER

1 3 0 0 7 | c = PAST-SMOKER

1 0 0 0 2 | d = SMOKER

2 1 0 0 8 | e = CURR-SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 19

Latent Semantic Analysis77% accuracy (80/104)

evaluation data

a b f <-- classified as

63 0 0 | a = UNKNOWN

10 0 6 | b = NON-SMOKER

4 4 17 | f = ALL-SMOKER

November 10, 2006 I2B2 - Smoker Status Challenge 20

Conclusions

• Results dominated by UNKNOWN – sets lower bound of 61%

• Errors dominated by confusion in ALL-SMOKER – reduction to 3 classes improves results significantly

• Decision tree aided feature selection• Manual tuning of feature sets performed since

records focus well beyond smoking status• Unsupervised clustering found "right" number of

clusters perhaps, did well in that light

November 10, 2006 I2B2 - Smoker Status Challenge 21

Software Resources

• Supervised Experiments– SenseTools (free, from Duluth)

http://www.d.umn.edu/~tpederse/sensetools.html

– Weka (free, from Waikato)

http://www.cs.waikato.ac.nz/ml/weka/

• Unsupervised Experiments– SenseClusters (free, from Duluth)

http://senseclusters.sourceforge.net