Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of...
-
Upload
jocelyn-miller -
Category
Documents
-
view
221 -
download
0
Transcript of Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of...
![Page 1: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/1.jpg)
Aspect Guided Text Categorization with Unobserved Labels
Dan Roth, Yuancheng TuUniversity of Illinois at Urbana-Champaign
![Page 2: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/2.jpg)
Text Categorization
An archetypical Multi-Class Classification (MCC) problem F : X → Y a document, d X , a collection of classes Y = {c∈ 1, c2, . . . , cN}
Sports
Health
Business…Science
C1C2C3…CN
2
![Page 3: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/3.jpg)
Motivation: what are we missing?
Class labels (Y) contain information which can help classification
How can we explore the label space?
C1
C2
C3…CN
Sports
Health
Business…Science
3
![Page 4: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/4.jpg)
Car Navigation Command Classification
4
XY
![Page 5: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/5.jpg)
Aspect Variables
5
X
Y
find nearest restaurant
Show me where I can eat nearby
Find nearest restaurantAction Detail Modifier Topic Manner
z1 z2 z3 z4 z5
NullNull
![Page 6: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/6.jpg)
Significance of the Aspect Variables
Predicting better aspects implies predicting better class labels
Adding constraints to the aspect space
Predicting previously unobserved labels
6
If Topic = “restaurant”, then Action ≠ “turn”
Observed Label
1. turn on the radio2. GPS navigation
Unobserved Label
turn on GPS
![Page 7: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/7.jpg)
Outline
Car Command Text Categorization Task Data and aspects
Unobserved labels
Constrained Conditional Model (CCM) Aspects variables to introduce constraints
Objective function
Training and Inference
Experimental Results Standard multiclass classification setting
Predicting Unobserved Labels
Conclusion
7
![Page 8: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/8.jpg)
xX
x1
x6
x2
x5
x4x3
x7
X
Yy1
Adding Constraints by Hidden Aspects
Intuition: introduce structure on hidden variables
![Page 9: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/9.jpg)
Adding Constraints Through Hidden Aspects
xX
x1
x6
x2
x5
x4x3
x7
xX
YZ1
Z2
Z4
Z3
Z5
y1
Use constraints to capture the dependencies
X
![Page 10: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/10.jpg)
Objective Function of CCM
10
Weight Vector for “local” learners
Aspect functions
Penalty for violatingthe constraint.
How far away is y from a “legal” assignment
![Page 11: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/11.jpg)
11
Training and Inference
Learning + Inference (L+I) Ignore constraints during training
Inference Based Training (IBT) Consider constraints during training
References to CCM (aka ILP formulation) Roth&Yih04, Has been shown useful in the context of many NLP problems:
SRL, Summarization; Co-reference; Information Extraction; Transliteration
07; Punyakanok et.al 05,08; Chang et.al 07,08; Clarke&Lapata06,07;
Denise&Baldrige07;Goldwasser&Roth'08; Martin,Smith&Xing'09
![Page 12: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/12.jpg)
12
Learning + Inference
x1
x6
x2
x5
x4x3
x7
Z1Z2
Z5
Z4
Z3
f1(x)
f2(x)
f3(x)f4(x)
f5(x)
X
Y
Learning + Inference (L+I)Learn models independently
![Page 13: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/13.jpg)
13
-1 1 111Y’ Local Predictions
Inference Based TrainingExample: Perceptron-based Global Learning
x1
x6
x2
x5
x4x3
x7
f1(x)
f2(x)
f3(x)f4(x)
f5(x)
X
Y
-1 1 1-1-1YTrue Global Labeling
-1 1 11-1Y’ Apply Constraints:
![Page 14: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/14.jpg)
Outline Car Command Text Categorization Task
Data and aspects
Unobserved labels
Constrained Conditional Model (CCM) Aspects variables to introduce constraints
Objective function
Training and Inference
Experimental Results Standard multiclass classification setting
Predicting Unobserved Labels
Conclusion
14
![Page 15: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/15.jpg)
Car Navigation Command Classification
15
XY
![Page 16: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/16.jpg)
Evaluation Metrics
Standard Accuracy The percentage of correctly labeled examples
Weighted Aspect-based Metric (WAM) A weighted Hamming distance computed at the aspect level
16
![Page 17: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/17.jpg)
Experiments and Evaluation Standard MCC Setting
17
Algorithm Accuracy (%)
WAM(%)
Baseline 67.84 86.14
MCC(L+I) 71.18 89.65Error Reduction (%)
10.39 25.32
Accuracy Fless Kappa
Human Annotation 75% 0.764
![Page 18: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/18.jpg)
Experiments and Evaluation
Standard MCC Setting
18
Aspects
CCM Baseline
Error Reduction(%)
Topic 86.14 81.55 24.88
Action 88.31 82.72 32.35
Manner 89.98 87.35 20.79
Modifier
91.15 89.51 15.64
Detail 92.68 89.59 29.68
![Page 19: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/19.jpg)
Experiments and Evaluation
Predicting Unobserved Labels
19
Algorithm Accuracy (%)
WAM(%)
Baseline 0.00 58.43
MCC(L+I) 28.16 70.27Error Reduction (%)
28.16 28.48
Unobserved Label
turn on GPS
Observed Label
1. turn on the radio2. GPS navigation
![Page 20: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/20.jpg)
Conclusion
Summary
Text Categorization with a meaningful,
structured label space
A model that exploits the structure by
adding hidden aspect variables
Adding constraints and reformulating
the task as a structure prediction
problem
Predicting unobserved new labels
20
![Page 21: Aspect Guided Text Categorization with Unobserved Labels Dan Roth, Yuancheng Tu University of Illinois at Urbana-Champaign.](https://reader034.fdocuments.in/reader034/viewer/2022051416/56649e9f5503460f94ba1644/html5/thumbnails/21.jpg)
Thank You!AND
Questions?
21