Challenges and Solutions in Automatically Annotating CMC Data · UKP Lab, Technische Universität...

Challenges and Solutions in Automatically Annotating CMC Data

Torsten Zesch Senior Researcher

UKP Lab, Technische Universität Darmstadt

Associated Researcher

German Institute for International Educational Research, Frankfurt

International workshop:

Building Corpora of Computer-Mediated Communication: Issues, Challenges, and Perspectives

Feb 14–15, 2013

TU Dortmund

2

Challenging Properties of CMC Data

Very heterogeneous (Twitter, Wikipedia, Chat, Discussion forum, …)

No tool fits all

Even access might be challenging

One’s junk is the other’s research interest :)

Many common assumptions do not hold

Non-standard language

“lmao s/o to the cool ass asian officer 4 #1 not runnin my license and #2 not takin dru boo

to jail . Thank u God . #amen”

Standard tools might not work & need to deal with spelling errors, abbreviations, and

variants

Often (relatively) small samples

Hard to build automatic models

3

Working with CMC Data

Mix of manual and automatic tasks

Data

Automatic Analysis

Manual Analysis

4

Wrong tool for the job?

Challenge

5

Availability of Tools & Models

Challenge

6

Comprehensive NLP Framework

Proposed Solution

7

Apache UIMA Unstructured Information Management Architecture

Component-based architecture

Designed to scale

Flexible type system

High-density in-memory data representation

2004 – IBM alphaWorks project

still used e.g. in IBM LanguageWare

2006 – Apache Incubator project

2009 – OASIS Standard

2010 – Full Apache project

2010 – Used in IBM’s Watson

Jeopardy Challenge

http://uima.apache.org

8

Pipeline Architecture

Parser

Writer

Semantic analysis

Syntactic analysis

Morphological analysis

Linguistic preprocessing

Reader

Named entity

tagger

PoS tagger

Stemmer

Lemmatizer

Compound

splitter

Tokenizer

Sentence

splitter

Stopword

tagger

a

bc

Datastore / Results

a

bc

a

bc

a

bc

a

bc

9

Information Flow S

ou

rce D

ocu

men

t C

oll

ecti

on

An

no

tate

d D

ocu

men

t C

oll

ecti

on

Tokeniz

er

Sente

nce S

plit

ter

PO

S T

agger

Documents

Tokens Tokens Tokens

Sentences Sentences

POS

Documents Documents Documents

Module1 Module2 Module3

10

So far, so good … but the framework is empty

11

DKPro Core

Linguistic preprocessing

Tokenization, tagging, parsing, co-reference, n-gram models

Support for various file formats

XML, PDF, WSDL, Wikipedia, TEI, IMS-CWB, ANNIS, …

Already available open source

http://code.google.com/p/dkpro-core-asl/

DKPro Similarity (http://code.google.com/p/dkpro-similarity-asl/)

DKPro Spelling (http://code.google.com/p/dkpro-spelling-asl/)

Coming soon

Information Retrieval, Keyphrase Extraction, Summarization, Lexical Cohesion

DKPro Framework







http://code.google.com/p/dkpro-similarity-asl/






http://code.google.com/p/dkpro-spelling-asl/






12

DKPro

Core

DKPro

Semantics

DKPro

Information

Retrieval

Tokenizer

Part

of Speech

Tagger

Lemmatizer

Named

Entity

Recognition

Parser I/O

Keyphrase

Extraction

Text

Segmenting

Link

Discovery

Text

Summary

Apache

Lucene

Indexing

API Query API Terrier 2

Latent

Semantic

Indexing

Terrier 3

Darmstadt Knowledge Processing

Software Repository

Relatedness Spelling

13

Tokenizer

Part

of Speech

Tagger

Lemmatizer

Named

Entity

Recognition

Parser I/O

Keyphrase

Extraction

Text

Segmenting

Link

Discovery

Text

Summary

Apache

Lucene

Indexing

API Query API Terrier 2

Latent

Semantic

Indexing

Terrier 3

Darmstadt Knowledge Processing

Software Repository

Relatedness Spelling

DKPro

Core

DKPro

Semantics

DKPro

Information

Retrieval

14

Multilinguality

Automatically select the correct language-

specific components

Database Writer

Named Entity Recognition

POS Tagging

Sentence splitting

Tokenization

Text Document Reader

de en

TreeTagger German

TreeTagger English

de

en

15

Other Dimensions

Flexibility

Change I/O or replace tagger without breaking the pipeline

Scalability

Process large amounts of data on Hadoop cluster

Efficiency

Integrated access to large resources, e.g. web-scale language models

Robustness & Accuracy

Not influenced by the choice of the framework

But: UIMA encourages to improve existing tools instead of developing new ones

16

Pro

Large collection of interoperable components

Contra

Programming knowledge needed (only very little)

Not (mainly) targeted towards CMC

Proposed Solution

Comprehensive NLP Framework

17

Cool Tool – Wrong Environment

Challenge

18

Domain Adaptation / Retraining

Challenge

19

Examples

Different smiley styles :) :-) (^_^) ^o #smiley

Different smiley styles : ) : - ) ( ^ _ ^ ) ^ o # smiley

JJ NN NNS : ) : : ) ( SYM SYM SYM ) SYM NN # NN

20

Annotate & Retrain

Proposed Solution

21

Learning Curves

[Steedman, Mark, et al. "Bootstrapping statistical parsers from small datasets." EACL 2003]

Collins-CFG parser LTAG Parser

22

Case Study Retraining Dependency Parsers for German

All the work done by Irina Alles

Joint supervision with Chris Biemann

4 Parsers

MaltParser

MateTools Parser

Stanford Parser

MDParser (DKFI inhouse)

23




4 Parsers

MaltParser

MateTools Parser

Stanford Parser


24




4 Parsers

MaltParser

MateTools Parser

Stanford Parser


2 annotated corpora

TIGER Treebank (50,000 sentences of German newspaper text)

TueBa-D/Z treebank (65.524 sentences of German newspaper text)

MaltEval using CoNLL evaluation format (LA – labelled accuracy)

25

Training Time

26

Testing Time

27

Parsing Accuracy

28

Parsing Accuracy – Ensemble Parser

29

Annotate & Retrain

Proposed Solution

Pro

Often remarkably few training instances are necessary

Contra

Re-trainability of tools needs to be improved

Some tools behave weird when trained on data that is too different

30

I am in a special situation …

Challenge

31

Proposed Solution

Build special tools

32

Retraining

Tool stays exactly the same – new behavior due to new training data

Customization

Customize tool when retraining is not enough

Build special tools

Proposed Solution

Enable researchers to customize tools

33

Use Case – Twitter Tagger

Mixture of customized tool and retraining

Customized tokenizer

Retrained tagger

34



Different smiley styles :) :-) (^ ^) ^o #smiley

A A N E E E EMO EMO HASH

ADJ ADJ NN EMO EMO EMO E E #



ADJ NN NN O O O O O O O O O O O NN O NN

35









Different Tokenization

36









Specialized Tagset.

37

Use Case – Twitter Tagger Retrained Model

Spending the day withhh mommma !


V D N P N ,


VVG DT NN NN NNS SENT

Twitter Tokenizer +

Twitter Tagger

Standard Tokenizer +

TreeTagger

38



V D N P N ,



Twitter Tokenizer +

Twitter Tagger


TreeTagger

Different Tagsets


39



V D N P N ,



Twitter Tokenizer +

Twitter Tagger


TreeTagger

Better model Better Tagging


40



V D N P N ,

V ART NN PP NN PUNC



V ART NN NN NN PUNC

Twitter Tokenizer +

Twitter Tagger


TreeTagger

Mapping to generalized POS models


41

Pro

Very flexible

Contra

Still too complicated for non programmers

Build special tools

Proposed Solution

Enable researchers to customize tools

42

Availability of Tools & Models

DKPro framework

Domain Adaptation / Retraining

a little annotation can get you a long way

Customization

active field of research in our Lab

interesting research project opportunities

Summary

Challenges and Solutions in Automatically Annotating CMC Data · UKP Lab, Technische Universität...

Documents

Transcript of Challenges and Solutions in Automatically Annotating CMC Data · UKP Lab, Technische Universität...