SDOW (ISWC2011)

DIGITAL Institute for Information and Communication Technologies

Pragmatic metadata matters:How data about the usage of data affects

semantic user modelsClaudia Wagner, Markus Strohmaier, Yulan He

Sunday, October 23, 2011

2

ExampleSemantic Metadata

sioc:UserAccount

rdf:type

sioc:name

sioc:Post

rdf:type

sioc:content

sioc:has_creator

foaf:Personsioc:account_of


3

ExamplePragmatic Metadata


4

Aim

sioc:UserAccount

rdf:type

sioc:name

sioc:Post

rdf:type

sioc:topic

sioc:content

sioc:has_creator

foaf:Personsioc:account_of

?foaf:interest

?

Can pragmatic metadata support the generation of semantic metadata and if yes how?


5

Experimental Setup§ Methodology

§ Topic Modeling Algorithms to learn topics (probability distributions of words) and annotate users and posts with topics

§ Incorporated different types of pragmatic metadata into the Topic Models

§ Compared different models via their predictive performance

§ Dataset§ Boards.ie§ Forums, Posts and Users§ User`s authoring and replying behavior

§ Training Dataset: First and last week of February 2006§ Test Dataset: 3 future posts of each user


6

Evaluation

§ Compare different models by testing their predictive performance on held out posts.

§ Assumption: a better user topic model reacts less perplex on future posts authored by a user and needs less trainings samples.

Sum over all words in a user`s future post

Log Likelihood of a word of user`s future post given the model we learned


7

MethodologyLDA

§ How to learn topics and annotate users with topics?

§

T1 T2 T3

T1:mac: 0.3iMac: 0.13PC: 0.03computer: 0.04....

Text

Latent Dirichlet Allocation (LDA) (Blei et al, 2003)


8

MethodologyDMR

§ How to incorporate metadata into topic models?

§ Dirichlet Multinomial Regression (DMR) Topic Models (Mimno et al, 2008)

§ Observe feature vector x per document§ Draw „fresh“ alpha for each document which depends

on observed features x and the feature distribution per topic λt

∝dt= exp(λt Xdt)


9

Methodology

ID Alg Doc Metadata

M1 LDA Post -M2 LDA User -

M3 DMR Post author

M4 DMR User author

M5 DMR Post reply-user

M6 DMR User reply-user

M7 DMR Post related-user

M8 DMR User related-user

authored Post 1

Post 2

Post 3

Post 4

Post 5

Post 6

replies toUser 1

User 2

authored

Post 7 Future

Past


10

§ Different user activities performed on content

Baseline LDA (M1 and M2)

Post training scheme (M3, M5 and M7)

Models which take user replies into account.(M6 and M8)


11

Results

ID Alg Doc Metadata

M1 LDA Post -M2 LDA User -

M3 DMR Post author

M4 DMR User author

M5 DMR Post reply-user

M6 DMR User reply-user

M7 DMR Post related-user

M8 DMR User related-user

authored Post 1

Post 2

Post 3

Post 4

Post 5

Post 6

replies toUser 1

User 2

authored

Post 7 Future

Past


12

Results§ The topics of users who reply to a user are also likely for

this user§ Therefore, if 2 users get replies from the same users

than they are more likely to talk about the same topics

§ Topic models which incorporate pragmatic metadata per user can indeed improve models

§ Topic models which incorporate pragmatic metadata per post often over-fit data§ Model Assumptions are too strict!

§ Idea: Incorporate behavioral user similarities§ Intuition: users which are similar are more likely to talk

about the same topics§ How to measure behavioral similarity?

§ forum usage§ communication behavior


13

Methodology

ID Alg Doc Metadata

M9 DMR Post top 10 forums

M10 DMR User top 10 forums

M11 DMR Posttop 10 communication partner

M12 DMR Usertop 10 communication partner

authored

Post 1

Post 2

Post 3User 1

authored Post 4

Post 5

Post 6User 2

f1 f2 f3 f4 f5 f6 f7 f8 f9 f10

f15f20 f31 f12 f5f6 f17 f18 f19 f10

Post 7 Future

Past


14

Baseline LDA (M1 and M2)

Post training scheme (M3, M9 and M11)

User training scheme (M4, M10 and M12)

Models M12 incorporates user similari;es based on their communica;onbehavior


15

Results

§ Topic models seem to benefit from taking behavioral user similarities into account

§ Users who behave similar (regarding their forum usage and communication behavior) are likely to talk about the same topics

§ Common communication-partner seem to be more predictive for common topics than common forums


16

Conclusions§ Pragmatic metadata may help to learn better semantic

user models

§ But pragmatic metadata observed on a post level often over-fits data

§ Pragmatic Metadata on a user level seems to improve the predictive performance of topic models§ If posts of 2 users are “used” in a similar way then

they are more likely to talk about the same topics § If 2 users behave similar (tend to post to same forums

or tend to talk to same users) they are more likely to talk about same topics.

§ Common communication-partner seem to be more predictive for common topics than common forums


17

Limitations and Future Work§ Perplexity and semantic interpretability of topics do not

necessarily correlate (Chang et al., 2009)§ Separate evaluation of semantic coherence of topics

§ Analyzing different types of behavior- and usage-related metadata and explore to what extent they may reveal information about the semantics of data§ behavior on social streams such as Twitter§ tagging behavior§ navigation behavior


18

References

§ David M. Blei, Andrew Ng, Michael Jordan. Latent Dirichlet allocation. JMLR (3) (2003) pp. 993-1022

§ Chang, J., Boyd-graber, J., Gerrish, S., Wang, C. and Blei, D. Reading Tea Leaves: How Humans Interpret Topic Models, Neural Information Processing Systems, NIPS (2009)

§ Mimno, D.M. and McCallum, A. Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression. In Proceedings of UAI. (2008), pp. 411-418


SDOW (ISWC2011)

Education

Transcript of SDOW (ISWC2011)