Poincaré Embeddings for Learning Hierarchical

RepresentationsJuly 4, 2017

Tatsuya Shirakawa

ABEJA Inc.

CONFIDENTIAL

Tatsuya Shirakawa

CONFIDENTIAL

Today’s Paper

Paper Stats• Guys from FAIR

• Sumitted to arXiv at May 26, 2017

https://arxiv.org/abs/1705.08039

• Sumitted to NIPS2017?

Key Contributions• Introducing hyperbolic geometry

(Poincaré disk model) into word/graph

embeddings paradigm

• Automatically capture hierarchical

structure of data

• Achieved incredible better results than

previous works.

CONFIDENTIAL

1. Problems

2. Hyperbolic Geometry

3. Poincaré Embeddings(and Some Incredible Results)

Agenda

4

CONFIDENTIAL

Problems

5

CONFIDENTIAL

Find good representation(embedding) of items such that underlying hierarchical relation structure are well reconstructed

The Problem

CONFIDENTIAL

Embed nouns in WordNets such that related nouns are close in embedded space

Taxonomy Embedding

7

http://www.nltk.org/book_1ed/ch02.html

CONFIDENTIAL

Embed nodes in given graph such that missing links are well-reconstructed

Graph Link Prediction

8

http://ml.cs.tsinghua.edu.cn/~jiaming/publications/

CONFIDENTIAL

Back Theory

9

CONFIDENTIAL

• Geometry with negative curvature

• Many models (realizations):- Poincaré half space model- Poincaré disk model

…each is isometric

Hyperbolic Geometry

10

CONFIDENTIAL

• Defined on upper half spacewith metric

• Distance btw points is

Poincaré Half Space Model

11

CONFIDENTIAL

12

Tree representation in H

https://arxiv.org/abs/1006.5169

• Tree structure is well

represented in Poincaré

half space

CONFIDENTIAL

• A realization of hyperbolic geometry

• Defined onequipped with metric of

• Distance btw points is

Poincaré Disk Model

13M.C. Escher's Circle Limit III, 1959

CONFIDENTIAL

(for simplicity: 2-dim, identify as )

Relation to Poincaré Half Space Model

14https://arxiv.org/abs/1006.5169

CONFIDENTIAL

• Euclidean space is too narrow to embed hierarchical (tree) structures

Why not Euclidean Space?

15

Surface Area

/ # of leaf nodes

Volume

/ # of nodes

Euclidean Ball O(R^n) O(R^n)

b-ary tree O(b^R) O(b^R)

※ R=radius of ball or depth of tree

CONFIDENTIAL

• b-array tree can be interpreted as discrete analogue of Poincaré disk

Why Hyperbolic Space?

16

CONFIDENTIAL

• Hyperbolic space is far more appropriate than Euclidean space to represent hierarchical structure

• Many equivalent models- Poincaré half space model- Poincaré disk model…

Conclusion Here

17

CONFIDENTIAL

• R. Kleinberg, “Geographic routing using hyperbolic spaces”, 2007

• M. Boguna et al., “Sustaining the internet with hyperbolic mapping”, 2010

• P. D. Hoff et al., “Latent space approaches to social network analysis”, 2016

• A. B. Adcock et al., “Tree-like structure in large social and information networks’, 2013

• D. Krioukov et al., “Hyperbolic geometry of complex networks”, 2010

Prior Works around hyperbolic geometry applications

18

CONFIDENTIAL

Poincaré Embeddings

19

CONFIDENTIAL

1. Parametrize each item in Poincaré ball

2. Optimize them by Riemannian optimization under metric of

Proposed Method

CONFIDENTIAL

1. Compute stochastic (Euclidean) gradient

2. Correct metric

3. Apply GD

4. Project onto space

Riemannian SGD

21

CONFIDENTIAL

Embed nouns in WordNets such that related nouns are close in embedded space

Taxonomy Embedding

22

http://www.nltk.org/book_1ed/ch02.html

CONFIDENTIAL

Maximize

Reconstruction setting:- D is full relations

Prediction setting

- D is subset of full relations

Objective Function

23

randomly chosen 10 negative samples

CONFIDENTIAL

Result

24

CONFIDENTIAL

25

CONFIDENTIAL

Embed nodes in given graph such that missing links are well-reconstructed

Graph Link Prediction

26

http://ml.cs.tsinghua.edu.cn/~jiaming/publications/

CONFIDENTIAL

Minimize the cross entropy of probability

Objective Function

27

CONFIDENTIAL

Result

28

CONFIDENTIAL

• Poincaré embeddings automatically capture hierarchical structure from data

• Riemannian SGD provides the way to optimize Poincaré embeddings

• Achieved quite good results on word/graph embedding tasks

Summary

29