Post on 30-Jun-2020
Structural pattern recognition
Romain Raveauxromain.raveaux@univ-tours.fr
Maître de conférences
Université de Tours
Laboratoire d’informatique (LIFAT)
Equipe RFAI
Optimisation combinatoire et apprentissage structurel pour
l'appariement et la classification de graphes
Contributions and perspectives
Contributions and perspectives on combinatorial optimization
and machine learning onto graph space
Application to : graph matching and graph classification
Presentation : Romain Raveaux
• CV
• Teaching
• Research
• Projects
• Administrative tasks
R. Raveaux 4
CV
• Diplomas and qualification
2010-2011
CNU Qualification : 27
2006-2010
Doctorat de l’Université de La Rochelle. Mention : Très honorable avec félicitations
Fouille et classification de graphes : Application à l’analyse d’images cadastrales couleurs.
2004-2006
Master Génie Informatique, Université de Rouen.
Option : Extraction et Indexation de l’information.
2004-2006 :
Master Génie Électrique et Informatique Industrielle,Université de Rouen.
Option : Réseaux et Télécoms.
R. Raveaux 5
CV
• Since 2012, teacher at Université de Tours• At Polytech’Tours (Engineering school)
• Computer Science department
• Since 2012, researcher at LIFAT• Computer science laboratory
• Team : Pattern recognition and image analysis
• 2011-2012 : Research engineers at SOOD company
• 2010-2011 : Attaché Temporaire d’Enseignement et de Recherche à l’IUT d’informatique de La Rochelle
R. Raveaux 6
Teaching
• Teaching : 241 h EqTD
• Projects : 30 h EqTD
• Responsability : 25 EqTD
R. Raveaux 7
Research
• Topic : Machine learning, discrete optimization, computer vision
• Supervisor of : 3 PhD students and 3 master students
• 15 journal papers and 25 conferences.
• Projects : members of 3 projets• ALPAGE, CARAMBA, VISIT
• Collaboration : National (GREYC, LITIS, LORIA). International (Campinas,Brazil, Griffith University, Australia)
• Scientific committee: GBR, past: GREC
R. Raveaux 8
Content
• Introduction• Title explanation• Data• Graph-based applications• Applicative Tools
• Combinatorial optimization for graph matching and graph classification• Graph matching• Graph classification
• Machine learning in graph space for matching and classification• Graph matching• Graph classification
• Conclusion• Summary• Perspective
R. Raveaux 9
Title explanation
• Graph matching in a single picture :
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux 10
Title explanation
• Graph classification in a single picture :
1
2
3
G
Class 4Classifier
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux 11
Title explanation
• Combinatorial optimization in a single picture :
A1 B2
3 C D
1-A 1-B 1-C
root
2-A 2-B
3-B
G2G1
1-D
3-C
3-C
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux 12
Title explanation
• Machine learning in a single picture : Decision processData
Data set 1
Decision
Learningprocess
Model
Data set 2
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph based applications - Applicative Tools
R. Raveaux 13
Title explanation
• Graph space in a single picture :
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data - Graph-based applications - Applicative Tools
R. Raveaux 14
Data
• Data are crucial for computer scientists
• Data driven :• problems
• models
• solution methods (solving methods)
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux 15
Type of Data
Taken from M. Bronstein. CVPR Tutorial 2017 16
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Focus on structured Data
• String
• Tree
• Graph
17
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
String
Taken from: Marçal Rusiñol et al: Symbol spotting in vectorized technical drawingsthrough a lookup table of region strings. PAA 2010
18R. Raveaux
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
Tree
R. Raveaux et al:Structured representations in a content based image retrieval context. J. Visual Communication and Image Representation 24(8): 1252-1268 (2013)
19R. Raveaux
Graph
20
• Data are represented as graphs:• By nature (Social Network, Molecule, Protein interaction network)
• By construction (from images)
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph
21
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
A graph can be represented by 4-tuple 𝐺 = (𝑉, 𝐸, 𝜇, 𝜉), with:
V the set of vertices,
𝐸 ⊆ 𝐸 ∩ 𝑉 × 𝑉 the set of edges,
𝜇: 𝑉 → 𝐿𝑉 the function that assigns attributes to vertices,
𝜉: 𝑉 → 𝐿𝐸 the function that assigns attributes to edges,
𝐿𝑉 the set of all possible attributes for vertices,
and 𝐿𝐸 the set of all possible attributes for edges.
R. Raveaux
Graph
22
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Region Adjacency Graph
23
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Region Adjacency Graph
Impact of noise on Graph-Based Representation 24
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Neighborhood graph
25
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Skeleton Graph
26
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph of molecules : Chemoinformatics
27
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Domain structure vs Data on a domain
Taken from [Brontein 2016, CoRR] 28
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Fixed vs different domain
29
1D,2D, 3D shapes(Different graphs)
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Structured data
• We will focus on graphs:• Graph as a generalization of Euclidean data : vector, matrix, tensors ….
• Graphs as a generalization of strings and trees.
• By nature, data are more likely to be graphs.
30
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph-based applications
• Graph classification/clustering/regression
• Vertex classification/clustering/regression
• Graph matching
• Graph distance
• Graph-based search• Subgraph search• Subgraph spotting• Similarity search
• Graph prototypes• Median graphs/ Super graphs
Artificial Intelligence
is the set of tools to solve these problems : Modelisation, Machine learning, Optimization … …
31
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph classification
32
1. cancerous or not cancerous molecules
2. determination of the boiling point
Molecular graph
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Vertex classification
Taken from [Brontein 2016, CoRR] 33
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Vertex classification/clustering
Taken from [Brontein 2016, CoRR] 34
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Semi-supervised vertex classification
• Setting: • Some nodes are labeled (black circle)
• All other nodes are unlabeled
• Task: • Predict node label of unlabeled nodes
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux 35
Graph matching
• When parts of the object must be tracked or compared.
• Detect and recognize at once
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux 36
Graph matching (https://goo.gl/8dYCZb)
• Mettre vidéo graph matching
37
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph matching (https://goo.gl/wi5m1E)
38
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph distance/similarity
How similar are theses graphs ?
39
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph prototype
d1d2
d3
40
G2
G3
G1
Gm
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
Graph-based search
q G2
G1
G3
G4
G5
G6
41
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Applicative Tools
R. Raveaux
What do we need to solve these problems ?
• Modelisation, Machine learning, Optimization, …:• Graph comparison :
• Graph matching
• Graph distance
• Learning capability with graphs :• Learn to match
• Learn to classify
42
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Tools
R. Raveaux
Two ways to solve the problems
• Graph/Node embedding : [Luqman et al., 2017, PR], [Kipf and Welling, 2017, ICLR].• The graphs/nodes are projected into a vector space
• Graph space : [Neuhaus and Bunke., 2007, MPAI], [Riesen, 2015, ACVPR].• The graphs are compared through graph matching methods.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Tools
R. Raveaux 43
Graph space vs graph embedding
Graph space Graph embedding
Graphcomparison
Pros • Manage structure and features at once
• Preserve the structure of graphs
• Fast algorithms available• Many learning techniques
Cons • Slow (Often) a combinatorialproblem
• Lack of learning techniques
• Does not capture the combinatorialnature of the problem
• Loss of topological information
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Tools
R. Raveaux 44
Our positionning
• Many works have focus their interest on graph embeddings.• Review on graph embeddings : [Goyal and Ferrara, 2017, KBSyst][Cai et al,
2017, IEEE TKDE].
• We think it is unfortunate to reduce the graphs into vectors.
• We explore new avenues:• It is interesting to develop learning techniques in graph space.
• It is interesting to design optimization methods in graph space.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Tools
R. Raveaux 45
Content
• Combinatorial optimization for graph matching and graph classification• Graph matching
• Graph classification
• Machine learning in graph space for matching and classification• Graph matching
• Graph classification
Introduction : Combinatorial Optimization : Machine Learning : ConclusionTitle explanation – Data- Graph-based applications - Tools
R. Raveaux 46
Combinatorial optimization for graph matchingand graph classification• Learning free methods.
• Pure combinatorial problems
Introduction : Combinatorial Optimization : Machine Learning : ConclusionProblems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 47
Graph matching
• Problems
• State of the art
• Deadlocks
• Contributions
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 48
Graph matching : Problems
• Exact matching
• Error-Tolerent matching• Subgraph matching
• Error-correcting matching
• can be expressed as QAP, MAP-inference, GED problems.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 49
Graph matching : the Graph Edit Distance (GED) Problem
2
1
G
3
4
a
c
b
G’
2 3
4
a 3
4
a b
4
a b
c
This solution costs:
𝑑 =
𝑜𝑝𝑖 ∈ 𝑜𝑝𝑒𝑟𝑎𝑡𝑖𝑜𝑛𝑠
𝐶(𝑜𝑝𝑖)
Deletion Substitution Substitution Both
Vertices operations
• OP1: 1 ϵ C1
• OP3: 2 a C2
• OP4: 3 b C3
• OP6: 4 c C4
Edges operations
• OP2: (1,2) ϵ C5
• OP5: (2,3) (a,b) C6
• OP7: (3,4) (b,c) C7
• OP8: (2,4) ϵ C8
The goal is to find 𝑑min the set of operations with the minimum cost.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
Graph matching : the Qadratic AssignmentProblem
𝜙 𝑖 = 𝑘𝜙 𝑗 = 𝑙𝐷𝑖,𝑘,𝑗,𝑙 = 𝑐 𝑖𝑗, 𝑘𝑙𝐵𝑖,𝑘 = 𝑐 𝑖, 𝑘𝑁 = 𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑛𝑜𝑑𝑒𝑠 = 2
i
j
k
l
G1 G2
ij kl
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 51
Graph matching : MAP-inference of a ConditionalRandom Field (CRF)Finding the most likely configuration of discrete CRF. CRF=G1
i
j
k
l
G1 G2
ij kl
𝑥𝑖 𝑎 𝑑𝑖𝑠𝑐𝑟𝑒𝑡𝑒 𝑣𝑎𝑟𝑖𝑎𝑏𝑙𝑒 𝑎𝑠𝑠𝑜𝑐𝑖𝑎𝑡𝑒𝑑 𝑡𝑜 𝑛𝑜𝑑𝑒 𝑖𝑥𝑖 𝑐𝑎𝑛 𝑡𝑎𝑘𝑒 𝑖𝑡𝑠 𝑣𝑎𝑙𝑢𝑒 ∈ 𝑘, 𝑙𝑥𝑖𝑗 𝑡𝑎𝑘𝑒𝑠 𝑖𝑡𝑠 𝑣𝑎𝑙𝑢𝑒 ∈ 𝑘, 𝑙 × {𝑘, 𝑙}
𝑉 = 𝑖, 𝑗 𝑎𝑛𝑑 𝐸 = 𝑖𝑗𝜃 𝑥𝑖 = 𝑐𝑜𝑠𝑡 𝑡𝑜 𝑎𝑠𝑠𝑜𝑐𝑖𝑎𝑡𝑒 𝑖 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑙𝑎𝑏𝑒𝑙 𝑥𝑖𝜃 𝑥𝑖𝑗 = 𝑐𝑜𝑠𝑡 𝑡𝑜 𝑎𝑠𝑠𝑜𝑐𝑖𝑎𝑡𝑒 𝑖𝑗 𝑤𝑖𝑡ℎ 𝑡ℎ𝑒 𝑙𝑎𝑏𝑒𝑙 𝑥𝑖𝑗
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 52
Graph matching : Related Problems
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
Errot-tolerant graph matching : NP-hard problem. [Zeng et al. 2009, PVLDB]: Combinatorial problem
R. Raveaux 53
Graph matching models
• Graph matching can be expressed by : • Integer Quadratic Program
• Integer Linear Program
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 54
i
j
k
l
G1 G2
ij kl
Graph matching models : Integer QuadraticProgram
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 55
Graph matching models : Integer Linear Program
• Linear constraints
• Linear objective function
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 56
Graph matching : State of the art
Subgraph GM problem Methods
Models
Exact Heuristique
ILP #papers : 0 #papers : 0
IQP #papers : 0 #papers : 7IPFP[Leordeanuet al., 2009]
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 57
Graph matching : State of the art
Error-correcting GM problem
Methods
Models
Exact Heuristique
ILP #papers : 0 #papers : 1ILP [Justice andHero, 2006]
IQP #papers : 0 #papers : 3mIPFP[Bougleuxet al., 2017b]
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 58
Graph matching : State of the artMethods Error-correcting GM Subgraph GM
Constrained quadraticprogramming
#papers : 1 mIPFP [Bougleuxet al., 2017b]
#papers : 4 – IPFP [Leordeanuet al., 2009]
Spectral #papers : 0 #papers : 2 SM[Leordeanuand Hebert, 2005]
Branch and bound #papers : 2 A* [Riesenet al., 2007]
#papers : 0
Bio-inspired #papers : 1 GEDEVO [Ibragimovet al., 2013]
#papers : 2 AG [Cross et al., 1997]
Hungarian method #papers : 3 SFBP [Serratosa,2015]
#papers : 0
Probabilistic #papers : 1 BayesianGED [Myers et al.,2000]
#papers : 1 [Zass andShashua, 2008]
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 59
Deadlocks :
• Facts:• Exact methods are rarely study
• Few works have paid attention to ILP models.
• Deadlocks :• To study exact methods
• To derive heuristics from exact methods
• There is a need to study heuristics according to speed and effectiveness criteria.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 60
Exact methods for error-correcting matching
• Integer Linear Program
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 61
• Notations
i
j
k
l
G1 G2
ij lk
x
x
y
Error-correcting matching : Integer Linear Program
62
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
[Lerouge et al., 2017, PR]R. Raveaux
• Objective function:
Vertex substitutions Edge substitutions vertex deletions
vertex insertions edge insertionsedge deletions
Error-correcting matching : Integer Linear Program
63
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
substitutionsdeletion
Insertion
Insertion
deletionsubstitutions
Vertices
mapping
constraints
Edges
mapping
constraints
Error-correcting matching : Integer Linear Program
64
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
• Topological constraints
i
j
k
l
G1 G2
ij lk
x
x
y
Error-correcting matching : Integer Linear Program
65
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Heuristic : LocBra
• A Matheuristic is derive from F1.
• Δ = 𝐻𝑎𝑚𝑚𝑖𝑛𝑔 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒
• Adding branching constraints
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 66
Heuristic : Anytime
• Export all the list of dUB
• Interruptability: At each iteration, we can
stop the algorithm if it exceeds CT
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
[Abu-Aisheh et al., 2016, PRL]
R. Raveaux 67
Antyime
A A’B B’
C’C D
A-A’ A-B’ A-C’ A-ϵ
root
B-A’ B-C’ B-ϵ
C-C’ C-ϵ
D--ϵ
B-A’ B-B’ B-ϵ
g=1
h=4
f=5
g=2
h=5
f=7
g=3
h=2
f=5
g=1
h=5
f=6
g=4
h=2
f=6
g=4
h=3
f=7
g=4
h=3
f=7
g=1
h=5
f=6
g=1.5
h=4
f=5.5
g=3
h=3
f=6B-B’
g=4
h=3
f=7
g=1.5
h=4
f=5.5
g=2
h=3.8
f=5.8
g: current cost
h: estimated cost
f : total cost
G1 G2
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 68
https://goo.gl/xJn5nq
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 69
https://goo.gl/CebSg2
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 70
Benchmarking
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 71
Conference paper: Z. Abu-Aisheh, R. Raveaux and J-Y Ramel: A Graph Database Repository and
Performance Evaluation Metrics for Graph Edit Distance. GbRPR 2015 : 138-147.
Database # graphs avg. (max) # nodes
Decomposition Overview Purpose
GREC 50 12.5 (20) MIX, 5, 10, 15 and 20 Classification
CMU 111 30 (30) 30 Matching
MUTA 80 40 (70) MIX ,10, 20, 30, … and
70
Classification
PAH 94 20.7 (28) - Classification
Acyclic 185 8.2 (11) - Classification
Alkane 150 8.9 (10) - Classification
Datasets
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
[Abu-Aisheh et al.,2015a, GBR]
R. Raveaux 72
ICRP 2016 Contest
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
[Abu-Aisheh et al., 2017, PRL]
R. Raveaux 73
Summary
• Error tolerent graph matching :• New ILP models : F1, F2, F3 :
• New exact methods : Branch-bound, mathematical solver : • Fast convergence, evaluation with heuristics
• New heuristics : Matheuristic and Anytime• Accurate and flexible
• New benchmarks
• A contest
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 74
Graph classification
• Problems
• State of the art
• Deadlocks
• Contributions
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 75
Graph classification
• Learning-free
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 76
Problem
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 77
State of the art : speeding up the kNN problem
Generic (dissimilarity space) Graph oriented methods
Dissimilarity functionMetric property [Uhlmann, 1991, IPL] Fast GED methods [Riesen,
2015, ACVPR ]
Dissimilarity spaceSpace partition, prototypes, hashtable, proximity graph, line search [Malkov and Yashunin, 2016, PAMI]
Graph prototypes [Musmanno and Ribeiro, 2016, EJOR]
• Structuring the dissimilarity space lies at the art of machine learning (out of scope here)
• Graph distance is a combinatorial problem.• Many methods have focused on : fast GED methods.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 78
Deadlocks
• Is there a way to specialize a line search method to operate on graph space instead of the generic dissimilarity space ?
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 79
Is there a way to specialize a line search method to operate on graph space instead of the generic dissimilarity space ?
• Merging the GED problem and the kNN problem in a single problem:
Γ set of all possible matchings between 𝐺 ∈ 𝑇𝑒𝑆 and 𝐺𝑗 ∈ 𝑇𝑟𝑆
Merging
The set of all possible matchings between 𝐺 and all 𝐺𝑗 ∈ 𝑇𝑟𝑆
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 80
Is there a way to specialize a line search method to operate on graph space instead of the generic dissimilarity space ?
• Merging the GED problem and the kNN problem in a single problem:
Merging
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
[Abu-Aisheh et al., 2017, PRL]
R. Raveaux 81
Solving the MGED problem
• A Branch and bound method
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
[Abu-Aisheh et al., 2017, PRL]
R. Raveaux 82
Data sets
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 83
Methods
• Fast GED methods : BP, FBP, DF, BS-1
• Our proposal : one-tree (with different initialisation)
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 84
Classification test
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 85
Summary
• « Learning free » methods for classification
• Merging GED problem and kNN in a single problem (MGED)
• New algorithm for this problem
• Efficient when the data size increases
• Can be combined with other learning based-methods:• Learning the distance
• Learning prototypes
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 86
Machine learning in graph space for matching and classification
Introduction : Combinatorial Optimization : Machine Learning : ConclusionProblems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 87
Graph matching
• Problem
• State of the art
• Deadlocks
• Contributions
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 88
Learning graph mathching problem
A possible loss :
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 89
State of the art
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 90
Deadlocks
• How to deal with insertion and deletion costs ?
• Can an heuristic output solutions closer to optimality thanks to machine learning?
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 91
Parametrized GED
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
[Raveaux et al., 2017, GBR].
R. Raveaux 92
Parametrized GED
Φ = 𝑑 ∗, 1 ; 𝑑 ∗, 2 ; 𝑑 ∗, ε ; 𝑑 ∗∗, 12 ; 𝑑 ∗∗, εε𝛽 = 𝛽1 ; 𝛽2 ; 𝛽ε ; 𝛽4 ; 𝛽εε
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 93
Learning with parametrized GED algorithm
• The learning problem reformulated :
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 94
Learning algorithm : gradient descent
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 95
Data set
CMU-House
|TrS| 50
|TeS| 50
|V| 30
|E| 70
Attributes xy distance
Node matching cost 1
Edge matching cost L1 norm
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 96
Methods
• GED solver : BP
• BP without learning
• BP with our learning scheme
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 97
Learning graph matching testhttps://youtu.be/JrMR2-5mjA4
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 98
Learning Graph matching test
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 99
Summary
• Parametrized error correcting graph matching
• Learning scheme for graph matching
• Independent to the graph matching methods
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Matching : Problems – State of the art – Deadlocks – Contributions - Summary
[Raveaux et al., 2017, GBR].
R. Raveaux 100
Graph classification
• Problem
• State of the art
• Deadlocks
• Contributions
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 101
Problem
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 102
State of the art
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 103
State of the art
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 104
State of the art
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 105
State of the art
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 106
Deadlocks
• Learning graph distance for classification with local parameters for nodes and edges.
• Learning graph matching and graph prototypes in a hierarchical manner.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 107
Learning graph distance for classification with local parameters for nodes and edges.
Learning rule
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
[Martineau et al., 2018, PRL].
R. Raveaux 108
Learning graph distance : Learning algorithm
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
[Martineau et al., 2018, PRL].R. Raveaux 109
Benchmark
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 110
Data sets
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 111
Methods
• Learning schemes for global parameters :• Grid search [Riesen et al, 2009, IVC] (R-1NN)
• Constraint quadratic programming [Cortes et al, 2015, PRL] (C-1NN)
• 1NN methods
• 1NN based on median graphs
R. Raveaux 112
Results
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 113
Learning graph matching and graph prototypes in a hierarchical manner.
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 114
The basics of artificial neural networks
115
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
The basics of graph neural networks
116
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Adjacency matrix
117
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Degree matrix
118
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Key idea and Intuition [Kipf and Welling, 2016]
• The key idea is to generate node embeddings based on local neighborhoods.
• The intuition is to aggregate node information from their neighbors using neural networks.
• Nodes have embeddings at each layer and the neural network can be arbitrary depth. “layer-0” embedding of node u is its input feature, i.e. Fu.
119
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
GNN: a pictorial model
120
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Simple example of f
121
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
2 issues of this simple example
• Issue 1:• for every node, f sums up all the feature vectors of all neighboring nodes but
not the node itself.
• Fix: simply add the identity matrix to A
• Issue 2:• A is typically not normalized and therefore the multiplication with A will
completely change the scale of the feature vectors.
• Fix: Normalizing A such that all rows sum to one:
122
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Altogether: [Kipf and Welling, 2016]
123
• The two patched mentioned before +
• A better (symmetric) normalization of the adjacency matrix
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux
Our idea : graph matching based neural network
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 124
Graph convolution layer :
• A layer is composed of many filter graphs (GF)
• A layer outputs : A graph with node features and edge features
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 125
GNN for graph classification
• A GNN outputs : node embeddings
• Adding a global average pooling layer
• Average pooling
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 126
Benchmarking
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 127
Data set : MNIST
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 128
Data set
MNIST
# classes 2
|TrS| 1 000 graphs
|TeS| 5 000 graphs
|V| 196 and 75 nodes
Node Attributes Pixel intensity (gray level)
Edge Attributes none
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 129
Methods
• Lenet 5 [Lecun et al., 1998, IEEE]
• MoNet [Monti et al., 2016, CVPR]
• Our
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 130
Results
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 131
Summary
• Learning scheme for error-correcting graph matching in a classification context.
• Hierarchical leaning of filter graphs and graph similarity.
• Deep learning paradigm can be extended to graph space
Introduction : Combinatorial Optimization : Machine Learning : ConclusionGraph Classification : Problems – State of the art – Deadlocks – Contributions - Summary
R. Raveaux 132
Conclusion
• Summary• Perspective
Introduction : Combinatorial Optimization : Machine Learning : ConclusionSummary - Perspective
R. Raveaux 133
Summary
• Remember the cons:• Slow methods (Often) a combinatorial problem• Lack of learning techniques
• Our asnwsers:• Faster methods to compare/classify graphs
• Thanks to discrete optimization techniques.
• Learning algorithm operating in graph space to compare/classify graphs• Parametrized graph matching• Learning scheme merging combinatorial problems and gradient descent
• Deep learning paradigm can be extended to graph space
Introduction : Combinatorial Optimization : Machine Learning : ConclusionSummary - Perspective
R. Raveaux 134
Perspectives
• Combinatorial optimization• Application of optimization tools for pattern recognition problems
• Learning prototypes
• Machine learning• Theoretical understanding of what is learned • Common benchmark datasets
• Merging Machine learning and Combinatorial optimization• Integration of machine learning into combinatorial optimization methods
(branch and bounds, matheuristics)• Integration of combinatorial optimization into machine learning mehods
• Combinatorial layers in neural networks
Introduction : Combinatorial Optimization : Machine Learning : ConclusionSummary - Perspective
R. Raveaux 135
Thank you for your attention
• Any question ?
Introduction : Combinatorial Optimization : Machine Learning : ConclusionSummary - Perspective
Colleagues from the lab
R. Raveaux 136
Our references :
• [Martineau et al., 2018, PRL] : Maxime Martineau, Romain Raveaux, Donatello Conte, Gilles Venturini. Learning error-correcting graph matching with a multiclass neural network. Pattern Recognition Letters, Elsevier, 2018.
• [Raveaux et al., 2017, GBR] : Romain Raveaux, Maxime Martineau, Donatello Conte, Gilles Venturini: Learning Graph Matching with a Graph-Based Perceptron in a Classification Context. GbRPR 2017: 49-58
• [Abu-Aisheh et al., 2017, PRL] : Zeina Abu-Aisheh, Benoit Gaüzère, Sébastien Bougleux, Jean-Yves Ramel, Luc Brun, Romain Raveaux, Pierre Héroux, Sébastien Adam: Graph edit distance contest: Results and future challenges. Pattern Recognition Letters 100: 96-103 (2017)
• [Abu-Aisheh et al.,2015a, GBR] : Zeina Abu-Aisheh, Romain Raveaux, Jean-Yves Ramel: A Graph DatabaseRepository and Performance Evaluation Metrics for Graph Edit Distance. GbRPR 2015: 138-147
• [Abu-Aisheh et al., 2016, PRL] : Zeina Abu-Aisheh, Romain Raveaux, Jean-Yves Ramel: Anytime graph matching. Pattern Recognition Letters 84: 215-224 (2016)
• [Lerouge et al., 2017, PR] : Julien Lerouge, Zeina Abu-Aisheh, Romain Raveaux, Pierre Héroux, Sébastien Adam: New binary linear programming formulation to compute the graph edit distance. Pattern Recognition 72: 254-265 (2017)
R. Raveaux 137
References
• [Brontein 2016, CoRR] Michael M. Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: goingbeyond euclidean data. CoRR, abs/1611.08097, 2016.
• [Luqman et al., 2017, PR] Ramzi Chaieb, Karim Kalti, Muhammad MuzzamilLuqman, Mickaël Coustaty, Jean-Marc Ogier, and Najoua Essoukri Ben Amara. Fuzzy generalized median graphs computation: Application to content-based document retrieval. Pattern Recognition, 72:266 – 284, 2017.
• [Kipf and Welling, 2017, ICLR] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. CoRR, abs/1609.02907, 2016.
R. Raveaux 138
References
• [Neuhaus and Bunke., 2007, MPAI] M. Neuhaus and H. Bunke. Bridging the gap between graph edit distance and kernel machines. Machine Perception and Artificial Intelligence., 68:17–61, 2007.
• [Riesen, 2015, ACVPR] Kaspar Riesen. Structural Pattern Recognition withGraph Edit Distance - Approximation Algorithms and Applications. Advances in Computer Vision and Pattern Recognition. Springer, 2015. ISBN 978-3-319-27251-1. doi:10.1007/978-3-319-27252-8. URL https://doi.org/10. 1007/978-3-319-27252-8.
• [Goyal and Ferrara, 2017, KBSyst] Palash Goyal and Emilio Ferrara. Graph embedding techniques, applications, and performance: A survey. CoRR, abs/1705.02801, 2017. URL http://arxiv.org/abs/1705.02801
R. Raveaux 139
References
• [Cai et al, 2017, IEEE TKDE] HongYun Cai, Vincent W. Zheng, and Kevin Chen-Chuan Chang. A comprehensive survey of graph embedding: Problems, techniques and applications. CoRR, abs/1709.07604, 2017. URL http://arxiv.org/abs/1709.07604.
• [Zeng et al. 2009, PVLDB] Zhiping Zeng, Anthony K. H. Tung, Jianyong Wang, Jianhua Feng, and Lizhu Zhou. Comparing stars: On approximating graph edit distance. PVLDB, 2(1):25–36, 2009. doi:10.14778/1687627. 1687631. URL http://www.vldb.org/pvldb/2/vldb09-568.pdf.
• [Uhlmann, 1991, IPL] Jeffrey K. Uhlmann. Satisfying general proximity / similarity queries with metric trees. Information Processing Letters, 40(4):175 – 179, 1991. ISSN 0020-0190. doi:https://doi.org/10. 1016/0020-0190(91)90074-R. URL http://www.sciencedirect.com/science/article/pii/ 002001909190074R.
R. Raveaux 140
References
• [Malkov and Yashunin, 2016, PAMI] Yury A. Malkov and D. A. Yashunin. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. CoRR, abs/1603.09320, 2016. URL http: //arxiv.org/abs/1603.09320
• [Musmanno and Ribeiro, 2016, EJOR] Leonardo M. Musmanno and Celso C. Ribeiro. Heuristics for the generalized median graph problem. European Journal of Operational Research, 254(2):371 – 384, 2016. ISSN 0377-2217. doi:https://doi.org/10.1016/j.ejor.2016.03.048. URL http://www.sciencedirect.com/science/article/pii/S0377221716301941.
• [Lecun et al., 1998, IEEE] Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, pages 2278–2324, 1998.
R. Raveaux 141
References
• [Monti et al., 2016, CVPR] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. CoRR, abs/1611.08402, 2016.
R. Raveaux 142