Using Wikipedia and Conceptual Graph Structures to Generate Questions for Academic Writing Support
Image interpretation by using conceptual graph: introducing complex spatial relations
description
Transcript of Image interpretation by using conceptual graph: introducing complex spatial relations
![Page 1: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/1.jpg)
Image interpretation by using conceptual graph: introducing complex spatial relationsAline Deruyver, AFD
LSIIT UMR7005 CNRS ULP
![Page 2: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/2.jpg)
Spatial relations are a key point of image understanding
4
7 14 8
5
2
6 3
112
9 10
11
13
![Page 3: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/3.jpg)
Image interpretation Matching of heterogeneous graphs Discrete relaxation algorithm based on arc-consistency
checking Non univocal matching: two levels of constraints:
Between the nodes Inside the nodes
Introducing complex spatial relations
![Page 4: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/4.jpg)
set of pixels
Knowledge representation
matching
Data
Semantic concept
![Page 5: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/5.jpg)
Graph matching
A large amount of articles exists on graph matching.
Classical approaches using arithmetic properties associating with the arcs, allow very fast algorithms.
When the constraints associated with the arcs are qualitative, this kind of approach is more difficult to applied.
![Page 6: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/6.jpg)
Matching of heterogeneous graphs Discrete relaxation algorithm based on arc-consistency
checking Non univocal matching: two levels of constraints:
Between the nodes Inside the nodes
Introducing complex spatial relations
Image interpretation
![Page 7: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/7.jpg)
AC4 Algorithm : solve the graph matching problem by solving a problem of constraint satisfactionMohr and Henderson (1986)
The control of the consistency is an NP-difficult problem (very long computation time in practice) butA lot of real problems can be solved by checking the arc-consistency
![Page 8: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/8.jpg)
Limits
It is supposed that with a given node is associated only one value
It is supposed that we have a one to one matching.
Not often encountered in practice
![Page 9: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/9.jpg)
Matching of heterogeneous graphs Discrete relaxation algorithm based on arc-consistency
checking Non univocal matching: two levels of constraints
Between the nodes Inside the nodes
Introducing complex spatial relations
Image interpretation
![Page 10: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/10.jpg)
Solution: checking arc-consistency with bilevel constraints (A. Deruyver and al. Artificial Intelligence, 1997)
A notion of "intra-node" contraints is introduced.
A new structure of node is introduced
![Page 11: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/11.jpg)
Constraint satisfaction problem with two levels of constraint
Definition: Let Cmpi be a compatibility relation such that (a,b) Cmpi a
and b are compatibles. Let Cmpgi be a global compatibility constraint applied on a set
of values Si Di such that Si Cmpgi Si satisfies the global compatibility constraint.
Let Cij be a constraint between i and j. Let Si, Sj such that Si Di and Sj Dj, Si, Sj Cij means
that (Si, Sj) satisfies the oriented constraint Cij. Si, Sj Cij ai Si, (a'i, aj) Si x Sj, such that (ai, a'i)
Cmpi and (a'i,aj) Cij and aj Sj, (a'j,ai) Sj x Si, such that (aj, a'j) Cmpj and (ai,a'j) Cij.
The sets {S1 ... Sn} satisfy PSCDFDNC Cij Si, Sj Cij et Si Cmpgi and Sj Cmpgj.
![Page 12: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/12.jpg)
How to match? Region
Adjacency Graph
Semantic Graph
![Page 13: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/13.jpg)
Classical arc consistency checking fails in this context
It is not always possible de build a quotient node to
gather all the linked regions in a same node
![Page 14: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/14.jpg)
![Page 15: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/15.jpg)
![Page 16: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/16.jpg)
2nd level : intra-node constraints
![Page 17: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/17.jpg)
First level : inter-node constraints
![Page 18: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/18.jpg)
2nd level : intra-node constraints
![Page 19: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/19.jpg)
The regions in the nodes support themselves if it exits a path between each couple
of regions such that each region can be reached from
each interface
?
![Page 20: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/20.jpg)
A path = succession of spatial
relations defined in a finite set TT
![Page 21: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/21.jpg)
Intra-node constraints: compatibility between two values (Cmpi)
Finding a linear sub-graph of the region adjacency graph.
Finding a succession of values (regions) able to link two values that can be associated with a node of the semantic graph.
![Page 22: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/22.jpg)
Intra-node constraints: compatibility between two values (Cmpi)
Finding a linear sub-graph of the region adjacency graph.
Two kinds of constraints on the graph: Path constraints applied on the arcs. Constraints applied on the set of
nodes (union of regions associated with the nodes)
Finding a succession of values (regions) able to link two values that can be associated with a node of the semantic graph.
![Page 23: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/23.jpg)
Matching of heterogeneous graphs Discrete relaxation algorithm based on arc-consistency
checking Non univocal matching: two levels of constraints:
Between the nodes Inside the nodes
Introducing complex spatial relations
Image interpretation
![Page 24: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/24.jpg)
Complex spatial relations
Our model: a conceptual graph describing very precisely the spatial organization of the different parts of the object that we look for.
Adjacency relations are too poor
Some systems of complex spatial relations have been proposed.
![Page 25: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/25.jpg)
Complex spatial relations
The RCC8 system: it proposes 8 elementary relations in a topologic context:
Does not take into account the shape of the regions and the directional information.
E
N
E
NE N
E
NE N E N
DC(E,N) EC(E,N) TPP(E,N) PO(E,N) EQ(E,N) NTPP(E,N)
![Page 26: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/26.jpg)
Complex spatial relations Cardinal Direction Relation Formalism
(Skiadopoulos and Koubarakis) Use the notion of minimum bounding box
(powerful reduction of the information)
NW
W
SW
N
S
NE
E
SE
Oa
b
![Page 27: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/27.jpg)
Complex spatial relations RCC8 and CDRF are powerful formalisms. It is possible to retrieve the RCC8 relations
from the CRDF.
- But there is no notion of distance
- The minimum bounding boxes are not always enough to compute correct distances:
A
B
![Page 28: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/28.jpg)
Connectivity-Direction-Metric Formalism3 kinds of basic information: The connectivity C(x,y) “x is connected to
y” The notion of minimum bounding box A new notion of minimum bounding box of
border interface.
A
B
mbbbiwmbbbie
![Page 29: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/29.jpg)
Connectivity-Direction-Metric Formalism 8 distances can be defined between
minimum bounding boxes: Example:
A
B
A
B
dg1
dg2
dg3
dg4 ds1
ds2
ds3 ds4
![Page 30: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/30.jpg)
Connectivity-Direction-Metric Formalism We can define the distance d between two
minimum bounding boxes of border interface:
A
B
mbbbiw
mbbbie
d
![Page 31: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/31.jpg)
Connectivity-Direction-Metric Formalism
An elementary relation is a relation:
(1) of connectivity or non connectivity
(2) of directional relationships between mbb with none or one metric relation chosen among the metrics dsi and dgi (i=1…4) (with inferior and superior limits). We have 4 directional relations: N (North), S (South), W (West) et E (East).
(3) of directional relationships between mbbbi with one metric relation (with inferior and superior limits). We have 4 directional relations: Ni, Si, Wi et Ei.
Allows to retrieve the CDRF
![Page 32: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/32.jpg)
How to combine these relations?
Example: ( Ei or Wi ) and ( Si or Ni )
![Page 33: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/33.jpg)
With classical arc-consistency checking A and B and C must be satisfied for all the labels
A
B
C
With arc-consistency with bilevel constraints A and B and C must be satisfied but not necessary directly for all the labels
With quasi arc-consistency more complex constraints can be expressed :With number of relaxation equal to 0 A and B and C must be satisfied as for case (2) 1 (A and B) or (A and C) or (B and C) must be
satisfied 2 A or B or C must be satisfied
B
A
C
BA
C
(2)
(1)
(3)
![Page 34: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/34.jpg)
We want more !
We want (A or B) and CA
B
C
![Page 35: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/35.jpg)
We want more !
We want (A or B) and CA
B
C
OR
![Page 36: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/36.jpg)
We want more !
We want (A or B) and CA
B
C
OR
The kernel is divided in two new levels
•The shell : nodes linking interfaces according to logical combinations of (spatial) constraints (linked with a “OR” logical relationship)
•The core : node linking all the nodes of the shell (linked with a “AND” logical relationship).
![Page 37: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/37.jpg)
We can linked one interface with different nodes of the shell and described any logical combination
R1
R2
R3
R4
R1 , R2, R3 , et R4 are the number of authorized relaxations for each node of the shell.
AB
C
D
EF
Example of logical combination of spatial relations: (A or B or C) and (C or D) and (D or E) and (E or F)
![Page 38: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/38.jpg)
An example with the relation « is around of » to describe a flower:
A region center of the flower is defined recursively by : a region center has
a South-neighbor which is a center or a petal AND a West neighbor which is a center or a petal AND a North neighbor which is a center or a petal AND an East neighbor which is a center or a petal
•An intra-node constraint Cmpi is not a sufficient constraint (see left example)
![Page 39: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/39.jpg)
An example with the relation « is around of » to describe a flower:
Ni
Ni
Ei
EiWi
Wi
SiSi
Node “center of the flower”
Node “petal”
![Page 40: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/40.jpg)
Connectivity-Direction-Metric Formalism An example with the relation “is partially
around with a given distance »: the case of the hairs around the eyes
A
B C
Ni or (Wi and Ei) Ni or (Ei and Wi)
A : hairsB :left eyeC: right eye
A
B C
![Page 41: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/41.jpg)
Connectivity-Direction-Metric Formalism
A
Classical minimum bounding boxes:overlapping = it is not possible to compute a distance between A and B.
Minimum bounding boxes of border interface: Better possibilities of computation of distances.
B A B
![Page 42: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/42.jpg)
Experimentations
On human faces:
![Page 43: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/43.jpg)
Experimentations
On cars:
On flowers:
![Page 44: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/44.jpg)
Conclusion
Possibility to use the arc-consistency checking to solve pattern recognition problems. Adapted to solve non univocal matching (over-
segmented images, noise, occlusions, …) Using complex spatial relations (type CDF et CDMF)
allowing to describe very precisely a lot of objects. Aim: defining the conceptual basis of a logical and
symbolic approach that can be an alternative or a supplement to arithmetic approaches of image interpretation.
![Page 45: Image interpretation by using conceptual graph: introducing complex spatial relations](https://reader031.fdocuments.in/reader031/viewer/2022013004/56815805550346895dc5762d/html5/thumbnails/45.jpg)
Thank you for your attention