Introduction to Computer Sciencerolf/Edu/DM534/E20/DM534-2020-fall...Introduction to Computer...

Arthur Zimek

Color Histograms asFeature Spaces forRepresentation ofImages

A First Glimpse onClustering

ReferencesIntroduction to Computer Science

Arthur Zimek

University of Southern Denmark

DM534, fall 2020

Arthur Zimek

References

Outline

Color Histograms as Feature Spaces for Representation of Images

A First Glimpse on Clustering

Arthur Zimek

Distances

Features for Images

Summary

References

Outline

Color Histograms as Feature Spaces for Representation of ImagesDistancesFeatures for ImagesSummary

Arthur Zimek

Distances

Features for Images

Summary

References

Outline

Arthur Zimek

Distances

Features for Images

Summary

References

Similarity

I Similarity (as given by some distance measure) is acentral concept in data mining, e.g.:I clustering: group similar objects in the same cluster,

separate dissimilar objects to different clustersI outlier detection: identify objects that are dissimilar (by

some characteristic) from most other objectsI definition of a suitable distance measure is often crucial

for deriving a meaningful solution in the data miningtask

I images

I CAD objects

I proteins

I texts

I . . .5

Arthur Zimek

Distances

Features for Images

Summary

References

Spaces and Distance Functions

Common distance measure for (Euclidean) feature vectors:LP-norm

distP(p, q) =(|p1 − q1|P + |p2 − q2|P + . . .+ |pn − qn|P

Euclidean norm(L2):

Manhattan norm(L1):

Maximum norm(L∞, also: Lmax,supremum dist.,Chebyshev dist.)

Arthur Zimek

Distances

Features for Images

Summary

References

Spaces and Distance Functions

weighted Euclidean norm:dist(p, q) =

(w1|p1 − q1|2+

w2|p2 − q2|2 + . . .+ wn|pn − qn|2) 1

quadratic form*:dist(p, q) =((p− q)M(p− q)T)

*note that we assume vectors to be row vectors here

Arthur Zimek

Distances

Features for Images

Summary

References

Outline

Arthur Zimek

Distances

Features for Images

Summary

References

Categories of Feature Descriptors for Images

I distribution of colorsI textureI shapes (contoures)

Arthur Zimek

Distances

Features for Images

Summary

References

Color Histogram

I a histogram represents the distribution of colors overthe pixels of an image

I definition of an color histogram:I choose a color space (RGB, HSV, HLS, . . . )I choose number of representants (sample points) in the

color spaceI possibly normalization (to account for different image

sizes)

Arthur Zimek

Distances

Features for Images

Summary

References

Color Space Example: RGB cube

Arthur Zimek

Distances

Features for Images

Summary

References

Impact of Number of Representants

original images in full RGB space (2563 = 16, 777, 216)

Arthur Zimek

Distances

Features for Images

Summary

References

43 163

Arthur Zimek

Distances

Features for Images

Summary

References

43 163

Arthur Zimek

Distances

Features for Images

Summary

References

43 163

Arthur Zimek

Distances

Features for Images

Summary

References

43 163

Arthur Zimek

Distances

Features for Images

Summary

References

The histogram for each image is essentially a visualizationof a vector:(0.77, 0, 0, 0, 0.08, 0, 0.15, 0) (0.8, 0, 0, 0, 0.11, 0, 0.09, 0)(0.9, 0, 0, 0, 0.05, 0, 0.05, 0) (0.955, 0, 0, 0, 0.045, 0, 0, 0)

Arthur Zimek

Distances

Features for Images

Summary

References

Arthur Zimek

Distances

Features for Images

Summary

References

Arthur Zimek

Distances

Features for Images

Summary

References

Distances for Color Histograms

Euclidean distance for images P and Q using the colorhistograms hP and hQ:

dist(P,Q) =√

(hP − hQ) · (hP − hQ)T

(1, 0, 0) (0, 1, 0) (0, 0, 1)

dist(RED,PINK) =√

dist(RED,BLUE) =√

dist(BLUE,PINK) =√

A ‘psychologic’ distance would consider that red is (in ourperception) more similar to pink than to blue.

Arthur Zimek

Distances

Features for Images

Summary

References

Example for the Distance Computation ofHistograms

dist(P,Q) =√(hP − hQ) · (hP − hQ)T

dist(RED,PINK) =√

((1, 0, 0)− (0, 1, 0)) · ((1, 0, 0)− (0, 1, 0))T

(1,−1, 0) · (1,−1, 0)T

=√(1 · 1 + (−1) · (−1) + 0 · 0)

Arthur Zimek

Distances

Features for Images

Summary

References

Distances for Color Histograms

Quadratic form with ‘psychological’ similarity matrix

1 a12 . . .

a21 1 . . ....

. . ....

. . . 1

where aij ( ?= aji) describe the

subjective similarity of the features i and j in the colorhistogram:

distA(P,Q) =√(hP − hQ) · A · (hP − hQ)T

A′ =

1 0.9 00.9 1 00 0 1

dist(RED,PINK) =√

dist(RED,BLUE) =√

dist(BLUE,PINK) =√

Arthur Zimek

Distances

Features for Images

Summary

References

Outline

Arthur Zimek

Distances

Features for Images

Summary

References

Your Choice of a Distance Measure

There are hundreds of distance functions [Deza and Deza,2009].I For time series: DTW, EDR, ERP, LCSS, . . .I For texts: Cosine and normalizationsI For sets – based on intersection, union, . . . (Jaccard)I For clusters (single-link, average-link, etc.)I For histograms: histogram intersection, “Earth movers

distance”, quadratic forms with color similarityI With normalization: Canberra, . . .I Quadratic forms / bilinear forms: d(x, y) := xTMy for

some positive (usually symmetric) definite matrix M.

Note that:Choosing the appropriate distance function can be seen asa part of “preprocessing”.

Arthur Zimek

Distances

Features for Images

Summary

References

Summary

You learned in this section:I distances (Lp-norms, weighted, quadratic form)I color histograms as feature (vector) descriptors for

imagesI impact of the granularity of color histograms on

similarity measures

Arthur Zimek

General Purpose ofClustering

Partitional Clustering

Algorithm

Visualization:AlgorithmicDifferences

Summary

References

Outline

A First Glimpse on ClusteringGeneral Purpose of ClusteringPartitional ClusteringAlgorithmVisualization: Algorithmic DifferencesSummary

Arthur Zimek

Algorithm

Summary

References

Outline

Arthur Zimek

Algorithm

Summary

References

Purpose of Clustering

I identify a finite number of categories (classes, groups:clusters) in a given dataset

I similar objects shall be grouped in the same cluster,dissimilar objects in different clusters

I “similarity” is highly subjective, depending on theapplication scenario

Arthur Zimek

Algorithm

Summary

References

A Dataset can be Clustered in DifferentMeaningful Ways

(Figure from Tan et al. [2006].)

Arthur Zimek

Algorithm

Summary

References

Outline

Arthur Zimek

Algorithm

Summary

References

Criteria of Quality: Cohesion and Separation

I cohesion: how strong are the cluster objects connected(how similar, pairwise, to each other)?

I separation: how well is a cluster separated from otherclusters?

small within cluster distances large between cluster distances

Arthur Zimek

Algorithm

Summary

References

Optimization of Cohesion

Partitional clustering algorithms partition a dataset into kclusters, typically minimizing some cost function(compactness criterion), i.e., optimizing cohesion.

Arthur Zimek

Algorithm

Summary

References

Assumptions for Partitioning Clustering

Central assumptions for approaches in this family aretypically:I number k of clusters known (i.e., given as input)I clusters are characterized by their compactnessI compactness measured by some distance function

(e.g., distance of all objects in a cluster from somecluster representative is minimal)

I criterion of compactness typically leads to convex oreven spherically shaped clusters

Arthur Zimek

Algorithm

Summary

References

Outline

Arthur Zimek

Algorithm

Summary

References

Construction of Central Points: Basics

I objects are points x = (x1, . . . , xd) in Euclidean vectorspace Rd, dist = Euclidean distance (L2)

I centroid µC: mean vector of all points in cluster C

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

µCi =1

|Ci| ·∑

o∈Cio

Arthur Zimek

Algorithm

Summary

References

I objects are points x = (x1, . . . , xd) in Euclidean vectorspace Rd, dist = Euclidean distance (L2)

I centroid µC: mean vector of all points in cluster C

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

µCi =1

|Ci| ·∑

o∈Cio

Arthur Zimek

Algorithm

Summary

References

I measure of compactness fora cluster C:

TD2(C) =∑p∈C

dist(p, µC)2

(a.k.a. SSQ: sum ofsquares)

I measure of compactness fora clustering

TD2(C1,C2, . . . ,Ck) =

k∑i=1

TD2(Ci)

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

Arthur Zimek

Algorithm

Summary

References

I measure of compactness fora cluster C:

TD2(C) =∑p∈C

dist(p, µC)2

(a.k.a. SSQ: sum ofsquares)

I measure of compactness fora clustering

TD2(C1,C2, . . . ,Ck) =

k∑i=1

TD2(Ci)

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

Arthur Zimek

Algorithm

Summary

References

Basic Algorithm [Forgy, 1965, Lloyd, 1982]

Algorithm 2.1 (Clustering by Minimization of Variance)

I start with k (e.g., randomly selected) points as clusterrepresentatives (or with a random partition into k“clusters”)

I repeat:I assign each point to the closest representativeI compute new representatives based on the given

partitions (centroid of the assigned points)I until there is no change in assignment

Arthur Zimek

Algorithm

Summary

References

k-means

k-means [MacQueen, 1967] is a variant of the basicalgorithm:I a centroid is immediately updated when some point

changes its assignmentI k-means has very similar properties, but the result now

depends on the order of data points in the input file

Note that:The name “k-means” is often used indifferently for anyvariant of the basic algorithm, in particular also forAlgorithm 2.1 [Forgy, 1965, Lloyd, 1982].

Arthur Zimek

Algorithm

Summary

References

Outline

Arthur Zimek

Algorithm

Summary

References

k-means Clustering – Lloyd/Forgy Algorithm

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 recompute centroids:

µ ≈ (6.0, 4.3)

µ ≈ (5.0, 6.4)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 reassign points

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (5.0, 2.7)

µ ≈ (5.6, 7.4)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (4.0, 3.25)

µ ≈ (6.75, 8.0)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

no changeconvergence!

Arthur Zimek

Algorithm

Summary

References

k-means Clustering – MacQueen Algorithm

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 Centroids

(e.g.: fromprevious iteration):

µ ≈ (6.0, 4.3)

µ ≈ (5.0, 6.4)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign first point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign second point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (5.0, 4.0)

µ ≈ (5.75, 7.25)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign third point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (4.6, 4.0)

µ ≈ (6.7, 8.3)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign fourth point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assing fifth point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (4.0, 3.25)

µ ≈ (6.75, 8.0)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 reassign more points

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

possibly more iterations

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

possibly more iterationsconvergence

Arthur Zimek

Algorithm

Summary

References

k-means Clustering – MacQueen AlgorithmAlternative Run – Different Order

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 Centroids

(e.g.: fromprevious iteration):

µ ≈ (6.0, 4.3)

µ ≈ (5.0, 6.4)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign first point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign second point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign third point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign fourth point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (4.0, 8.5)

µ ≈ (4.3, 6.2)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 assign fifth point

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

µ ≈ (10.0, 1.0)

µ ≈ (4.7, 6.3)

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

101112 reasign more points

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

possibly more iterations

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 10 11 12

123456789

possibly more iterationsconvergence

Arthur Zimek

Algorithm

Summary

References

k-means Clustering – Quality

1 2 3 4 5 6 7 8 9 101112

123456789

101112

SSQ(µ1, p1) = |4− 10|2 + |3.25− 1|2 = 36 + 5 116 = 41 1

16SSQ(µ1, p2) = |4− 2|2 + |3.25− 3|2 = 4 + 1

16 = 4 116

SSQ(µ1, p3) = |4− 3|2 + |3.25− 4|2 = 1 + 916 = 1 9

16SSQ(µ1, p4) = |4− 1|2 + |3.25− 5|2 = 9 + 3 1

16 = 12 116

TD2(C1) = 58 34

SSQ(µ2, p5) = |6.75− 7|2 + |8− 7|2 = 116 + 1 = 1 1

16SSQ(µ2, p6) = |6.75− 6|2 + |8− 8|2 = 9

16 + 0 = 916

SSQ(µ2, p7) = |6.75− 7|2 + |8− 8|2 = 116 + 0 = 1

16SSQ(µ2, p8) = |6.75− 7|2 + |8− 9|2 = 1

16 + 1 = 1 116

TD2(C2) = 2 34

First solution: TD2 = 61 12

Second solution: TD2 ≈ 72.68Optimal solution: TD2 = 54 2

Note: SSQ(µ, p) = Euclidean(µ, p)2 = L22(µ, p).

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 101112

123456789

101112

SSQ(µ1, p1) = |10− 10|2 + |1− 1|2 = 0TD2(C1) = 0

SSQ(µ2, p2) ≈ |4.7− 2|2 + |6.3− 3|2 ≈ 18.2SSQ(µ2, p3) ≈ |4.7− 3|2 + |6.3− 4|2 ≈ 8.2SSQ(µ2, p4) ≈ |4.7− 1|2 + |6.3− 5|2 ≈ 15.4SSQ(µ2, p5) ≈ |4.7− 7|2 + |6.3− 7|2 ≈ 5.7SSQ(µ2, p6) ≈ |4.7− 6|2 + |6.3− 8|2 ≈ 4.6SSQ(µ2, p7) ≈ |4.7− 7|2 + |6.3− 8|2 ≈ 8.2SSQ(µ2, p7) ≈ |4.7− 7|2 + |6.3− 9|2 ≈ 12.6TD2(C2) ≈ 72.86

Second solution: TD2 ≈ 72.68

Optimal solution: TD2 = 54 25

Arthur Zimek

Algorithm

Summary

References

1 2 3 4 5 6 7 8 9 101112

123456789

101112

SSQ(µ1, p2) = |2− 2|2 + |4− 3|2 = 0 + 1 = 1SSQ(µ1, p3) = |2− 3|2 + |4− 4|2 = 1 + 0 = 1SSQ(µ1, p4) = |2− 1|2 + |4− 5|2 = 1 + 1 = 2TD2(C1) = 4

SSQ(µ2, p1) = |7.4−10|2 +|6.6−1|2 = 6 1925 +31 9

25 = 38 325

SSQ(µ2, p5) = |7.4− 7|2 + |6.6− 7|2 = 425 + 4

25 = 825

SSQ(µ2, p6) = |7.4− 6|2 + |6.6− 8|2 = 1 2425 + 1 24

25 = 3 2325

SSQ(µ2, p7) = |7.4− 7|2 + |6.6− 8|2 = 425 + 1 24

25 = 2 325

SSQ(µ2, p8) = |7.4− 7|2 + |6.6− 9|2 = 425 + 5 19

25 = 5 2325

TD2(C2) = 50 25

Second solution: TD2 ≈ 72.68Optimal solution: TD2 = 54 2

Arthur Zimek

Algorithm

Summary

References

Outline

Arthur Zimek

Algorithm

Summary

References

Discussion

prosI efficient: O(k · n) per iteration, number of iterations is

usually in the order of 10.I easy to implement, thus very popular

consI k-means converges towards a local minimumI k-means (MacQueen-variant) is order-dependentI deteriorates with noise and outliers (all points are used

to compute centroids)I clusters need to be convex and of (more or less) equal

extensionI number k of clusters is hard to determineI strong dependency on initial partition (in result quality

as well as runtime)43

Arthur Zimek

Algorithm

Summary

References

Summary

You learned in this section:I What is Clustering?I Basic idea for identifying “good” partitions into k clustersI selection of representative pointsI iterative refinementI local optimumI k-means variants [Forgy, 1965, Lloyd, 1982,

MacQueen, 1967]

Arthur Zimek

References

References I

M. M. Deza and E. Deza. Encyclopedia of Distances. Springer, 3rd edition, 2009.ISBN 9783662443415.

E. W. Forgy. Cluster analysis of multivariate data: efficiency versus interpretabilityof classifications. Biometrics, 21:768–769, 1965.

S. P. Lloyd. Least squares quantization in PCM. IEEE Transactions on InformationTheory, 28(2):129–136, 1982. doi: 10.1109/TIT.1982.1056489.

J. MacQueen. Some methods for classification and analysis of multivariateobservations. In 5th Berkeley Symposium on Mathematics, Statistics, andProbabilistics, volume 1, pages 281–297, 1967.

P.-N. Tan, M. Steinbach, and V. Kumar. Introduction to Data Mining. AddisonWesley, 2006.

Introduction to Computer Sciencerolf/Edu/DM534/E20/DM534-2020-fall...Introduction to Computer...

Documents

Transcript of Introduction to Computer Sciencerolf/Edu/DM534/E20/DM534-2020-fall...Introduction to Computer...

DM534 - Introduction to Computer Science, Week 47 Graph Theory

There and Arthur Zimek There and Back Again

DM534: Introduction to Relational Databasesrolf/Edu/DM534/E20/slides-dm534... · 2020. 11. 3. · Relational Query Languages Based on relational algebra For relational databases,

DM534: Introduction to Relational Databasesrolf/Edu/DM534/E19/slides-dm534... · 2019. 11. 19. · Relational Query Languages Based on relational algebra For relational databases,

Arthur at the Library Arthur at the Library

United States Environmental Protection Agencyyosemite.epa.gov/.../$File/scan0192.pdfUNITED STATES ENVIRONMENTAL PROTECTION AGENCY REGION 4 ATLANTA, GEORGIA In the Matter of: Zimek

Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus ... · Wacha Bounliphone, Arthur Gretton, Arthur Tenenhaus, Matthew Blaschko 32nd International Conference on Machine Learning

DM534 - Introduction to Computer Science, Week 48 Graph Theory

Arthur Conan Doyle's Arthur Conan Doyle's Arthur Conan Doyle's ...

Detecting Clusters in Moderate-to-high Dimensional Datazimek/publications/KDD2008/tutorialSlides.pdf · Kriegel/Kröger/Zimek: Detecting Clusters in Moderate-to-High Dimensional Data

Online Algorithms [2ex] a topic in [2ex] DM534 Introduction to …rolf/Edu/DM534/E18/lectureOnline.pdf · 2018. 12. 12. · Online Algorithms a topic in DM534 { Introduction to Computer

Arthur Jones - Arthur Jones - Strength Curve Analysis

DM534 - SDUrolf/Edu/DM534/E16/IntroCS2016-final.pdfBase Case: Let k = 1. A1 = A. a ij is the number of edges from i to j, which is identical to the number of walks of length 1 from

Unit 11 Unit 11 The Midnight Visitor The Midnight Visitor Robert Arthur Robert Arthur Robert Arthur Robert Arthur.

KM 454e-A-20190315130421 Meeting 11-8-18.pdf · * OJT Waiver Rachael Poole Arthur Obar Arthur Obar Arthur Obar Arthur Obar Arthur Obar Arthur Obar Arthur Obar Richard Williams Richard

Arthur to Airport Arthur to City Hall

SAP‘s Exchange InfrastructureSAP 2002, XI, Interoperability Summit 2002, Orlando, S. Zimek 8 Web Application Server – The Integration Infrastructure Portal Infrastructure SAP Web

MachineLearning: LinearRegressionandNeuralNetworksrolf/Edu/DM534/E16/DM534-marco.pdf · MathematicalConcepts MachineLearning LinearRegression LogisticRegression Outline ArtiﬁcialNeuralNetworks

There and Arthur Zimek There and Back Again2012-11-28).pdf · There and Back Again Outlier Detection between Statistical Reasoning and Efﬁcient Database Methods Arthur Zimek University

Correlation Clustering - uni-muenchen.de · Correlation Clustering Arthur Zimek Dissertation an der Fakult¨at fur¨ Mathematik, Informatik und Statistik der Ludwig–Maximilians–Universit¨at