Modeling and Querying Possible Repairs in Duplicate Detection George Beskales Mohamed A. Soliman...

Modeling and Querying Possible Repairs in Duplicate Detection

George BeskalesMohamed A. Soliman

Ihab F. IlyasShai Ben-David

Data Cleaning Real-world data is dirty

Examples: Syntactical Errors: e.g., Micrsoft Heterogeneous Formats: e.g., Phone number

formats Missing Values (Incomplete data) Violation of Integrity Constraints: e.g., FDs/INDs Duplicate Records

Data Cleaning is the process of improving data quality by removing errors, inconsistencies and anomalies of data

VLDB 2009 George Beskales 2

Duplicate Elimination Process

ID name ZIP Income

P1 Green 51519 30k

P2 Green 51518 32k

P3 Peter 30528 40k

P4 Peter 30528 40k

P5 Gree 51519 55k

P6 Chuck 51519 30k


ID name ZIP Income

C1 Green 51519 39k

C2 Peter 30528 40k

C3 Chuck 51519 30k

Compute PairwiseSimilarity

P1 P2

P3 P4P5

P60.3 0.5

0.9

1.0

Cluster Records

P1 P2

P3 P4P5

P6MergeClusters

C1 C3

C2

Unclean Relation

Clean Relation

Probabilistic Data Cleaning


(a) One-Shot Cleaning

Unclean Database

Deterministic Database

DeterministicDuplicate Elimination

RDBMS

Queries Results

Single Clean Instance

Queries

(b) Probabilistic Cleaning

Unclean Database

Uncertain Database

ProbabilisticDuplicate Elimination

Uncertainty and Cleaning-aware RDBMS

Probabilistic Results



Multiple Possible Clean Instances

Motivation

Probabilistic Data Cleaning:1. Avoid deterministic resolution of

conflicts during data cleaning

2. Enrich query results by considering all possible cleaning instances

3. Allows specifying query-time cleaning requirements







Multiple PossibleClean Instances

Uncertain Database

Uncertain Database



Multiple Possible Clean Instances

Queries

Unclean Database


Probabilistic ResultsQueries



Outline

Generating and Storing the Possible Clean Instances

Querying the Clean Instances

Experimental Evaluation







Uncertainty in Duplicate Detection


Person

Uncertain Clustering

X1

{P1}

{P2}

{P3,P4}

{P5}

{P6}

X2

{P1,P2}

{P3,P4}

{P5}

{P6}

X3

{P1,P2,P5}

{P3,P4}

{P6}

Possible Repairs

Uncertain Duplicates: Determining what records should be clustered is uncertain due to noisy similarity measurements

A possible repair of a relation is a clustering (partitioning) of the unclean relation

ID Name ZIP Income

P1 Green 51519 30k

P2 Green 51518 32k

P3 Peter 30528 40k

P4 Peter 30528 40k

P5 Gree 51519 55k

P6 Chuck 51519 30k

Challenges

1. The space of all possible clusterings (repairs) is exponentially large

2. How to efficiently generate, store and query the possible repairs?


The Space of Possible Repairs


All possible repairs 1

Possible repairs given by any fixed parameterized clustering algorithm

2

Possible repairs given by any fixed hierarchical clustering algorithm

Link-based algorithms

Hierarchical cut clustering

algorithm…

3

Hierarchical Clustering Algorithms

Records are clustered in a form of a hierarchy: all singletons are at leaves, and one cluster is at root

Example: Linkage-based Clustering Algorithm


r1 r2 r4 r5r3

Pair-

wis

e D

ista

nce

Distance Threshold () {r1,r2},{r3,r4,r5}

Uncertain Hierarchical Clustering

Hierarchical clustering algorithms can be modified to accept uncertain parameters

The number of generated possible repairs is linear


r1 r2 r4 r5r3

Possible Thresholds

[l , u ] {r1},{r2},{r3},{r4},{r5}

{r1},{r2},{r3},{r4,r5}

{r1,r2},{r3},{r4,r5}

{r1,r2},{r3,r4,r5}

Probabilities of Repairs


{P1,P2}

{P3,P4}

{P5}

{P6}

{P1,P2,P5}

{P3,P4}

{P6}

{P1}

{P2}

{P3,P4}

{P5}

{P6}

Clustering 1

Clustering 2

Clustering 3

1 3Pr = 0.2

3 10Pr = 0.7

0 1Pr = 0.1

ID Name ZIP Income

P1 Green 51519 30k

P2 Green 51518 32k

P3 Peter 30528 40k

P4 Peter 30528 40k

P5 Gree 51519 55k

P6 Chuck 51519 30k

~U(0,10)

Representing the Possible Repairs


ID … Income C P

CP1 … 31k {P1,P2} [1,3)

CP2 … 40k {P3,P4} [0,10)

CP3 … 55k {P5} [0,3)

CP4 … 30k {P6} [0,10)

CP5 … 39k {P1,P2,P5} [3,10)

CP6 … 30k {P1} [0,1)

CP7 … 32k {P2} [0,1)

U-clean Relation PersonC

{P1,P2}

{P3,P4}

{P5}

{P6}

{P1,P2,P5}

{P3,P4}

{P6}

{P1}

{P2}

{P3,P4}

{P5}

{P6}

Clustering 1

Clustering 2

Clustering 3

1 3Pr = 0.2

3 10Pr = 0.7

0 1Pr = 0.1





Multiple PossibleClean Instances

Uncertain Database

Queries

Unclean Database


Probabilistic ResultsQueries



Outline










Queries over U-Clean Relations

We adopt the possible worlds semantics to define queries on U-clean relations


U-Clean Relation(s) RC

Possible Clean Instances X1,X2,…,Xn

Possible Clean Instances Q(X1), Q(X2),…,Q(Xn)

U-Clean Relation Q(RC)

Definition

Implementation

Instances

Representation

Example: Selection Query


SELECT ID, Income FROM Personc WHERE Income>35k

ID Income C PCP2 40k {P3,P4} [0,10)

CP3 55k {P5} [0,3)

CP5 39k {P1,P2,P5} [3,10)

ID … Income C P

CP1 … 31k {P1,P2} [1,3)

CP2 … 40k {P3,P4} [0,10)

CP3 … 55k {P5} [0,3)

CP4 … 30k {P6} [0,10)

CP5 … 39k {P1,P2,P5} [3,10)

CP6 … 30k {P1} [0,1)

CP7 … 32k {P2} [0,1)

PersonC

Example: Projection Query


Income C P

30k {P1} v {P6} [0,1) v [3,10)

31k {P1,P2} [1,3)

32k {P2} [0,1)

40k{P3,P4} v

{P1,P2,P5} [0,1) v [3,10)

55k {P5} [0,3)

SELECT DISTINCT IncomeFROM Personc

ID …Income C P

CP1 … 31k {P1,P2} [1,3)

CP2 … 40k {P3,P4} [0,1)

CP3 … 55k {P5} [0,3)

CP4 … 30k {P6} [3,10)

CP5 … 40k {P1,P2,P5} [3,10)

CP6 … 30k {P1} [0,1)

CP7 … 32k {P2} [0,1)

PersonC

Example: Join Query


SELECT Income, PriceFROM Personc , Vehiclec

WHERE Income/10 >= Price

Income Price C P

40k 4k {P3,P4} ^ {V4} 1 :[0, 10) ^ 2 :[3,5)

... ... ... ...

ID … Income C P

CP1 … 31k {P1,P2} 1 :[1,3)

CP2 … 40k {P3,P4} 1:[0,10)

… … … … …

ID Price C P

CV5 4k {V4} 2 :[3,5)

CV6 6k {V3,V4} 2 :[5,10)

… …. …. ….

Personc Vehiclec

Aggregation Queries


CP2

CP3

CP4

CP6

CP7

CP1

CP2

CP3

CP4

CP2

CP4

CP5

Pr = 0.1Pr = 0.2

Pr = 0.7

X1 X2 X3

SELECT Sum(Income) FROM Personc

Sum=187k

Sum=156k

Sum=109k

ID … Income C P

CP1 … 31k {P1,P2} [1,3)

CP2 … 40k {P3,P4} [0,10)

CP3 … 55k {P5} [0,3)

CP4 … 30k {P6} [0,10)

CP5 … 39k {P1,P2,P5} [3,10)

CP6 … 30k {P1} [0,1)

CP7 … 32k {P2} [0,1)

Personc

Other Meta-Queries

1. Obtaining the most probable clean instance

2. Obtaining the -certain clusters3. Obtaining a clean instance corresponding

to a specific parameters of the clustering algorithms

4. Obtaining the probability of clustering a set of records together


Outline






A prototype as an extension of PostgreSQL

Synthetic data generator provided by Febrl (freely extensible biomedical record linkage)

Two hierarchical algorithms: Single-Linkage (S.L.) Nearest-neighbor based clustering algorithm (N.N.)

[Chaudhuri et al., ICDE’05]



0

500

1000

1500

2000

0 100000 200000 300000Dataset Size (Records)

Clus

teri

ng T

ime(

sec) Uncertain S.L.

Determinsitic S.L.Uncertain NNDeterminsitic NN



00.20.40.60.8

11.21.41.6


Resp

onse

Tim

e(se

c)AggregateMost Prob InstClustering Probalpha-certain



0%

20%

40%

60%

80%


Ove

rhea

dSelectionProjectionJoin


Conclusion

We allow representing and querying multiple possible clean instances

We modified hierarchical clustering algorithms to allow generating multiple repairs

We compactly store the possible repairs by keeping the lineage information of clusters in special attributes

New (probabilistic) query types can be issued against the population of possible repairs


Modeling and Querying Possible Repairs in Duplicate Detection George Beskales Mohamed A. Soliman...

Documents

Transcript of Modeling and Querying Possible Repairs in Duplicate Detection George Beskales Mohamed A. Soliman...