Improving Data Cleansing Accuracy: A model-based Approach

Mario Mezzanzanica, Roberto Boselli, Mirko Cesarini, and Fabio MercorioDepartment of Statistics and Quantitative Methods - C.R.I.S.P. Research Centre, University of Milan-Bicocca, Milan, Italy

{firstname.lastname}@unimib.it

Keywords: Data and Information Quality; Data Cleansing; Data Accuracy; Weakly-Structured Data

Abstract: Research on data quality is growing in importance in both industrial and academic communities, as it aims atderiving knowledge (and then value) from data. Information Systems generate a lot of data useful for studyingthe dynamics of subjects’ behaviours or phenomena over time, making the quality of data a crucial aspect forguaranteeing the believability of the overall knowledge discovery process. In such a scenario, data cleansingtechniques, i.e., automatic methods to cleanse a dirty dataset, are paramount. However, when multiple cleans-ing alternatives are available a policy is required for choosing between them. The policy design task still relieson the experience of domain-experts, and this makes the automatic identification of accurate policies a signifi-cant issue. This paper extends the Universal Cleaning Process enabling the automatic generation of an accuratecleansing policy derived from the dataset to be analysed. The proposed approach has been implemented andtested on an on-line benchmark dataset, a real-world instance of the Labour Market Domain. Our preliminaryresults show that our approach would represent a contribution towards the generation of data-driven policy,reducing significantly the domain-experts intervention for policy specification. Finally, the generated resultshave been made publicly available for downloading.

1 INTRODUCTION

Nowadays, public and private organizationsrecognise the value of data as a key asset to deeply un-derstand social, economic, and business phenomenaand to improve competitiveness in a dynamic businessenvironment, as pointed out in several works (Foxet al., 1994; Madnick et al., 2009; Batini et al., 2009).In such a scenario, the field of KDD - Knowledge Dis-covery in Databases (KDD) field (Fayyad et al., 1996)- has received a lot of attention from several researchand practitioner communities as it basically focuseson the identification of relevant patterns in data.

Data quality improvement techniques have rapidlybecome an essential part of the KDD process as mostresearchers agree that the quality of data is frequentlyvery poor, and this makes data quality improvementand analysis crucial tasks for guaranteeing the believ-ability of the overall KDD process1 (Holzinger et al.,2013b; Pasi et al., 2013a). In such a direction, thedata cleansing (a.k.a. data cleaning) research areaconcerns with the identification of a set of domain-dependent activities able to cleanse a dirty database

1Here the term believability is intended as ”the extent towhich data are accepted or regarded as true, real and credi-ble”(Wang and Strong, 1996)

(with respect to some given quality dimensions).

Although a lot of techniques and tools are avail-able for cleansing data (e.g., the ETL-based tools2),a quite relevant amount of data quality analysisand cleansing design still relies on the experienceof domain-experts that have to specify ad-hoc datacleansing procedures (Rahm and Do, 2000).

Here, a relevant question is how to guarantee thatcleansing procedures results are accurate, or (at least)it makes sense for the analysis purposes, makingthe selection of accurate cleansing procedures (w.r.t.the analysis purposes) a challenging task that mayconsider the user’s feedback for being accomplishedproperly (Cong et al., 2007; Volkovs et al., 2014).

This paper draws on two distinct worksof (Holzinger, 2012) and (Mezzanzanica et al.,2013). Basically, the former has clarified thatweakly-structured data can be modelled as a ran-dom walk on a graph during the DATA 2012keynote (Holzinger, 2012). This, in turns, has en-

2The ETL (Extract, Transform and Load) process fo-cuses on extracting data from one or more source systems,cleansing, transforming, and then loading the data into adestination repository, frequently a Datawarehouse. TheETL covers the data preprocessing and transformation tasksin the KDD process (Fayyad et al., 1996).

couraged the use of graph-search algorithms for bothverifying the quality of such data and identifyingcleansing alternatives, as shown by the work of(Mezzanzanica et al., 2013) and recently expressedalso in terms of AI Planning problem (Boselli et al.,2014c; Boselli et al., 2014a).

Here we show how the approach of (Mezzanzan-ica et al., 2013) for the automatic identification ofcleansing activities can be enhanced by selecting themore accurate one on the basis of the dataset to becleansed. The technique has been formalised and re-alised by using the UPMurphi tool (Della Penna et al.,2009; Mercorio, 2013). Furthermore, we report ourexperimental results on the basis of the on-line datasetbenchmark provided by (Mezzanzanica et al., 2013),making them publicly available to the community.

2 Motivation and Contribution

The longitudinal data (i.e., repeated observationsof a given subject, object or phenomena at distincttime points) have received much attention from sev-eral academic research communities among the time-related data, as they are well-suited to model manyreal-world instances, including labour and health-care domains, see, e.g. (Hansen and Jarvelin, 2005;Holzinger, 2012; Holzinger and Zupan, 2013; Prinzieand Van den Poel, 2011; Lovaglio and Mezzanzanica,2013; Devaraj and Kohli, 2000).

In such a context, graphs or tree formalisms,which are exploited to model weakly-structured data,are deemed also appropriate to model the expectedlongitudinal data behaviour (i.e., how the data shouldevolve over time).

Indeed, a strong relationship exists betweenweakly-structured and time-related data. Namely, letY (t) be an ordered sequence of observed data (e.g.,subject data sampled at different time t ∈ T ), the ob-served data Y (t) are weakly-structured if and only ifthe trajectory of Y (t) resembles a random walk on agraph (Kapovich et al., 2003; De Silva and Carlsson,2004).

Then, the Universal Cleansing framework intro-duced in (Mezzanzanica et al., 2013) can be used asa model-based approach to identify all the alterna-tive cleansing actions. Namely, a graph modellinga weakly-structured data(set) can be used to derivea Universal Cleanser i.e., a taxonomy of possible er-rors (the term error is used in the sense of inconsistentdata) and provides the set of possible corrections. Theadjective universal would mean that (the universe of)every feasible inconsistency and cleansing action isidentified with respect to the given model.

Table 1: An example of data describing the working careerof a person. P.T. is for part-time, F.T. is for full-time.

Event # Event Type Emplo- Dateyer-ID

01 P.T. Start Firm 1 12th/01/201002 P.T. Cessation Firm 1 31st /03/201103 F.T. Start Firm 2 1st /04/201104 P.T. Start Firm 3 1st /10/201205 P.T. Cessation Firm 3 1st /06/2013. . . . . . . . . . . .

The Universal Cleanser is presented as a foun-dation upon which data cleansing procedures can bequickly built. Indeed, the final step toward the cre-ation of an effective data cleanser requires the identi-fication of a policy driving the choice of which correc-tion has to be applied when an error can be correctedin several ways.

Here we show how a policy can be inferred fromthe dataset to be analysed, by evaluating the possiblecleansing actions over an artificially soiled dataset us-ing an accuracy like metric. This, in turn, enhancesthe Universal Cleansing framework since the cleans-ing procedures can be automatically generated, thusavoiding domain-experts to manually specify the pol-icy to be used.

In this regard, the term accuracy is usually re-lated to the distance between a value v and a valuev′ which is considered correct or true (Scannapiecoet al., 2005). As one might imagine, accessing thedata correct value can be a challenging task, thereforesuch a value is often replaced by an approximation.Our idea is to exploit the part of the data evaluated ascorrect (thus accurate) for deriving a cleansing policyfor cleansing the inconsistent part.

The following example should help in clarifyingthe matter.

Motivating Example. Some interesting informa-tion about the population can be derived using thelabour administrative archive, an archive managedby a European Country Public Administration thatrecords job start, cessation, conversion and extensionevents. The data are collected for administrative pur-poses and used by different public administrations.

An example of the labour archive content is re-ported in Table 1, a data quality examination on thelabour data is shortly introduced. Data consistencyrules can be inferred by the country law and commonpractice: an employee having one full-time contractcan have no part-time contracts at the same time, anemployee can’t have more than k part-time contract atthe same time (we assume k = 2). This consistency

behaviour can be modelled through the graph shownin Fig. 1. Consistent data can be described by a pathon the graph, while a path does not exist for inconsis-tent data.

unemp−−start empPT 1

|PT |= 1empPT 2|PT |= 2

empFT|FT |= 1

st

st

ex

st

cn

cs

ex

cs

ex

cn

cs

Figure 1: A graph representing the dynamics of a job careerwhere st = start, cs = cessation, cn = conversion, and ex =extension. The PT and FT variables count respectively thenumber of part-time and the number of full-time jobs.

The reader would notice that the working historyreported in Table 1 is inconsistent with respect tothe semantic just introduced: a full-time Cessation ismissing for the contract started by event 03. Look-ing at the graph, the worker was in node empFT 1when event 04 was received, however, event 04 bringsnowhere from node empFT 1. A typical cleansing ap-proach would add a full-time cessation event in adate between event 03 and event 04. This would al-low one to reach the node unemp through the addedevent and then the node empPT 1 via event 04, and thiswould make the sequence consistent with respect tothe given constraints. However several other inter-ventions could be performed e.g., to add a conver-sion event from full-time to part-time between event03 and event 04 (reaching node empPT 1). Notice that,in such a case, although the two distinct cleansing ac-tions would make the career consistent, the formerwould create a career having one single part-time con-tract active while the second would make active twodistinct part-time contracts. The question is: how tochoose between these two alternatives? Indeed, al-though the last option may be unusual, a domain ex-pert should take into account such a question sincechoosing one solution at the expense of the other maygenerate inconsistencies in the future, and vice-versa.

The comparison between archive contents and realdata is often either unfeasible or very expensive (e.g.lack of alternative data sources, cost for collecting thereal data, etc.). Then the development of cleansingprocedures based on business rules still represent themost adopted solution by industry although the task ofdesigning, implementing, and maintaining cleansingprocedures requires a huge effort and is an error proneactivity.

In such a scenario, one might look at the consis-tent careers of the dataset where similar situations oc-curred in order to derive a criterion, namely a policy,for selecting the most accurate cleansing option withrespect to the dataset to be cleansed.

Contribution. Here we support the idea that anapproach combining model driven cleansing (whichidentifies all the possible cleansing interventionsbased on a model of data evolution) and a cleansingpolicy (derived from the data and driving the selectionof cleansing alternatives) can strengthen the effective-ness of the KDD process, by providing a more reli-able cleansed dataset to data mining and analysis pro-cesses. Indeed, according to the whole KDD process,the data cleansing activities should be performed af-ter the preprocessing task but before starting the min-ing activities by using formal and reliable techniquesso that this would contribute in guaranteeing the be-lievability of the overall knowledge process (Redman,2013; Sadiq, 2013; Fisher et al., 2012).

This paper contributes to the problem of how toselect a cleansing policy for cleansing data. To thisend, we formalise and realise a framework, built ontop of the Universal Cleansing approach we providedin (Mezzanzanica et al., 2013), that allows one:

• to model complex data quality constraints overlongitudinal data, that represents a challenging is-sue. Indeed, as (Dallachiesa et al., 2013) argue,the consistency requirements are usually definedon either (i) a single tuple, (ii) two tuples or (iii) aset of tuples. While the first two classes can bemodelled through Functional Dependencies andtheir variants, the latter class requires reasoningwith a (finite but not bounded) set of data itemsover time as the case of longitudinal data, and thismakes the exploration-based techniques a goodcandidate for that task;

• to automatically generate accurate cleansing poli-cies with respect to the dataset to be analysed.Specifically, an accuracy based distance is usedto identify the policy that can better restore toits original version an artificially soiled dataset.The policy, once identified, can be used to cleanseoriginal inconsistent datasets. Clearly, severalpolicy selection metrics can be used and conse-quently the analysts can evaluate several cleansingopportunities;

• to apply the proposed framework on a real-lifebenchmark dataset modelling the Italian Manda-tory Communication domain (The Italian Min-istry of Labour and Welfare, 2012) actually used

at CRISP Research Centre, then providing our re-sults publicly available for downloading3.

The outline of this paper is as follows. Sec. 3outlines the Universal Cleansing Framework, Sec. 4describes the automatic policy identification process,Sec. 5 shows the experimental results, Sec. 6 surveysthe related works, and finally Sec. 7 draws the conclu-sions and outlines the future work.

3 UNIVERSAL CLEANSING

Here we briefly summarise the key elements of theframework presented by (Mezzanzanica et al., 2013)that allows one to formalise, evaluate, and cleanselongitudinal data sequences. The authors focus on theinconsistency data quality dimension, which refers to”the violation of semantic rules defined over a set ofdata items or database tuples” (Batini and Scanna-pieco, 2006).

Intuitively, let us consider an events sequence ε =e1,e2, . . . ,en modelling the working example of Tab.1.Each event ei will contain a number of observationvariables whose evaluation determines a snapshot ofthe subject’s state4 at time point i, namely si. Then,the evaluation of any further event ei+1 might changethe value of one or more state variables of si, generat-ing a new state si+1 in such a case. More formally wehave the following.

Definition 1 (Events Sequence). Let R =(R1, . . . ,Rl)be the schema of a database relation. Then,(i) An event e = (r1, . . . ,rm) is a record of the projec-tion (R1, . . . ,Rm) over Q ⊆ R with m ≤ l s.t. r1 ∈R1, . . . ,rm ∈ Rm;(ii) Let ∼ be a total order relation over events, anevent sequence is a ∼-ordered sequence of eventsε = e1, . . . ,ek.A Finite State Event Dataset (FSED) Si is an event se-quence while a Finite State Event Database (FSEDB)is a database S whose content is S =

⋃ki=1 Si where

k ≥ 1.

3.1 Data Consistency Check

Basically, the approach of (Mezzanzanica et al., 2013)aims at modelling a consistent subject’s evolutionover time according to a set of consistency require-ments. Then, for a given subject, the authors verify ifthe subject’s data evolve conforming to the model.

3Available at http://goo.gl/yy2fw34The term “state” here is considered in terms of a value

assignment to a set of finite-domain state variables

To this end the subjects’ behaviour is encoded on atransition system (the so-called consistency model) sothat each subject’s data sequence can be representedas a pathway on a graph. The transition systems canbe viewed as a graph describing the consistent evolu-tion of weakly-structured data. Indeed, the use of asequence of events ε = e1,e2, . . . ,en as input actionsof the transition system deterministically determinesa path π = s1e1 . . . snensn+1 on it (i.e., a trajectory),where a state s j is the state resulting after the appli-cation of event ei on si. This allows authors to castthe data consistency verification problem into a modelchecking problem, which is well-suited for perform-ing efficient graph explorations.

Namely, a model checker generates a trajectoryfor each event sequence: if a violation has been found,both the trajectory and the event that triggered the vi-olation are returned, otherwise the event sequence isconsistent. Generally speaking, such a consistencycheck can be formalised as follows.Definition 2 (ccheck). Let ε = e1, . . . ,en be a se-quence of events according to Definition 1, thenccheck : FSED→ N×N returns the pair < i,er >where:1. i is the index of a minimal subsequence εi =

e1, . . . ,ei such that εi+1 is inconsistent while ∀ j :j ≤ i≤ n, the subsequences ε j are consistent.

2. er is zero if εn is consistent, otherwise it is a nat-ural number which uniquely describes the incon-sistency error code of the sequence εi+1.

By abuse of notation, we denote as f irst{ccheck(S)}the index i of the pair whilst second{ccheck(S)} de-notes the element er.

The authors implemented the ccheck functionthrough model-checking based techniques and ap-plied this approach for realising the (Multidimen-sional) Robust Data Quality Analysis (Boselli et al.,2013).

For the sake of completeness, we remark that theconsistency requirements over longitudinal data de-scribed in (Mezzanzanica et al., 2013) could be ex-pressed by means of FDs. Specifically, to the bestof our knowledge, one might proceed as follows.Let us consider the career shown in Table 1 and letR = (R1, . . . ,Rl) be a schema relation for these data.Moreover, let R be an instance of R and let Pk =R1 oncond R2 . . . oncond Rk be the kth Self Join (wherecond is Ri.Event = Ri+1.Event +1) i.e., P is R joinedk times with itself on the condition cond which (as aneffect) put subsequent tuples in a row (with respect tothe Event attribute). For simplicity we assume Table1to report data on only one career 5.

5The above schema can be modified to manage data of

The kth self join allows one to express constraintson Table 1 data using functional dependencies al-though this presents several drawbacks that we sketchas follows: (1) an expensive join of k relations isrequired to check constraints on sequences of k ele-ments; (2) the start and cessation of a Part Time (e.g.events 01 and 02) can occur arbitrarily many timesin a career before the event 04 is met (where the in-consistency is detected), therefore there can not be avalue of k higher enough to surely catch the incon-sistency described; (3) the higher the value of k, themore the set of possible sequence (variations) to bechecked; (4) Functional Dependencies do not provideany hint on how to fix inconsistencies, as discussedin (Fan et al., 2010).

3.2 Universal Cleanser (UC)

When an inconsistency has been caught, the frame-work performs a further graph exploration for gener-ating corrective events able to restore the consistencyproperties.

To clarify the matter, let us consider again an in-consistent event sequence ε = e1, . . . ,en as the corre-sponding trajectory π = s1e1 . . . snensn+1 presents anevent ei leading to an inconsistent state s j when ap-plied on a state si. What does ”cleansing” such anevent sequence mean? Intuitively, a cleansing eventsequence can be seen as an alternative trajectory onthe graph leading the subject’s state from si to a newstate where the event ei can be applied (without violat-ing the consistency rules). In other words, a cleans-ing event sequence for an inconsistent event ei is asequence of generated events that, starting from si,makes the subject’s state able to reach a new state onwhich the event ei can be applied resulting in a con-sistent state (i.e., a state that does not violate any con-sistency property). For example, considering the in-consistency described in Sec. 1 and the graph shownin Fig. 1, such cleansing action sequences are pathsfrom the node empFT to the node empPT 1 or to thenode unemp. For our purposes, we can formalise thisconcept as follows.

Definition 3 (Cleansing event sequence). Let ε =e1, . . . ,en be an event sequence according to Def. 1and let ccheck be a consistency function according toDef. 2.

Let us suppose that the ε sequence is inconsistent,then ccheck(ε) returns a pair < i,err > with i> 0 andan error-code err > 0.

several careers in the same table and it can be enhanced tomanage sequences shorter than k by using the left outer joininstead of the inner join as well.

A Cleansing event sequence εc is a non empty se-quence of events εc = c1, . . . ,cm able to cleanse theinconsistency identified by the error code err, namelythe new event sequence ε′ = e1, . . . ,ei,c1, . . . ,cm,ei+1is consistent, i.e., second{ccheck(ε′)}= 0.

Then, the Universal Cleanser UC can be seen asa collection that, for each error-code err returns theset C of the synthesised cleansing action sequencesεc ∈ C. According to the author of (Mezzanzanicaet al., 2013), the UC can be synthesised and used forcleansing a dirty dataset as shown in Figure 2(a).

4 AUTOMATIC POLICYIDENTIFICATION

The Universal Cleansing framework proposed by(Mezzanzanica et al., 2013) has been consideredwithin this work since it presents some relevant char-acteristics useful for our purposes, namely:

1. it is computed off-line only once on the consis-tency model;

2. it is data-independent since, once the UC has beensynthesised, it can be used to cleanse any datasetconforming to the consistency model;

3. it is policy-dependent since the cleansed resultsmay vary as the policy varies.

Focusing on (3), when several cleansing interven-tions are available for a given inconsistency, a policy(i.e., a criterion) is required for identifying which onehas to be applied in the case.

The Universal Cleansing framework requires apolicy to drive the data cleansing activities, and thepolicy identification task usually depends on the anal-ysis purposes and the dataset characteristics as well.For these reasons, here we aim at automatically iden-tifying an optimal cleansing policy, so that the datacleansing process (reported in Fig.2(a)) can be ap-plied straightforward.

Clearly, a policy can be optimal if an optimalitycriterion is defined. This concept can be formalisedaccording to the proposed framework as follows.

Definition 4 (Optimal Cleansing Policy). Letε = e1, . . . ,en be an event sequence according toDef. 1 and let ccheck be a consistency functionaccording to Def. 2.Let err ∈ N+ be an error-code identified on ε by theccheck function and let C the set of all the cleansingaction sequences ci ∈ C able to cleanse the error codeerr. A cleansing policy P : N→ 2C maps the errorcode err to a list of cleansing action sequences ci ∈ C .

Cleanse

Incons.

Cleansing

Feedback

Consistency

Model

Cons.

Verification

UC

Generation

S

Dirty

Universal

Cleanser

Cleansing

PolicyS

Cleansed

Sequence

consistent

Inconsistency found

on the sequence

(a) The Universal Cleansing Frameworks, as presented by(Mezzanzanica et al., 2013). The UC is generated from theconsistency model, then a policy is specified so that both thepolicy and the UC can be used for cleansing the data.

Consistency

VerificationAccurate

Policy

S

Source

Cleanse

Dirty

Universal

Cleanser

Policy

Selection

Accuracy

Function

Consist.

Model

(b) The process for identifying an accurate cleansingpolicy. The data sequences considered consistent are fedto the policy selection process that identifies the optimalpolicy according to the specified accuracy function.

Figure 2: (a) The Universal Cleansing framework (taken from (Mezzanzanica et al., 2013)) and the Accurate Policy identifi-cation process as described in Sec.4

.

Finally, let d : N+×C×C→ R be the accuracyfunction, an optimal cleansing policy P ∗ with re-spect to d assigns to each error-code the most ac-curate cleansing action sequence c∗, namely ∀err ∈UC,∀ci ∈ C ,∃caccuratesuch that d(err,c∗,caccurate) ≤d(err,ci,caccurate).

In Def. 4 the accuracy function basically evaluatesthe distance between a proposed corrective sequence(for a given error-code) against the corrective actionsconsidered as accurate (i.e., the corrective sequencethat transforms a wrong data to its real value). Unfor-tunately, the real value is hard to be accessed, as dis-cussed above, therefore a proxy, namely a value con-sidered as consistent according to the domain rules isfrequently used.

In order to automatically identify a policy, we par-tition the source dataset into the consistent and in-consistent subsets respectively. Then, the consistentevent sequences are used for identifying an accuratepolicy so as to cleanse the inconsistent subset. Basi-cally, this can be accomplished through the followingsteps, whose pseudo-code is provided in Procedures1, 2 and 3.

COMPUTE TIDY DATASET. Use the ccheck func-tion for partitioning the source dataset S into Stidy

and Sdirty which represent the consistent and in-consistent subsets of S respectively;

ACCURATE POLICY IDENTIFICATION. For all ε

from Stidy (lines 2-12) and for all event ei ∈ ε

(lines 3-11), generate a new sequence ε′ obtainedfrom ε by removing the event ei (line 5) and verifythe consistency on ε′ (line 6). If ε′ is inconsistentthen use the resulting error-code err for retrieving

the set of cleansing action sequences C from theUniversal Cleanser (line 7);

SYNTHESISE POLICY. Compute the cleansing se-quences minimising the the distance between ciand the event deleted ei (line 1). Once identified,enhance the UC properly. We recall that ei is themost accurate correction (i.e. this is not a proxy)as it has been expressly deleted from the consis-tent sequence.

ACCURATE POLICY IDENTIFICATION. For eacherror-code err of the UC (line 1-3), compute themore accurate ci and add it to the map P indexedby err.

Finally, for the sake of clarity, a graphical repre-sentation of this process has been given in Figure2(b).

Procedure 1 COMPUTE TIDY DATASET

Input: S The Source DatasetOutput: Stidy ⊆ S

1: Stidy← /0

2: for all ε = e1, . . . ,en ∈ S do3: < i,err >← ccheck(ε)4: if err = 0 then5: Stidy← Stidy∪{ε}6: end if7: end for8: return Stidy

5 EXPERIMENTAL RESULTS

Procedure 2 ACCURATE POLICY IDENTIFICATION

Input: Consistency Model,UC Universal Cleanser,ccheck Consistency Function,d Accuracy function,S The Source Dataset, the FSEDB according toDef.1

Output: Cleansing Policy P1: Svalidation← /0 //Will be used in Proc.42: Stidy←COMPUTE TIDY DATASET(S)3: for all ε = e1, . . . ,en ∈ Stidy do4: for all ei ∈ ε do5: ε′i← remove event from(ε,ei)6: < i,err >← ccheck(ε′i) //check ε′i consis-

tency7: if err 6= 0 then8: SYNTHESISE POLICY(UC[err])9: Svalidation← Svalidation∪{ε′i}

10: end if11: end for12: end for13: for all err ∈UC do14: P [err]←minci∈UC[err](UC[err][ci])15: end for16: return P

Procedure 3 SYNTHESISE POLICY

Input: UC[err] The Set of Cleansing Sequences forerr

1: cmin←minci∈UC[err] d(err,ci,ei)2: UC[err][cmin]←UC[err][cmin]+1

In this section the policy generation process de-scribed in Sec. 4 has been used for deriving a policybased upon the on-line benchmark domain providedby (Mezzanzanica et al., 2013). In Sec. 5.1 the do-main model is described while in Sec. 5.2 some pre-liminary results are outlined.

5.1 The Labour Market Dataset

The scenario we are presenting focuses on the Ital-ian labour market domain studied by statisticians,economists and computer scientists at the CRISP Re-search Centre.

According to the Italian law, every time an em-ployer hires or dismisses an employee, or an employ-ment contract is modified (e.g. from part-time to full-time), a Compulsory Communication - an event - issent to a job registry. The Italian public administra-tion has developed an ICT infrastructure (The Ital-ian Ministry of Labour and Welfare, 2012) generat-

ing an administrative archive useful for studying thelabour market dynamics (see, e.g.(Lovaglio and Mez-zanzanica, 2013)). Each mandatory communication isstored into a record which presents several relevant at-tributes: e id and w id are ids identifying the commu-nication and the person involved respectively; e dateis the event occurrence date whilst e type describesthe event type occurring to the worker’s career. Theevent types can be the start, cessation and extensionof a working contract, and the conversion from a con-tract type to a different one; c flag states whether theevent is related to a full-time or a part-time contractwhile c type describes the contract type with respectto the Italian law. Here we consider the limited (fixed-term) and unlimited (unlimited-term) contracts. Fi-nally, empr id uniquely identifies the employer in-volved.

A communication represents an event arrivingfrom the external world (ordered with respect toe date and grouped by w id), whilst a career is a lon-gitudinal data sequence whose consistency has to beevaluated. To this end, the consistency semantics hasbeen derived from the Italian labour law and from thedomain knowledge as follows.

c1: an employee cannot have further contracts if afull-time is active;c2: an employee cannot have more than K part-timecontracts (signed by different employers), in our con-text we shall assume K = 2;c3: an unlimited term contract cannot be extended;c4: a contract extension can change neither the con-tract type (c type) nor the modality (c f lag), for in-stance a part-time and fixed-term contract cannot beturned into a full-time contract by an extension;c5: a conversion requires either the c type or thec f lag to be changed (or both).

5.2 Accurate Policy Generation Results

The process depicted in Fig.2(b) was realised as de-scribed in Sec.4 by using the following as input.

The Consistency Model was specified using theUPMurphi language and according to the consis-tency constraints specified in Sec. 5.1;

The on-line repository provided by (Mezzanzanicaet al., 2013) and containing: (i) an XML datasetcomposed of 1,248,814 records describing thecareer of 214,429 people. Such events have beenobserved between 1st January 2000 and 31st De-cember 2010; and (ii) the Universal Cleanserof the Labour Market Domain collecting all thecleansing alternatives for 342 different error-codes.

An Accuracy Function usually intended as the dis-tance between a value v and a value v′ which isconsidered correct or true. As the consistencywe model is defined over a set of data itemsrather than a single attribute, the accuracy func-tion should deal with all the event attribute val-ues, as we clarified in the motivating example.To this end, the edit distance over the merged at-tribute values was used. Specifically, it is well-suited for comparing two strings by measuringthe minimum number of operations needed totransform one string into another, taking into ac-count insertion, deletion, substitution as well asthe transposition of two adjacent characters. No-tice that the edit distance is often used for mea-suring similarities between phrases, then it is suc-cessfully applied in natural language processing,bio-informatics, etc. Thus, as a mandatory com-munication is composed by strings, the edit dis-tance metric represents a good candidate for real-ising the accuracy function.

The Proc. 1, 2, and 3 of Sec.4 have been imple-mented through C++ while the ccheck function basi-cally calls the UPMurphi planner searching for an in-consistency. The process required about 2 hours run-ning on a x64 Linux-machine, equipped with 6GB ofRAM and connected to a MySQL DMBS for retriev-ing the data.

The output of such a process is an accurate policyaccording to the Def. 4, which we visualise in Fig 3.

The x axis reports the error-codes caught by ap-plying Procedure 2 on Stidy while two y-axes areprovided. The rightmost reports how many timesan error-code has been encountered during the pol-icy generation process using a logarithmic scale (theblack triangle) while the leftmost shows the percent-age of successfully cleansing action applications.

Here, some results are commented as follows.

Result Comments. The error-code numbering wasperformed using a proximity criterion, the similarthe inconsistencies the closer the error codes. Thecoloured bars refer to the leftmost axis showing thepercentage of cases that have been successfully cor-rected using a specific cleansing action. For thisdataset, 17 cleansing actions of the universal cleanserwere applied at least once.

To give a few examples, we discovered that themost adopted cleansing intervention is C11 as it hasbeen applied up to 49,358 times for a situation like thefollowing: a person having only an active part-timecontract with a given company A received a cessationevent for another part-time contract with another com-pany B. This clearly represents an inconsistency for

which the most accurate correction event was C11,namely the introduction of a single corrective eventci = (start,PT,Limited,Company B).

On the contrary, the error-code 84 represents anunemployed subject that starts his career receiving aconversion event of a (non-existing) working contract,namely e1 = (conversion,FT, Unlimited,CompanyA). Although one might easily argue that a start ofa contract with company A is the only thing capableof making career consistent, it might not be so easy toidentify the most accurate attribute values of the eventto be added (e.g., the contract type). Indeed, severalcleansing alternatives were identified as accurate dur-ing the process, namely to consider (C11) a full-timeunlimited contract; (C15) a part-time unlimited con-tract and (C13) a part-time limited one.

Another case worth of mentioning is the error-code 31: a person is working for two companies Aand B and a communication is received about the startof a third part-time contract with Company C. Here,two cleansing alternatives are available: to close thecontract either with A (C2) or with B (C3). This isa classical example that reveals the importance of anaccurate policy identification, as one must define (andmotivate) why a correction has been preferred overanother. Thus, such a kind of result is quite importantfor domain-experts as it represents a criterion derived(automatically) on the basis of the dataset that theyintend to analyse and cleanse.

Considering the results of Fig. 3, for some incon-sistencies there is only a cleansing alternative (e.g.error-codes from 71 to 82). On the other hand, whenmore alternative cleansing are available, very fre-quently one of them greatly outperforms the othersin terms of number of successfully cleansed cases.Moreover, we discovered that similar error-codes aresuccessfully cleansed by the same cleansing actions(e.g. the error codes from 9 to 14, from 59 to66). Generally speaking, the overall plot of Fig.2(b)provides useful information for the development ofcleansing procedures.

Finally, it is worth noting that the edit distancevalue for every cleansed sequence was zero, and thisconfirms that the UC generated is really capable toidentify effective cleansing actions.

5.3 Accuracy Estimation

Here, the k-fold cross validation technique (see,e.g. (Kohavi, 1995)) was used to evaluate the policysynthesised according to the framework proposed.

Generally speaking, the dataset S is split into kmutually exclusive subsets (the folds) S1, . . . ,Sk of ap-proximatively equal size. Then, k evaluation steps are

performed as follows: at each step t the set of eventsS\St is used to identify the cleansing policy Pt , whilstthe left out subset St is used for evaluating the cleans-ing accuracy achieved by Pt . In each step, it is com-puted the ratio of the event sequences the cleanser hasmade equal to the original via Pt over the cardinal-ity of all the cleansed event sequences, namely |St |.Then, an average of the k ratios is computed. Thesesteps are described by the pseudo-code given in Proc.4.

Procedure 4 K-FOLD CROSS VALIDATION

1: S← Svalidation

2: //Step1: generate folds3: for all t ∈ {1, . . . ,k} do4: St ←random sampling(S,k) //such that ∪St =

S and ∩St = /0 and |St |= k5: end for6: //Step2: iterate on folds7: for all t ∈ {1, . . . ,k} do8: match← 09: S′← S\St

10: //Step2.1: synthesise a new policy Pt11: for all ε ∈ S′ do12: SYNTHESISE POLICY(UC[errε])13: end for14: for all err ∈UC do15: Pt [err]←minci∈UC[err](UC[err][ci])16: end for17: //Step2.2: evaluate policy Pt against Pt18: for all ε = e1, . . . ,en ∈ S′ do19: εcleansed ← apply policy(Pt [errε],ε)20: εtidy← retrieve original from(Stidy,ε)21: if εcleansed = εtidy then22: match← match+123: end if24: end for25: Mt ← match

|St |26: end for27: //Step3: compute policy accuracy28: M k← 1

k ∑kt=1 Mt

Step 1: Generate k folds by splitting the Svalidation

dataset (lines 1-5). Notice that the Svalidation wascomputed previously by Procedure 1.

Step 2: For each t up to k, (i) iteratively generate anew policy6 Pt by using only the dataset S \ St asinput (lines 7-16). (ii) Evaluate the instance of ε

cleansed according to the policy Pt , i.e. comprareεcleansed with the original instance of ε as appearsin Stidy, namely εtidy (lines 17-26).

6Notice that the creation of a new policy is not an ex-pensive task as it does not require to invoke ccheck.

Step 3: Compute the overall P accuracy, i.e., M k asthe average of the accuracy values Mt reached byeach fold iteration (line 28).

This method has the advantage of using all theavailable data for both estimating and evaluating thepolicy, during the k steps. Indeed, although k = 10is usually considered as a good value in the com-mon practice, we used a variable value of k rangingin {5,10,50,100} so that different values of accuracyM k can be analysed, as summarised in Tab. 2. Theresults confirm the high-degree of accuracy reachedby the policy P synthesised in Sec.5.2. Indeed, dur-ing the iterations no single M k value was lower than99.4% and peaks of 100% were reached during someiterations.

Table 2: Values of M k applying k-cross validation of Pro-cedure 4.

k 5 10 50 100M k 99.77% 99.78% 99.78% 99.78%

min(M k) 99.75% 99.75% 99.48% 99.48%max(M k) 99.77% 99.80% 99.92% 100%|St | 42,384 21,193 4,238 2,119|S| 211,904

6 RELATED WORK

The data quality analysis and improvement taskshave been the focus of a large body of research in dif-ferent domains, that involve statisticians, mathemati-cians and computer scientists, working in close co-operation with application domain experts, each onefocusing on its own perspective (Abello et al., 2002;Fisher et al., 2012). Computer scientists developed al-gorithms and tools to ensure data correctness by pay-ing attention to the whole Knowledge Discovery pro-cess, from the collection or entry stage to data visual-isation (Holzinger et al., 2013a; Ferreira de Oliveiraand Levkowitz, 2003; Clemente et al., 2012; Foxet al., 1994; Boselli et al., 2014b). From a statisti-cal perspective, data imputation (a.k.a. data editing)is performed to replace null values or, more in gen-eral, to address quality issues while preserving thecollected data statistical parameters (Fellegi and Holt,1976).

On the other hand, a common technique exploitedby computer scientists for data cleansing is the recordlinkage (a.k.a. object identification, record match-ing, merge-purge problem), that aims at linking thedata to a corresponding higher quality version and

to compare them (Elmagarmid et al., 2007). How-ever, when an alternative (and more trusted) data tobe linked are not available, the most adopted solu-tion (specially in the industry) is based on BusinessRules, identified by domain experts for checking andcleansing dirty data. These rules are implemented inSQL or in other tool specific languages. The maindrawback of this approach is that the design relieson the experience of domain experts; thus exploringand evaluating cleansing alternatives quickly becomea very time-consuming task: each business rule hasto be analysed and coded separately, then the overallsolution still needs to be manually evaluated.

In the database area, many works focus onconstraint-based data repair for identifying errors byexploiting FDs (Functional Dependencies) and theirvariants. Nevertheless, the usefulness of formal sys-tems in databases has been motivated in (Vardi, 1987)by observing that FDs are only a fragment of the first-order logic used in formal methods while (Fan et al.,2010) observed that, even though FDs allow one todetect the presence of errors, they have a limited use-fulness since they fall short of guiding one in correct-ing the errors.

Two more relevant approaches based on FDsare database repair (Chomicki and Marcinkowski,2005a) and consistent query answering (Bertossi,2006). The former aims to find a repair, namely adatabase instance that satisfies integrity constraintsand minimally differs from the original (maybeinconsistent) one. The latter approach tries tocompute consistent query answers in response toa query i.e., answers that are true in every repairof the given database, but the source data is notfixed. Unfortunately, finding consistent answers toaggregate queries is a NP-complete problem alreadyusing two (or more) FDs (Bertossi, 2006; Chomickiand Marcinkowski, 2005b). To mitigate this problem,recently a number of works have exploited heuristicsto find a database repair, as (Yakout et al., 2013;Cong et al., 2007; Kolahi and Lakshmanan, 2009).They seem to be very promising approaches, eventhough their effectiveness has not been evaluated onreal-world domains.

More recently, the NADEEF (Dallachiesa et al.,2013) tool has been developed in order to create a uni-fied framework able to merge the most used cleans-ing solutions by both academy and industry. In ouropinion, NADEEF gives an important contribution todata cleansing also providing an exhaustive overviewabout the most recent (and efficient) solutions forcleansing data. Indeed, as the authors remark, con-sistency requirements are usually defined on either a

single tuple, two tuples or a set of tuples. The first twoclasses are enough for covering a wide spectrum ofbasic data quality requirements for which FD-basedapproaches are well-suited. However, the latter classof quality constraints (that NADEEF does not takeinto account according to its authors) requires reason-ing with a (finite but not bounded) set of data itemsover time as the case of longitudinal data, and thismakes the exploration-based technique a good candi-date for that task.

Finally, the work of (Cong et al., 2007), to thebest of our knowledge, can be considered a milestonein the identification of accurate data repairs. Basi-cally, they propose (i) a heuristic algorithm for com-puting a data repair satisfying a set of constraints ex-pressed through conditional functional dependenciesand (ii) a method for evaluating the accuracy of the re-pair. Compared with our work their approach differsmainly for two aspects. First, they use conditionalFDs for expressing constraints as for NADEEF, whilewe consider constraints expressed over more than twotuples at a time. Second, their approach for evaluatingthe accuracy of a repair is based upon the user feed-back. Indeed, to steam the user effort only a sampleof the repair - rather than the entire repair - is pro-posed to the user according to authors. Then a con-fidence interval is computed for guaranteeing a pre-cision higher than a specified threshold. On the con-trary, our approach reduces the user effort in the syn-thesis phase exploiting a graph search for computingaccurate cleansing alternatives.

7 CONCLUDING REMARKS ANDFURTHER DIRECTIONS

In this paper the Universal Cleansing frameworkwas extended in order to enable the automatic iden-tification of an accurate policy (on the basis of thedataset to be analysed) used for choosing the cleans-ing intervention to be applied when there are severalavailable. This represents the final step to achievethe selection of accurate cleansing procedures. Tothis end, we modelled the consistency behaviour ofthe data through UPMurphi, a model checking basedtool used for verifying the data consistency. Then,the whole consistent portion of the source dataset wasmade inconsistent so that several cleansing interven-tions could be evaluated.

As a result, a data-driven policy was obtainedsummarizing only the cleansing actions that minimisethe accuracy function for a given error-code. Here, weused the well-known ”edit distance” metric for mea-suring the accuracy between the real data and the pro-

posed cleansing action, although our approach sup-ports the use of different metrics.

The main benefit of our approach is in the auto-matic policy generation, as it (i) speeds up the devel-opment of cleansing procedures, (ii) provides insightsabout the dataset to be cleansed to domain-experts and(iii) dramatically reduces the human effort requiredfor identifying an accurate policy.

Finally, our approach has been successfully testedfor generating an accurate policy over a real (weakly-structured) dataset describing people working careersand tested according to the well-known k-fold crossvalidation method.

As a further step, we have been working to-ward evaluating the effectiveness of our approach onbiomedical domain. Finally, we intend to deploy ourpolicy on the system actually used at CRISP ResearchCentre, which is based on an ETL tool. This wouldrepresent a contribution toward the realisation of ahybrid approach that combines a mode-based policygeneration with a business-rules-based cleansing ap-proach.

REFERENCES

Abello, J., Pardalos, P. M., and Resende, M. G. (2002).Handbook of massive data sets, volume 4. Springer.

Batini, C., Cappiello, C., Francalanci, C., and Maurino, A.(2009). Methodologies for Data Quality Assessmentand Improvement. ACM Comput. Surv., 41:16:1–16:52.

Batini, C. and Scannapieco, M. (2006). Data Quality: Con-cepts, Methodologies and Techniques. Data-CentricSystems and Applications. Springer.

Bertossi, L. (2006). Consistent query answering indatabases. ACM Sigmod Record, 35(2):68–76.

Boselli, R., Cesarini, M., Mercorio, F., and Mezzanzan-ica, M. (2013). Inconsistency knowledge discov-ery for longitudinal data management: A model-based approach. In SouthCHI13 special session onHuman-Computer Interaction & Knowledge Discov-ery, LNCS, vol. 7947. Springer.

Boselli, R., Cesarini, M., Mercorio, F., and Mezzanzanica,M. (2014a). Planning meets data cleansing. In The24th International Conference on Automated Plan-ning and Scheduling (ICAPS), pages 439–443. AAAI.

Boselli, R., Cesarini, M., Mercorio, F., and Mezzanzan-ica, M. (2014b). A policy-based cleansing and inte-gration framework for labour and healthcare data. InHolzinger, A. and Igor, J., editors, Interactive Knowl-edge Discovery and Data Mining in Biomedical In-formatics, volume 8401 of LNCS, pages 141–168.Springer.

Boselli, R., Cesarini, M., Mercorio, F., and Mezzanzanica,M. (2014c). Towards data cleansing via planning. In-telligenza Artificiale, 8(1):57–69.

Chomicki, J. and Marcinkowski, J. (2005a). Minimal-change integrity maintenance using tuple deletions.Information and Computation, 197(1):90–121.

Chomicki, J. and Marcinkowski, J. (2005b). On the compu-tational complexity of minimal-change integrity main-tenance in relational databases. In Inconsistency Tol-erance, pages 119–150. Springer.

Clemente, P., Kaba, B., Rouzaud-Cornabas, J., Alexandre,M., and Aujay, G. (2012). Sptrack: Visual analysis ofinformation flows within selinux policies and attacklogs. In AMT Special Session on Human-ComputerInteraction and Knowledge Discovery, volume 7669of LNCS, pages 596–605. Springer.

Cong, G., Fan, W., Geerts, F., Jia, X., and Ma, S. (2007).Improving data quality: Consistency and accuracy. InProceedings of the 33rd international conference onVery large data bases, pages 315–326. VLDB Endow-ment.

Dallachiesa, M., Ebaid, A., Eldawy, A., Elmagarmid, A. K.,Ilyas, I. F., Ouzzani, M., and Tang, N. (2013). Nadeef:a commodity data cleaning system. In Ross, K. A.,Srivastava, D., and Papadias, D., editors, SIGMODConference, pages 541–552. ACM.

De Silva, V. and Carlsson, G. (2004). Topological estima-tion using witness complexes. In Proceedings of theFirst Eurographics conference on Point-Based Graph-ics, pages 157–166. Eurographics Association.

Della Penna, G., Intrigila, B., Magazzeni, D., and Mer-corio, F. (2009). UPMurphi: a tool for universalplanning on PDDL+ problems. In Proceeding of the19th International Conference on Automated Plan-ning and Scheduling (ICAPS) 2009, pages 106–113.AAAI Press.

Devaraj, S. and Kohli, R. (2000). Information technol-ogy payoff in the health-care industry: a longitudinalstudy. Journal of Management Information Systems,16(4):41–68.

Elmagarmid, A. K., Ipeirotis, P. G., and Verykios, V. S.(2007). Duplicate record detection: A survey. Knowl-edge and Data Engineering, IEEE Transactions on,19(1):1–16.

Fan, W., Li, J., Ma, S., Tang, N., and Yu, W. (2010). To-wards certain fixes with editing rules and master data.Proceedings of the VLDB Endowment, 3(1-2):173–184.

Fayyad, U., Piatetsky-Shapiro, G., and Smyth, P. (1996).The kdd process for extracting useful knowledge fromvolumes of data. Communications of the ACM,39(11):27–34.

Fellegi, I. P. and Holt, D. (1976). A systematic approach toautomatic edit and imputation. Journal of the Ameri-can Statistical association, 71(353):17–35.

Ferreira de Oliveira, M. C. and Levkowitz, H. (2003). Fromvisual data exploration to visual data mining: A sur-vey. IEEE Trans. Vis. Comput. Graph., 9(3):378–394.

Fisher, C., Laurıa, E., Chengalur-Smith, S., and Wang, R.(2012). Introduction to information quality. Author-House.

Fox, C., Levitin, A., and Redman, T. (1994). The notion ofdata and its quality dimensions. Information process-ing & management, 30(1):9–19.

Hansen, P. and Jarvelin, K. (2005). Collaborative infor-mation retrieval in an information-intensive domain.Information Processing & Management, 41(5):1101–1119.

Holzinger, A. (2012). On knowledge discovery and interac-tive intelligent visualization of biomedical data - chal-lenges in human-computer interaction & biomedicalinformatics. In Helfert, M., Francalanci, C., and Fil-ipe, J., editors, DATA. SciTePress.

Holzinger, A., Bruschi, M., and Eder, W. (2013a). On in-teractive data visualization of physiological low-cost-sensor data with focus on mental stress. In Cuzzocrea,A., Kittl, C., Simos, D. E., Weippl, E., and Xu, L.,editors, CD-ARES, volume 8127 of Lecture Notes inComputer Science, pages 469–480. Springer.

Holzinger, A., Yildirim, P., Geier, M., and Simonic, K.-M. (2013b). Quality-based knowledge discovery frommedical text on the web. In (Pasi et al., 2013b), pages145–158.

Holzinger, A. and Zupan, M. (2013). Knodwat: A scientificframework application for testing knowledge discov-ery methods for the biomedical domain. BMC Bioin-formatics, 14:191.

Kapovich, I., Myasnikov, A., Schupp, P., and Shpilrain, V.(2003). Generic-case complexity, decision problemsin group theory, and random walks. Journal of Alge-bra, 264(2):665–694.

Kohavi, R. (1995). A study of cross-validation and boot-strap for accuracy estimation and model selection. InProceedings of the 14th International Joint Confer-ence on Artificial Intelligence - Volume 2, IJCAI’95,pages 1137–1143, San Francisco, CA, USA. MorganKaufmann Publishers Inc.

Kolahi, S. and Lakshmanan, L. V. (2009). On approximat-ing optimum repairs for functional dependency viola-tions. In Proceedings of the 12th International Con-ference on Database Theory, pages 53–62. ACM.

Lovaglio, P. G. and Mezzanzanica, M. (2013). Classifica-tion of longitudinal career paths. Quality & Quantity,47(2):989–1008.

Madnick, S. E., Wang, R. Y., Lee, Y. W., and Zhu, H.(2009). Overview and framework for data and in-formation quality research. J. Data and InformationQuality, 1(1):2:1–2:22.

Mercorio, F. (2013). Model checking for universal planningin deterministic and non-deterministic domains. AICommun., 26(2):257–259.

Mezzanzanica, M., Boselli, R., Cesarini, M., and Mercorio,F. (2013). Automatic synthesis of data cleansing activ-ities. In Helfert, M. and Francalanci, C., editors, The2nd International Conference on Data Management

Technologies and Applications (DATA), pages 138 –149. Scitepress.

Pasi, G., Bordogna, G., and Jain, L. C. (2013a). An intro-duction to quality issues in the management of webinformation. In (Pasi et al., 2013b), pages 1–3.

Pasi, G., Bordogna, G., and Jain, L. C., editors (2013b).Quality Issues in the Management of Web Information,volume 50 of Intelligent Systems Reference Library.Springer.

Prinzie, A. and Van den Poel, D. (2011). Modeling com-plex longitudinal consumer behavior with dynamicbayesian networks: an acquisition pattern analysis ap-plication. Journal of Intelligent Information Systems,36(3):283–304.

Rahm, E. and Do, H. (2000). Data cleaning: Problems andcurrent approaches. IEEE Data Engineering Bulletin,23(4):3–13.

Redman, T. C. (2013). Data’s credibility problem. HarvardBusiness Review, 91(12):84–+.

Sadiq, S. (2013). Handbook of Data Quality. Springer.

Scannapieco, M., Missier, P., and Batini, C. (2005). DataQuality at a Glance. Datenbank-Spektrum, 14:6–14.

The Italian Ministry of Labour and Welfare (2012).Annual report about the CO system, avail-able at http://www.cliclavoro.gov.it/Barometro-Del-Lavoro/Documents/Rapporto_CO/Executive_summary.pdf.

Vardi, M. (1987). Fundamentals of dependency theory.Trends in Theoretical Computer Science, pages 171–224.

Volkovs, M., Chiang, F., Szlichta, J., and Miller, R. J.(2014). Continuous data cleaning. ICDE (12 pages).

Wang, R. Y. and Strong, D. M. (1996). Beyond accuracy:What data quality means to data consumers. J. ofManagement Information Systems, 12(4):5–33.

Yakout, M., Berti-Equille, L., and Elmagarmid, A. K.(2013). Don’t be scared: use scalable auto-matic repairing with maximal likelihood and boundedchanges. In Proceedings of the 2013 internationalconference on Management of data, pages 553–564.ACM.

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

1

10

100

1000

10000N

um

ber

of p

olic

y a

pplic

ations (

pe

rcenta

ge

)

Num

be

r o

f p

olic

y a

pplic

atio

ns (

log

scale

)

Error-code

C0

C1

C2

C3

C4

C5

C6

C7

C8

C9

C10

C11

C12

C13

C14

C15

C16

Total

Figure 3: A plot showing the cleansing interventions successfully performed for each error code, during the policy selectionprocess performed against the benchmark dataset. The x axis reports the 90 error-codes identified on the soiled dataset.The rightmost y-axis reports how many times an error-code was encountered during the policy generation process using alogarithmic scale (the black triangle) while the coloured bars refer to the leftmost axis showing the percentage of cases thatwere successfully corrected using a specific cleansing action.