The iBench Integration Metadata Generator

Patricia C. Arocena∗

University of Torontoprg@cs.toronto.edu

Boris GlavicIllinois Institute of Technology

bglavic@iit.edu

Radu Ciucanu†

University of Oxfordradu.ciucanu@cs.ox.ac.uk

Renee J. Miller ∗University of Torontomiller@cs.toronto.edu

ABSTRACTGiven the maturity of the data integration field it is sur-prising that rigorous empirical evaluations of research ideasare so scarce. We identify a major roadblock for empiri-cal work - the lack of comprehensive metadata generatorsthat can be used to create benchmarks for different integra-tion tasks. This makes it difficult to compare integrationsolutions, understand their generality, and understand theirperformance. We present iBench, the first metadata genera-tor that can be used to evaluate a wide-range of integrationtasks (data exchange, mapping creation, mapping composi-tion, schema evolution, among many others). iBench per-mits control over the size and characteristics of the meta-data it generates (schemas, constraints, and mappings). Ourevaluation demonstrates that iBench can efficiently gener-ate very large, complex, yet realistic scenarios with differ-ent characteristics. We also present an evaluation of threemapping creation systems using iBench and show that theintricate control that iBench provides over metadata scenar-ios can reveal new and important empirical insights. iBenchis an open-source, extensible tool that we are providing tothe community. We believe it will raise the bar for empiricalevaluation and comparison of data integration systems.

1. INTRODUCTIONDespite the large body of work in data integration, the

typical evaluation of an integration system consists of exper-iments over a few real-world scenarios (e.g., the AmalgamIntegration Test Suite [20], the Illinois Semantic IntegrationArchive, or Thalia [15]) shared by the community, or ad hocsynthetic schemas and data sets that are created for a spe-cific evaluation. Usually, the focus of such an evaluation isto exercise and showcase the novel features of an approach.It is often hard to reuse these evaluations.

∗Supported in part by NSERC BIN.†Supported in part by EPSRC platform grant DBOnto.

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License. To view a copyof this license, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Forany use beyond those covered by this license, obtain permission by emailinginfo@vldb.org.Proceedings of the VLDB Endowment, Vol. 9, No. 3Copyright 2015 VLDB Endowment 2150-8097/15/11.

Patterson [21] states that “when a field has good bench-marks, we settle debates and the field makes rapid progress.”Our canonical database benchmarks, the TPC benchmarks,make use of a carefully designed schema (or in the case ofTPC-DI, a data integration benchmark, a fixed set of sourceand destination schemas) and rely on powerful data gener-ators that can create data with different sizes, distributionsand characteristics to test the performance of a DBMS. Fordata integration however, data generators must be accom-panied by metadata generators to create the diverse meta-data (schemas, constraints, and mappings) that fuel integra-tion tasks. To reveal the power and limitations of integra-tion solutions, this metadata must be created systematicallycontrolling its detailed characteristics, complexity and size.Currently, there are no openly-available tools for systemati-cally generating metadata for a variety of integration tasks.

1.1 Integration ScenariosAn integration scenario is a pair of a source and target

schema (each optionally containing a set of constraints) anda mapping between the schemas. Integration scenarios arethe main abstraction used to evaluate integration systems.Typically in evaluations, the relationship between the sourceand the target schema is controlled to illustrate differentfeatures of a system. We present the two primary ways suchscenarios have been generated and illustrate how they havebeen used in evaluating different integration tasks.

Example 1 (Primitives). Libraries of primitivescan be used to create scenarios of different shapes and sizes.A set of synthetic schema evolution scenarios modeling mi-cro and macro evolution behavior was proposed for evaluat-ing mapping adaptation [25]. A set of finer-grained schemaevolution primitives was proposed for testing mapping com-position [9]. Each primitive is a simple integration scenario.A more general set of mapping primitives was proposed inSTBenchmark, to permit the testing and comparison of map-ping creation systems [2]. The primitives used in each ap-proach depended on the integration task. In STBenchmark,the task was mapping creation. For composition [9], someadditional primitives that were trivial for mapping creation(like an add-attribute primitive) were used because theycan make the composition non-trivial (i.e., the compositionmay be a second-order (SO) mapping [14]).

Using mapping primitives one can either test a single solu-tion or compare multiple solutions on integration scenarioswith different properties (by selecting different subsets ofprimitives). In addition, scalability can be tested by com-bining primitives and applying them repeatedly to generate

108

larger scenarios (larger schemas and mappings). As an al-ternative to using mapping primitives to create integrationscenarios, one can also use ad hoc integration scenarios tai-lored to test specific features of an integration method. Thisapproach was the norm before the primitive approach wasintroduced, and remains common.

Example 2 (Ad Hoc Scenarios). MapMerge [1] is asystem for correlating mappings using overlapping sets oftarget relations, or using relations that are associated viatarget constraints. To evaluate their approach, the authorsused three real biological schemas (Gene Ontology, UniProtand BioWarehouse) and mapped the first two schemas to thethird (BioWarehouse). These real scenarios reveal applica-bility of their technique in practice, but do not allow con-trol over the metadata characteristics (number of mappings,their complexity, etc.). Unfortunately, the primitives of pre-vious approaches did not meet their needs as the mappingscreated by sets of primitives rarely shared target relationsand the scenario generators did not permit detailed controlover how target constraints were generated.

Hence, to evaluate MapMerge, the authors created theirown ad hoc integration scenarios. These were designed tolet the researchers control the degree of sharing (how manymappings used the same target relation) and the set of tar-get schema constraints, the two crucial parameters for theMapMerge algorithm. This set-up permitted control over themapping scenario complexity which was defined as the num-ber of hierarchies in the target schema. However, this defi-nition of complexity is not appropriate for other tasks (e.g.,mapping composition) that do not depend so directly on thecharacteristics of a schema hierarchy. Furthermore, this sce-nario generator is limited to this one specific schema shape.These characteristics make it unlikely that others can benefitfrom this test harness for evaluating their work.

The widespread use of ad hoc scenarios is necessitated bythe lack of a robust metadata generator.

1.2 ContributionsWe present iBench, a flexible integration metadata gener-

ator and make the following contributions.• We define the metadata-generation problem and detailimportant design decisions for a flexible, easy-to-use solu-tion to this problem (Section 2). Our problem definitionincludes the generation of (arbitrary) independent schemaswith an (arbitrary) set of mappings between them, wherethe independent variables controlled by a user include theschema size, the number of mappings (mapping size), themapping complexity, and the mapping language. The prob-lem definition also permits the generation of schemas wherethe relationship between the schemas is controlled in a pre-cise way. Hence, this relationship can be viewed as an in-dependent variable, controlled by a user of the generatorthrough the use of schema primitives.• We show that the metadata-generation problem is NP-hard and that even determining if there is a solution to aspecific generation problem may be NP-hard.• We present MDGen, a best-effort, randomized algorithm,(Section 3) that solves the metadata-generation problem.The main innovation behind MDGen is in permitting flexiblecontrol over independent variables describing the metadatain a fast, scalable way.• We present an evaluation of the performance of iBench(Section 4) and show that iBench can efficiently generate

both large integration scenarios and large numbers of sce-narios. We show that iBench can easily generate scenarioswith over 1 Million attributes, sharing among mappings,and complex schema constraints in seconds. iBench can beeasily extended with new (user-defined) primitives and newintegration scenarios to adapt it to new applications andintegration tasks. We present a performance evaluation ofthis feature where we take several real-world scenarios andscale them up by a factor of more than 103 and combinethese user-defined primitives with native iBench primitives.We show that this scaling is in most cases linear.• We demonstrate the power of iBench by presenting a novelevaluation of three mapping discovery algorithms, Clio [12],MapMerge [1], and ++Spicy [17] (Section 5). Our evalu-ation systematically varies the degree of source and targetsharing in the generated scenarios. This reveals new insightsinto the power (and need for) mapping correlation (used inMapMerge) and core mappings (used in ++Spicy) on com-plex scenarios. As the first generator that varies the amountof generated constraints, we show for the first time how thisimportant parameter influences mapping correlation. Wealso quantify how increasing target keys improves the qual-ity of ++Spicy’s mappings.

2. METADATA-GENERATION PROBLEMWe now motivate our design decisions and formalize the

metadata-generation problem. We use a series of exampleswhich showcase the need for certain features. This culmi-nates in a formal problem statement and a complexity anal-ysis which demonstrates that the problem is NP-hard.

2.1 Generating Integration ScenariosExample 3. Alberto has designed a new mapping inver-

sion algorithm called mapping restoration. Like the Fagininverse [11], not every mapping has a restoration, but Al-berto believes that most mappings do. To demonstrate thisempirically, he would like to compare the success rates of im-plementations for restoration and the Fagin inverse. To dothis, he must be able to generate large sets of mappings.

Alberto’s problem would be solved by a mapping gener-ator. We define an integration scenario as a triple M =(S,T,Σ), where S and T are schemas and Σ is a set of log-ical formulas over (S,T). Relation (and attribute) namesin the schemas may overlap, but we assume we can identifyeach uniquely (for example, using S.R.A and T.R.A). Wegeneralize this definition later (Definition 2) to permit con-straints (e.g., keys, foreign keys), instances of the schemas,and transformations implementing the mappings.

To fulfill Alberto’s requirements, a metadata generator isneeded that can efficiently create integration scenarios. Oneapproach would be to generate schemas and constraints andapply an existing mapping discovery algorithm to generatemappings for these schemas. However, this can be quiteexpensive and the output would be biased based on the id-iosyncrasies of the algorithm. We take a different approachmotivated by data generation and permit a user to controlthe basic metadata characteristics. Ideally, Alberto shouldbe able to specify the values (or admissible intervals of val-ues) for the characteristics he would like to control as aninput to the generator. The metadata generator then has tocreate an integration scenario (or many scenarios) that ful-fill these restrictions. This is only possible if the restricted

109

Parameter Description

πSourceRelSize Number of attributes per source relationπTargetRelSize Number of attributes per target relation

πSourceSchemaSize Number of relations in source schemaπTargetSchemaSize Number of relations in target schema

πNumMaps Number of mappings

πSourceMapSize Number of source atoms per mapping

πTargetMapSize Number of target atoms per mapping

πMapJoinSize No. of attributes shared by two atoms

πNumSkolem % of target variables not used in sourceπSkolemType Type of value invention (all, random, key)

πSkolemSharing % of value inv. fcts shared amg mappings

πSourceSharing % mappings that share source relations

πTargetSharing % mappings that share target relations

Table 1: Scenario Parameters

Parameter Min Max

πSourceRelSize and πTargetRelSize 2 5

πSourceSchemaSize and πTargetSchemaSize 2 10

πNumMaps 1 5

πSourceMapSize 1 1

πTargetMapSize 1 10

πNumSkolem 0% 50%

Table 2: MD task: indep. schemas & LAV mappings

characteristics can be measured efficiently, e.g., the numberof source relations. To define this more precisely, we will as-sume relational schemas and mappings specified as source-to-target tuple-generating dependencies, arguably the mostcommonly used mapping language in integration [24].

A source-to-target tuple-generating dependency [13] (s-ttgd) is a formula of the form ∀x(φ(x) → ∃y ψ(x,y)), wherex and y are disjoint vectors of variables; φ(x) and ψ(x,y)are conjunctions of atomic formulas over the source schemaS and target schema T, respectively. All variables of xare used in φ and all variables of y are used in ψ. S-ttgds are sometimes called global-and-local-as-view (GLAV)mappings [16]. Local-as-view (LAV) mappings have a singleatom in φ whereas global-as-view (GAV) mappings have asingle atom in ψ and no existential variables (y is empty).

A major choice in the design of a metadata generator iswhat scenario characteristics should be controllable by theuser. On the one hand, we would like to give the user fullcontrol over the generation, on the other hand, we do notwant to overwhelm her with a plethora of rarely used op-tions. Clearly, the number of mappings generated and sizeof these mappings (determined as the number of conjuncts)should be controllable. A user may also want to control theincompleteness in a mapping, i.e., the number of existentialvariables used in the target expression (‖y‖). Using thisintuition, one can construct a generator that creates two in-dependent schemas of a specified size (number of relationsand attributes per relation) and mappings (of a specifiedsize) between the schemas. To model this, we use scenarioparameters in our problem definition (Table 1) and allowa user to specify a minimum and maximum value for each.We will explain the last few later in this section.

Definition 1 (MD Task). Let Π be a set of scenarioparameters (shown in Table 1). A metadata-generation taskΓ is a set of constraints of the form minπ < π < maxπwhere π ∈ Π. For a given integration scenario M we saythat M is a solution for Γ if M |= Γ.

Note that we do not require that every scenario charac-teristic is constrained by an MD Task. This enables a userto only vary the characteristics she wants to control.

Example 4. To evaluate his hypothesis that restorationis more successful for LAV than for GLAV mappings, Al-berto creates the MD Task in Table 2 to create LAV map-pings (and later GLAV mappings by increasing parameter

πSourceMapSize) and runs his algorithm over the generatedmappings. A scenario that is a solution for the task in Ta-ble 2 is shown in Figure 1. The red lines represent cor-respondences (mapping variables shared between the sourceand target), ignore the black lines for now. Mappings forthis scenario are m1 and m2.

m1 : ∀a, b Cust[a, b]→ ∃c Customer[c, a, b]m2 : ∀a, b Emp[a, b]→ ∃c Person[c, b]

To see how the compliance of the scenario with the con-straints of the task is evaluated consider πSourceRelSize. Thetask requires that 2 ≤ πSourceRelSize ≤ 5. Since both relationCust and Emp have 2 attributes, the scenario fulfills thisconstraint. As another example, the task specifies that upto 50% of the target variables may be existential (Skolems).Both m1 and m2 fulfill this constraint (1/3 and 1/2).

2.2 Controlling IncompletenessExample 5. Alberto has extended his system to support

second-order (SO) tgds [14], a broader class of mappingswith finer control over incompleteness. He needs a genera-tor that can create SO tgd mappings and control the incom-pleteness. To create SO tgds that are not equivalent to anys-t tgd, he would like to control how arguments of Skolemfunctions representing unknown values are chosen and howSkolem functions are reused among mappings.

While s-t tgds permit incompleteness via existential vari-ables, this incompleteness can also be represented via Sko-lem functions. Indeed, because of this, we called the sce-nario parameter controlling this πNumSkolem. The mappingm1 from Example 4 is equivalent to the following SO tgd.

m1 : ∃f ∀a, b Cust[a, b]→ Customer[f(a, b), a, b]

Using this representation, some limitations of s-t tgds be-come clear. First, all functions depend on all universallyquantified variables in the mapping. Second, two mappingscannot share the same value-invention (Skolem) function(i.e., function f cannot be used in both m1 and m2). Wedefine the metadata-generation problem to include scenariosthat use plain SO tgds [4], a strict superset of s-t tgds withmore flexible value-invention semantics. Arenas et al. [4] ar-gue that plain SO tgds are the right choice for importantintegration problems related to composition and inversion.

A plain SO tgd [4] is an existential second-order formula ofthe form: ∃f( (∀x1(φ1 → ψ1))∧· · ·∧(∀xn(φn → ψn)) ) where(a) each φi is a conjunction of relational source atoms overvariables from xi such that every variable from xi appears inat least one atom and (b) each ψi is a conjunction of atomicformulas that only uses terms of the form T (y1, ..., yk) whereT ∈ T and each yi is either a variable from xi or a termf(x1, ..., xk) where f is a function symbol in f and each xj ∈xi. We refer to each clause of a plain SO tgd, ∀xi(φi → ψi),as a mapping to keep the analogy with s-t tgds.

To control the complexity of SO tgd generation, we use ad-ditional parameters. The πSkolemSharing parameter controlsthe percentage of Skolems (from f) shared between map-pings. The πSkolemType controls how the functions are pa-rameterized: for value All each function depends on all uni-versal variables in the mapping, for Random each functiondepends on a random subset of universal variables, and forKeys each function depends on the primary keys of the map-ping’s source relations. The settings of πSkolemSharing = 0%and πSkolemType = All, limits the mappings to s-t tgds.

110

Example 6. By modifying the parameter πSkolemType inTable 2 to Random and πSkolemSharing to a max of 100%,the following two mappings may be created as a solution.

∃f (∀a, b Cust[a, b], Emp[a, b]→ Insider[f(a), a, b]∀a, b Emp[a, b]→ Person[f(a), b])

In addition to supporting scenario parameters that con-trol schema and mapping size, we let a user limit the map-ping language to the important classes of s-t tgds, LAV (byrestricting the number of source atoms per mapping to 1using parameter πSourceMapSize), and GAV (by restrictingthe number of target atoms per mapping πTargetMapSize = 1and πNumSkolem = 0% ) that have been identified as impor-tant in a survey of mapping languages [24]. Many integra-tion systems do not support all languages, and even whenmultiple mapping languages are supported, a system mayprovide different performance depending on the language.

2.3 Schema PrimitivesExample 7. Alice has built a system that creates exe-

cutable data exchange transformations from s-t tgds. Tothoroughly test her approach, she needs a large number ofdiverse scenarios (schemas and mappings). She decides tofollow Alberto’s methodology to generate large sets of map-pings. For each she creates a transformation script and runsit over source databases (created by a data generator likeToXgene [8]). While she was able to test her system’s per-formance over a wide variety of metadata with little effort,she feels that the performance of her system on independentschemas with random mappings is not necessarily predic-tive of its performance for real world settings. Similar tohow database benchmarks such as TPC-H provide realisticqueries and data she would like to create scenarios that giveher a better understanding of how her system will performon real mappings that are likely to arise in practice.

As Alice in the example, current evaluations do not testintegration systems on arbitrary scenarios, rather they con-trol quite precisely the relationship between the source andtarget schema. This has often been modeled using map-ping primitives. A mapping primitive is a parameterizedintegration scenario that represents a common pattern. Forinstance, vertical and horizontal partitioning are commonpatterns that a metadata generator should support. Prim-itives act as templates that are instantiated based on userinput. A metadata generator should allow a user to deter-mine what primitives to use when generating an integrationscenario. By doing so, the user can control the relation-ship between the schemas. For example, she can use onlyprimitives that implement common transformations used inontologies (e.g., that increase or decrease the level of gener-alization in an isa hierarchy) or she can choose to use onlythose primitives that implement common relational schemaevolution transformations. The two existing mapping gener-ators follow this approach. The first using schema evolutionprimitives proposed by Yu and Popa [25] to test a mappingadaptation system and extended by Bernstein et al. [9] totest a mapping composition system. The second, STBench-mark, used a set of transformation primitives for testingmapping creation and data exchange systems [2].

Example 8. Consider the integration scenario in Fig-ure 1 that could be created using two primitives. The first,copy-and-add-attribute (ADD), creates the source rela-tion Cust(Name, Addr) and copies it to a target relation that

contains another attribute, Customer(Name, Addr, Loyalty).The new attribute Loyalty does not correspond to any sourceattribute. The second primitive, vertical-partitioning-hasA(VH) creates a source relation, Emp(Name, Company), andvertically partitions it into: Person(Id, Name) and WorksAt

(EmpRec,Firm,Id). The VH primitive creates a has-a rela-tionship between the two target relations (modeled by a for-eign key (FK)). Specifically, the primitive VH creates newkeys for the two target relations (Id for Person and EmpRec

for WorksAt) and declares WorksAt.Id to be a FK for Person.

A generator should support a comprehensive set of primi-tives described in the literature and should be extensible toincorporate new, user-defined primitives (see Section 2.5).Furthermore, a generator should avoid proliferation of prim-itives. To achieve this, following the approach of STBench-mark, primitives should be parameterized. Primitives shouldwork seamlessly with scenario parameters. The generatorshould create scenarios that include mappings (relations andconstraints) created by primitives and independent map-pings (relations and constraints) whose generation is con-trolled by the scenario parameters.

Comprehensive Set of Primitives. A partial list ofprimitives is given in Figure 4 (the complete list of 17 primi-tives is in our technical report [6].) We define the metadata-generation problem to use primitives including the mappingprimitives of STBenchmark [2] (e.g., VP) and the schemaevolution primitives [9, 25] (e.g., ADD). We also include asprimitives, ad hoc scenarios that we found in evaluationsin the literature. For example, VA (see Example 2) is theschema hierarchy scenario used to evaluate MapMerge [1].In the MD Task, a user specifies the number of times eachprimitive should be instantiated via a parameter θP (withdefault value of 0) where P is the name of the primitive.

Parameterized Primitives. As done in STBenchmark,primitives can be parameterized to change features like thenumber of relations created. This permits, as a simple ex-ample, a single parameter λNumPartitions (we will use λ forprimitive parameters and θ for setting the number of in-stances of a primitive) to be used to vertically partition arelation into two, three, or more relations.

Example 9. Alice has created the task in Table 3 (left)to create LAV mappings. Notice she has not set the source(or target) schema size parameter or the number of mappingsparameter. Hence, mappings will only be created by invokingprimitives. This configuration will create n source relationsand n mappings for 40 ≤ n ≤ 100 (depending on how manyprimitives are instantiated). The number of target relationsdepends on the number of partitions chosen for each prim-itive instantiation (these primitives use the λNumPartitionparameter so each primitive invocation will create between 2and 10 target relations). Alice finds that her transformationsperform well on the hundreds of scenarios she creates usingthis configuration file. She now wants to test how quicklyperformance deteriorates as the scenarios become less well-behaved with more portions of the schemas and mappingsbeing independent. She now uses the configuration file inTable 3 on the right and creates 200 scenarios. These sce-narios will all have exactly 100 primitive invocations, butin addition will have between 0 and 100 arbitrary mappings(and relations). She can then plot her performance vs. thepercentage of independent mappings (0 to 50% independentmappings).

111

Source

Custaaaa a

Nameaaa

AddraaaaEmpaaaaaa

NameaaaCompany

Targetaaaaa

Customer

Nameaa

AddraaaLoyaltyi

Personaa

Idaa aaa

Nameaa

WorksAtaEmpRec

Firmaaa

Idaa aaa

Figure 1: Random Mapping (-),ADD and VH Primitives (-)

Source

Custaaaa a

Nameaaa

AddraaaaEmpaaaaaa

NameaaaCompany

Executive

Nameaaa

Positiona

Targetaaaaa

Customer

Nameaa

AddraaaLoyaltyi

Personaa

Idaa aaa

Nameaa

WorksAtaEmpRec

Firmaaa

Idaa aaa

Figure 2: ADL with Target Sharing

Source

Sales

Year1QT

2QT

3QT

4QT

Target

Perf

YearQuarter

Amount

Figure 3: A Pivot UDP

Name Description Example FKsADD Copy a relation and add new attributes R(a, b)→ S(a, b, f(a, b)) NoADL Copy a relation, add and delete attributes in tandem R(a, b)→ S(a, f(a)) NoCP Copy a relation R(a, b)→ S(a, b) NoHP Horizontally partition a relation into multiple relations R(a, b) ∧ a = c1 → S1(b)

R(a, b) ∧ a = c2 → S2(b)No

SU Copy a relation and create a surrogate key R(a, b)→ S(f(a, b), b, g(b)) No

VH Vertical partitioning into a HAS-A relationship R(a, b)→ S(f(a), a) ∧ T (g(a, b), b, f(a)) Yes

VI Vertical partitioning into an IS-A relationship R(a, b, c)→ S(a, b) ∧ T (a, c) YesVNM Vertical partitioning into an N-to-M relationship R(a, b)→ S1(f(a), a) ∧M(f(a), g(b)) ∧ S2(g(b), b) Yes

VP Vertical partitioning R(a, b)→ S1(f(a, b), a) ∧ S2(f(a, b), b) Yes

Figure 4: Exemplary Native Primitives

Parameter Min Max

θVH 10 25θV I 10 25θV NM 10 25θV P 10 25λNumPartitions 2 10πSourceRelSize 5 10πTargetRelSize 7 12

Parameter Min Max

θVH 25 25θV I 25 25θV NM 25 25θV P 25 25λNumPartitions 3 3πSourceRelSize 5 10πTargetRelSize 7 12

πNumMappings 100 200

πSourceSchemaSize 100 200πTargetSchemaSize 300 400

Table 3: Example MD Tasks

Primitives and Scenario Parameters. To satisfy Alice’sneeds, unlike previous metadata generators (e.g., STBench-mark) where the scenario is generated as a union of a numberof primitive instantiations, we define the metadata genera-tion problem in a more flexible fashion where the user canrequest a number of instances for each primitive type and/oruse scenario parameters to generate independent metadata.

2.4 Sharing Among MappingsExample 10. Alice notices an anomaly in her results.

Even though her transformations remove subsumed (redun-dant) tuples created by older transformation systems likeClio [22], varying the amount of redundancy in the sourceinstances does not affect performance for schemas createdonly using primitives. She realizes that no target relation isbeing populated by more than one mapping, i.e., there is noredundancy created in the target. She has forgotten to set animportant iBench parameter controlling the degree to whichmappings can share source or target relations.

By combining multiple instances of the primitives, a usercan generate diverse integration scenarios with a great dealof control over the scenario characteristics. However, whilereal-world schemas contain instances of these primitives,they often contain metadata that correspond to a combi-nation of primitives. To create such realistic scenarios, it isimportant to permit sharing of metadata among mappingsand control the level of sharing. For example, in a real-world scenario typically some relations may be used by mul-tiple mappings (e.g., employees and managers from a sourceschema are both mapped to a person relation in the target

schema), but it is uncommon for all mappings to share asingle source relation. This type of sharing of metadata hasbeen recognized as an important feature by Alexe et al. [2],but they tackle this issue by providing some primitives thatcombine the behavior of other simpler primitives.

Example 11. Suppose we apply a third primitive, copy-add-delete-attribute (ADL). Without sharing, this prim-itive would create a new source and target relation wherethe latter is a copy of the former excluding (deleting) somesource attribute(s) and creating (adding) some target at-tribute(s). With sharing, primitives can reuse existing re-lations. If we enable target sharing, then an application ofADL may create a new source relation Executive and copyit into an existing target relation. In our example, see Fig-ure 2, the existing target Person is chosen (this relation ispart of a VH primitive instance). ADL deletes the sourceattribute Position and adds the target attribute Id. Byaddition, we mean that no attribute of the source relationExecutive is used to populate this target attribute. Noticethat the resulting scenario is a very natural one. Parts of thesource relations Emp and Executive are both mapped to thetarget Person relation while other parts (other attributes) ofthese source relations are partitioned into a separate relation(in the case of Emp) or removed (in the case of Executive).

To create realistic metadata, a generator could create newprimitives that represent combinations of the existing primi-tives (as done in STBenchmark). However, implementing allpossible combinations is infeasible in terms of implementa-tion effort. In addition, it is likely overwhelming for a user tochoose from long lists of primitives, making such a generatorhard to use. We define the metadata-generation problem us-ing an alternative route under which a user is able to controlthe amount of source or target sharing among invocationsof the primitives and among independent mappings.

2.5 User-Defined PrimitivesExample 12. Alice wishes to test her solution on map-

pings that include a pivot. A simple example of this is de-picted in Figure 3. There are four separate source attributes

112

representing quarters, each containing the sales amount forthat quarter. In the target, information about quarters isrepresented as data (the Quarter attribute). The desiredmapping is mp (omitting the universal quantification).

mp : Sales(Y,1QT,2QT,3QT,4QT) → Perf(Y, ’1’, 1QT),Perf(Y, ’2’, 2QT), Perf(Y, ’3’, 3QT), Perf(Y, ’4’, 4QT)

Sharing provides great flexibility in creating realistic andcomplex scenarios. However, a user may sometimes requireadditional control. For example, a proposed integrationtechnique may be designed to exploit scenarios with a veryspecific shape. Despite our attempt to be comprehensive increating primitives, there may be other shapes of interest. Inaddition, it is often helpful to use real scenarios (for example,the real schemas currently being used to evaluate integrationsystems) as building blocks in creating integration scenar-ios. We do not see metadata generation as a replacement forthe use of real schemas, real mappings and real data, ratherwe would like generation to make such real scenarios evenmore valuable. Hence, we include user-defined primitives(UDP) in the definition of the metadata generation problem.

The advantage of supporting UDPs is that we can system-atically scale these scenarios (creating new scenarios withmany copies of the new primitive) and can combine themwith other native primitives or other UDPs. All native prim-itives and UDPs should be able to share metadata elements.

Example 13. By creating a new primitive Pivot, Alicecan now combine this primitive with other primitives to cre-ate large scenarios with many pivots and can combine thesewith vertical or horizontal partitioning. If desired, the UDPsand these native primitives may share metadata (relations).This permits her to understand the impact of pivoting (ac-curacy and performance) on her integration task.

In Section 4.2, we will show how some already commonlyused real scenarios can be turned into UDPs.

2.6 Generation of Other MetadataExample 14. Patricia has designed a new algorithm to

rewrite SO tgds into s-t tgds. Her algorithm uses functionaldependencies (FDs) to rewrite SO tgds that could not berewritten without these additional constraints. The algo-rithm is sound, but not complete. To test her algorithm,she needs to not only be able to generate SO tgds, but alsovary the amount of source and target constraints. We havedefined the metadata-generation problem to include not onlymappings, but also constraints and as a result, she was ableto use an early prototype of iBench to generate 12.5 millionSO tgds and evaluate how the success rate of her algorithmis affected by the number of constraints [7].

Primitives permit the controlled generation of specific con-straints. In addition a metadata generator should be able togenerate arbitrary constraints such as FDs (including keys)and INDs (including foreign-keys). The user must be ableto control the type and complexity of constraints (e.g., thenumber of attributes in key constraints or the number ofnon-key functional dependencies). Hence, the full list ofscenario parameters (28 included in our technical report)also includes parameters for this purpose [6]. In addition,we have scenario parameters that control other parts of themetadata generation process like renaming. By default,primitives (and independent mappings) reuse source namesin the target and invent random names (using the name gen-erator of STBenchmark) for target attributes and relations

that are not mapped from the source. This can be changedto invoke renaming on some or all mappings. Renaming isimplemented as a plug-in, we plan to include more flexiblename generators in the future.

In addition, a user may request the generation of data(just source, or both source and target), and transformationcode that implements the generated mapping. We now givethe complete definition of an integration scenario.

Definition 2. An integration scenario as a tuple M =(S,T,ΣS,ΣT,Σ, I, J, T ), where S and T are schemas, ΣS

and ΣT are source and target constraints, Σ is a mappingbetween S and T, I is an instance of S satisfying ΣS, J isan instance of T satisfying ΣT, and T is a program thatimplements the mapping Σ.

A user may request any subset of scenario metadata types.In addition, she may specify whether J should be a solutionfor I (meaning (I, J) |= Σ and T (I) = J) or whether Jshould be an independent instance of T.

2.7 Complexity and UnsatisfiabilityWe now formally state the metadata-generation problem

and investigate its complexity, showing that it is NP-hardin general. Furthermore, we demonstrate that there existtasks for which there is no solution. Proofs to the theoremscan be found in our technical report [6]. These results haveinformed our MDgen algorithm presented in Section 3.

Definition 3 (Metadata-Generation Problem).Given an MD Task Γ, the metadata-generation problem isto produce a solution M for Γ.

Theorem 1 (Complexity). The metadata-generationproblem is NP-hard.

Furthermore, the problem of checking whether a scenarioMis a solution for a task is also NP-hard.

Theorem 2. Let Γ be a task and M a scenario. Theproblem of checking whether M |= Γ is NP-hard.It is possible to define tasks for which no solution exists. Forinstance, consider a task with no target sharing, a targetschema of size one, and two CP primitive instances. Thereis no integration scenario that is a solution for this task,because for a scenario to contain two instances of the CPprimitive with no sharing of target relations, the scenariohas to contain at least two target relations.

3. THE MDGEN ALGORITHMWe now present our MDGen algorithm that solves the

metadata-generation problem for an input task Γ. Sincethe problem is NP-hard in general, a polynomial-time so-lution must either be an approximation or incomplete. Asdiscussed in Section 2.7, the user may accidentally specifya task that has no solution. We could deal with this byaborting the generation and outputting the conflicting re-quirements (to help a user see what needs to be relaxedto avoid conflicts). Or alternatively, we can relax the con-straints (e.g., by increasing the relation size) and producea solution for the relaxed constraints. We have chosen thesecond approach since we found it more useful in using thegenerator ourselves, as a relaxation that increases the sizeof one or more scenario parameters is usually sufficient. Wepresent a greedy algorithm where we never reverse a deci-sion once it has been made. Since our algorithm is best-effort(and checking if a solutions exists is hard), we might relaxthe constraints even when an exact solution exists.

113

Algorithm 1 MDGen (Γ)

1: Initialize empty Integration Scenario M{Instantiate Primitives}

2: for each θ ∈ Θ do3: numInst = random(minθ,maxθ)4: for i ∈ {1, . . . ,numInst} do5: InstantiatePrimitive(θ,M,Γ)

{Add Independent Mappings/Relations}6: RandomMappings(M,Γπ){Value Invention}

7: ComplexValueInvention(M,Γ){Generate Additional Schema Constraints}

8: GenRandomConstraints(M,Γ){Rename Schema Elements}

9: ApplyRenamingStrategy(M,Γ){Generate Data}

10: for each S ∈ {S,T} do11: for each R ∈ S do12: genData(M, R,Γ)

3.1 OverviewAlgorithm 1 (MDGen) takes as input an MD Task Γ. We

first instantiate primitives (native and UDPs). For eachθ ∈ Θ, we uniformly at random create a number of instances(described in more detail in Section 3.2) that lies betweenminθ and maxθ (Lines 1 to 2 of Algorithm 1). During thisprocess we keep track of the characteristics of the scenariobeing generated. We always obey the restrictions on thenumber of primitive instantiations and primitive parame-ters, but allow violations of scenario parameters. Thus, weguarantee that a returned solution obeys ΓΘ and Γλ (whichis always possible), but not that this solution fulfills ΓΠ.To create independent parts of the schemas and mappings,we use a primitive called random mappings (Line 6). Thisprimitive takes as input the number of source relations, tar-get relations, and mappings to create. These values are setbased on ΓΠ minus the number of relations (mappings) thathave created been during primitive instantiation. If we havealready created larger schemas or more mappings than re-quested by ΓΠ, then additional metadata is not created (andthe solution may violate the respective scenario parameters).We found this solution most satisfying for users. In general,if the relationship between the source/target schemas doesnot matter for an evaluation, then the user will just use sce-nario primitives (recall Example 4 where Alberto just neededa large set of random mappings over independent schemas)and the generator will always find a correct solution. How-ever, if the relationship matters and primitives are used, wealways satisfy the primitive constraints, and relax, if neces-sary, scenario restrictions.

Additional scenario parameters are handled within sepa-rate post-processing steps which are applied to the scenario(with mappings) that we have created. These include valueinvention, the generation of additional constraints, and therenaming of target relations and attributes (if requested).For reasons of space, we only explain one exemplary of thesepostprocessing steps, value invention, in Section 3.3.

We also generate data for the source schema (by call-ing ToXgene, a general purpose data generator [8]). Theuser can control the number of tuples generated per relation(via the scenario parameter πRelInstSize). In the future, weplan to give the user control over which value generatorsare used to create attribute values (types of data generators

Algorithm 2 InstantiatePrimitive (θ,M,Γ)

1: {Determine Sharing}2: sourceShare = (random(0, 100) < πsourceShare)3: targetShare = (random(0, 100) < πtargetShare){Determine Restrictions on Scenario Elements}

4: Req = determinePrimReqs(θ,Γ){Generate Source Relations}

5: for i ∈ {1, . . . , Req(‖S‖)} do6: if sourceShare = true then7: tries = 08: while R 6|= Req ∧ tries+ + < MAXTRIES do9: R = pickSourceRelation(M,Γ, i)

10: end while11: if R 6|= Req then12: R = genSourceRelation(M, Req, i)

13: else14: R = genSourceRelation(M, Req, i)

{Generate Target Relations}15: Repeat Lines 5 to 14 using Target Schema{Generate Mappings}

16: for i ∈ {1, . . . , Req(‖ΣST ‖)} do17: σ = genMapping(M, Req, i)

{Generate Transformations and Constraints}18: genFKs(M, Req)19: genTransformations(M, Req)

and the probability of using them). However, this is notimplemented in the current release.

3.2 Instantiate PrimitivesAs mentioned before, our algorithm is greedy in the sense

that once we have created the integration scenario elementsfor a primitive instance we never remove these elements.In this fashion, we iteratively accumulate the elements ofa scenario by instantiating primitives. Algorithm 2 is usedto instantiate one primitive of a particular type θ. WhenπSourceShare or πTargetShare are not set (their default is0), each primitive instantiation creates new source and/ortarget relations. Otherwise, we achieve reuse among map-pings by using relations created during a previous primi-tive instantiation instead of generating new relations. Thechoice of whether to reuse existing relations when instanti-ating a primitive is made probabilistically (Lines 2 and 3)where πSourceShare (respectively, πTargetShare) determinesthe probability of reuse. Once we have determined whetherwe would like to reuse previously generated schema elementsor not, the next step is to determine the requirements Reqon scenario elements based on the primitive type θ we wantto instantiate, the scenario parameters ΓΠ, and primitiveparameters Γλ (Line 4). Recall that our algorithm makes abest-effort attempt to fulfill the input task restrictions. Toavoid backtracking and to resolve conflicts between primitiverequirements and the scenario restrictions our algorithm vio-lates scenario parameters if necessary. For instance, assumethe user requests that source relations should have 2 at-tributes (πSourceRelSize) and that VP primitives should splita source relation into three fragments (πNumPartitions). Itis impossible to instantiate a VP primitive that fulfills theseconditions, because to create three fragments, the source re-lation has to contain at least three attributes. We definea precedence order of parameters and relax restrictions ac-cording to this order until a solution is found. In this exam-ple, we would choose to obey the restriction on πNumPartitionsand violate the restriction on πSourceRelSize. In Lines 5 to

114

Algorithm 3 ComplexValueInvention (M,Γ)

1: V I ← attribute→ Skolem {Map from attributes to Skolems}{Associate Victim Attributes With Skolems}

2: for each i < (getNumAttrs(S) ·ΠSkolemSharing) do3: S.A = pickRandomAttr(S) {Pick random attribute}4: ~A = pickRandomAttr(S, πSkolemType)5: f = pickFreshSkolemName()

6: Add(VI, S.A, f( ~A))

{Adapt Mappings}7: for each σ ∈ Σst do8: for each (S.A→ f( ~A)) ∈ V I do9: if S ∈ σ then

10: xA ← vars(σ, S.A)

11: args← vars(σ, ~A)

12: term← f( ~A)[ ~A← args]13: for each R(~x) ∈ RHS(σ) do14: R(~x)← R(~x)[xA ← term]

14, we generate source relations from scratch or reuse ex-isting source relations. Since not every source relation canbe reused (it may not fit the requirements of the primitive)we select relations and check whether they meet the require-ments for the current primitive. These source relations arenot picked completely at random, but we only choose amongrelations that fulfill basic requirements of the primitive (e.g.,a minimum number of attributes) to increase the chance thatsharing is successful. After MAXTRIES tries we fall back togenerating a source relation from scratch. When generat-ing a relation we also generate primary key constraints ifrequired by the primitive or requested by the user. Thegeneration of target relations (Lines 15) is analogous to thesource relation generation. Then, we generate mappings be-tween the generated source and target relations. The struc-ture of the generated mappings is given by the primitivetype (see Table 4) and further influenced by certain primi-tive and scenario parameters. For instance, θNumPartitionsdetermines the number of fragments for a VP primitive andπSourceRelSize determines the number of attributes per rela-tion. The generation of transformations is similar.

3.3 Value InventionPart of the value invention process has already been com-

pleted while instantiating primitives. Based on the param-eter πSkolemType, we have parameterized value invention inmappings as described in Section 2.2. In addition, we use thescenario property πSkolemSharing to control how much addi-tional sharing of Skolem functions we inject into mappings.Algorithm 3 outlines the process. We randomly select at-tributes from the generated source schema (called victims),and associate them with fresh Skolem terms that depend onother attributes from this source relation (Lines 2-6). Foreach selected attribute S.A, the attributes which are usedas the arguments of the Skolem function are chosen basedon parameter πSkolemType, e.g., if πSkolemType = Keys thenwe use the key attributes of relation S. In Lines 7-14, werewrite any mappings that use a victim to use the generatedSkolem term instead. Here ψ[x ← y] denotes the formuladerived from ψ by replacing all occurrences of x with y.For instance, assume the Skolem term f(a) was assigned toattribute b of relation S(a, b). We would transform the map-ping S(x1, x2) → T (x1, x2) into S(x1, x2) → T (x1, f(x1)).Hence, if S is used in three mappings that exchange b, thenthe Skolem function f will be shared by these mappings.

3.4 Complexity and CorrectnessAside from solving the MD generation problem, the main

goal in the design of MDGen was scalability. MDGen runsin PTime in the size of the generated integration scenario.

Theorem 3 (Complexity). The MDGen algorithmruns in time polynomial in n, where n is the maximum ofthe number of created relations and created mappings.

Since by design the algorithm may violate input scenario pa-rameters, in order to produce a solution without backtrack-ing, a main question is what kind of correctness guaranteescan be given for the solution returned by the algorithm un-der such relaxations of the inputs. If the algorithm does nothave to relax any restriction, then the scenario produced bythe algorithm is a solution to the input task.

Theorem 4 (Correctness). Let Γ be an MD Task.MDgen fulfills the following correctness criteria.• If MDgen returns a result without relaxing any restric-

tions given by the task, then the result is a solution for Γ.• A solution returned by the algorithm always conforms to

the primitive restrictions given in the input task.• Let Γrelax be the MD Task generated by replacing the re-

strictions in the task Γ with their relaxed version produced bythe algorithm when running over Γ. The scenario returnedby the algorithm is a solution for Γrelax.

4. iBENCH EVALUATIONWe now evaluate the scalability of iBench for various tasks,

using native primitives and UDPs. We ran our experimentson an Intel Xeon X3470 with 8× 2.93 GHz CPU and 24GBRAM, reporting averages over 100 runs.

4.1 Native Metadata ScenariosWe conducted four experiments to investigate the influ-

ence of the schema size on the time needed to generate largemetadata scenarios. All four experiments share a baselineconfiguration, that uses the same parameter settings for thenumber of attributes per relation (5-15), the same range forthe size of keys (1-3), same size for mappings (2-4 source re-lations and 2-4 target relations). We generated scenarios ofvarious sizes (100 up to 1M attributes where we use the sumof source and target attributes as a measure of schema size)by using the baseline configuration and varying the numberof primitives. In these experiments, iBench always producedsolutions satisfying all configuration constraints.

Figure 5(a) shows the metadata generation time in sec-onds for generating four types of scenarios (on logscale). Thefirst configuration denoted as (0,0,0) has no constraints, nosource sharing, and no target sharing. The other three con-figurations are created by introducing 25% FD constraints(25,0,0), 25% source sharing (0,25,0), and 25% target shar-ing (0,0,25), respectively. iBench can generate scenarios with1M attributes in 6.3 sec for the (0,0,0) configuration andshows a linear trend as expected. For the 25% constraintconfiguration, we also observe a linear trend: 13.89 sec fora 1M attribute scenario. For the source and target shar-ing configurations, we observed a non-linear trend. WhileiBench generates scenarios with 100K attributes in 2.1 and2.5 sec, respectively, for 1M attributes, iBench requires sev-eral minutes. Here, we noticed high variance in elapsedtimes: 215.91 sec with a standard deviation of 11.67 secfor source sharing, and 213.89 sec with a standard devia-tion of 14.14 sec for target sharing. This variance is due

115

10-2

10-1

100

101

102

103

102 103 104 105 106

Genera

tion T

ime (

Seco

nds)

# Generated Attributes

(0, 0, 0)(25, 0, 0)(0, 25, 0)(0, 0, 25)

(a) Gen. Time (s) - Native

0

10

20

30

0 200 400 600 800 1000

Genera

tion T

ime (

Seco

nds)

Instances of Loaded Primitive

A1A2A3FH

B1B2B3

(b) Gen. Time (s) - UDPs

10K

20K

30K

40K

50K

00 200 400 600 800 1000

# G

enera

ted R

ela

tions

Instances of Loaded Primitive

A1A2A3FH

B1B2B3

(c) Gen. Relations: UDPs

50K

100K

150K

200K

00 200 400 600 800 1000

# G

enera

ted A

ttri

bute

s

Instances of Loaded Primitive

A1A2A3FH

B1B2B3

(d) Gen. Attributes: UDPs

Figure 5: iBench Scalability Results for Native and UDP Primitives

0.5M

1M

1.5M

2M

2.5M

0CM CM CM CM CM CM CM CM CM CM

Targ

et

Siz

e (

# A

tom

s)

Ontology STB Mapping Primitives

Constants Nulls

28272625242827262524

(a) Ontology+STB: #Mappings

0

0.1

0.2

0.3

0.4

0.5

24 25 26 27 28R

el. Im

pro

vem

en

t

Mapping Primitives

Ont-TotalOnt-NullsSTB-TotalSTB-Nulls

(b) Rel. Improv.: 6(a)

2-2

20

22

24

26

28

210

212

24 25 26 27 28

Tim

e (

Seco

nd

s)

Mapping Primitives

Ont-COnt-MSTB-CSTB-M

(c) Gen. Time: 6(a)

23

25

27

29

211

213

24 25 26 27 28

Tim

e (

Seco

nd

s)

Mapping Primitives

Ont-COnt-MSTB-CSTB-M

(d) Exec. Time: 6(a)

Figure 6: Scalability of Clio (C) and MapMerge (M)

to the greedy, best-effort nature of the sharing algorithm.Despite this, we are still able to generate in reasonable timescenarios that are 10-100 times larger, and with much morerealistic sharing among mappings, than used in any previousevaluation of which we are aware.

4.2 UDP-based Metadata ScenariosTo analyze the performance of iBench’s UDP feature, we

used seven real-life metadata scenarios from the literatureand we scale them by factors of up to 1,000 times. Thefirst three UDPs are based on the Amalgam Schema andData Integration Test Suite [20], which describes metadataabout bibliographical resources. We denote them by A1,A2, and A3. The next three UDPs are based on a biologicalmetadata scenario called Bio [3], which employs fragmentsof the Genomics Unified Schema GUS (www.gusdb.org) andthe generic Biological Schema BioSQL (www.biosql.org).We denote them by B1, B2, and B3. The last UDP (denotedas FH) is based on a relational encoding of a graph dataexchange setting [10], comprising information about flights(with intermediate stops) and hotels. For each UDP, wevary the number of instances from 0 to 1,000. In all cases,we also requested 15 instances of each native primitive. Wepresent the generation time in Figure 5(b), and the numbersof generated relations and attributes in Figure 5(c) and 5(d),respectively. As we scale the number of loaded UDPs, themetadata generation time grows linearly in most cases. Weobserve the worst behavior for A2 and A3, which contain thelargest number of relations and up to 40 FKs (Figure 5(c)).While profiling our code, we realized that this non-linearbehavior is due to a limitation in the Java-to-XML bindinglibrary we used for manipulating relations in the UDP andwill be optimized for future iBench releases.

5. INTEGRATION SYSTEM EVALUATIONTo showcase how iBench can be used to empirically evalu-

ate schema mapping creation, we present a novel evaluationcomparing MapMerge [1] against Clio [12] and ++Spicy [17].

Systems. Clio creates mappings and transformations thatproduce a universal solution, MapMerge correlates Clio map-pings to remove data redundancy, while ++Spicy createsmappings and transformations that attempt to produce coresolutions [19]. We obtained Clio and MapMerge fromthe authors and downloaded ++Spicy from the web (www.db.unibas.it/projects/spicy). Since these three systemscreate mappings, we used iBench to generate schemas, schemaconstraints, mappings, and instance of the source schemas.We omitted the iBench mappings from the input, so eachsystem created its own mappings and own transformationcode (which we ran over the iBench data to create targetdata that we could compare). We used an Intel Core i7with 4×2.9 GHz CPU and 8 GB RAM. Notice that this is adifferent machine than the rack server used in Section 4 sinceall three systems we compare required input given througha user-interface. As this approach is more human labor-intensive, here we report averages over 5 runs.

Original MapMerge Evaluation. The original evalua-tion compared transformations that implement Clio’s map-pings (input to MapMerge) with transformations that im-plement correlated mappings produced by MapMerge. Theevaluation used two real-life scenarios of up to 14 mappings,and a synthetic scenario (Example 2) with one source rela-tion, up to 272 binary target relations and 256 mappings.It was concluded that MapMerge improved the quality ofmappings by both reducing the size of the generated targetinstance, and increasing the similarity between the sourceand target instances. We implemented the synthetic sce-nario used in the original evaluation as an iBench primitive(VA) both to further our goal of having a comprehensiveset of primitives and to perform a sanity (reproducibility)check [6]. We obtained the same generated instance size andcomparable times to the original evaluation (these measuresare described below). Our goal in the new experiments wasto test the original experimental observations over more di-verse and complex metadata scenarios. In particular, we useiBench to generate scenarios with a variable number of con-

116

straints and variable degree of sharing to explore how theseimportant factors influence mapping correlation.

Original ++Spicy Evaluation. ++Spicy evaluated thequality of its solutions using four fixed scenarios from theliterature with 10 (or fewer) tgds and 12 (or fewer) egds.They compared the size of core solutions that were com-puted w.r.t. tgds only with the size of core solutions thatused both tgds and egds. They also used synthetic scenar-ios (created using STBenchmark and similar to our STB sce-narios described below), to evaluate the scalability of theiralgorithms. While their scalability experiments control thenumber of constraints, their quality evaluation did not. Ourgoal was to quantify the quality gains of ++Spicy as thenumber of constraints is varied and as the degree of sharingis varied (hence on more complex scenarios). Also we com-pare ++Spicy directly to MapMerge on the same scenarios,a comparison that has not been done before.

Metadata Scenarios. We designed two scenarios usingiBench. The Ontology scenarios consist of primitives thatcan be used to map a relational schema to an ontology.Three of these primitives are different types of vertical par-titioning, into a HAS-A, IS-A, or N-to-M relationship (VH,VI, VNM). The fourth primitive is ADD (copies and addsnew attributes). The STB scenarios consist of primitivessupported by STBenchmark [2]: CP (copy a relation), VP(vertical partitioning), HP (horizontal partitioning), and SU(copy a relation and create a surrogate key). The Ontologyscenarios produce at least twice as many value inventions(referred to hereafter as nulls) as the STB scenarios (e.g.,one instance of each ontology primitive - ADD, VH, VI,and VNM - yields 8 nulls, while one instance of each STBprimitive - CP, VP, HP, and SU - only yields 4 nulls). TheOntology scenarios also include more keys compared to STB.

Measures. Intuitively, mappings that produce smaller tar-get instances are more desirable because they produce lessincompleteness. To assess the quality of the correlated map-pings, we measure the size of the generated target instanceand the relative improvement (of MapMerge or ++Spicyw.r.t. Clio), where the size of an instance is the number ofatomic values that it contains [1]. Assuming that the Cliomappings produce an instance of size x and the MapMergemappings produce an instance of size y, the relative improve-ment is (x − y)/x. The original MapMerge evaluation [1]used this measure and also measured the similarity betweensource and target instances using the notion of full disjunc-tion [23]. We were not able to use full disjunction in our ex-periments because we use arbitrary constraints and sharing,and these features often break the gamma-acyclicity pre-condition required for the implementation of this measure.Instead, we measure the number of nulls in the generatedtarget instance. ++Spicy used a similar measure, however,they measure size in tuples rather than atomic values. Wealso report the performance in terms of time for generatingmappings and time for executing them.

5.1 Scalability of Clio and MapMergeFor the two aforementioned scenarios, we generate an

equal number of instances for each primitive for varyingpowers of two. On the x-axis of Figures 6(a) to 6(b) wereport the total number of primitives, e.g., the value 128 forOntology scenarios corresponds to primitives ADD, VH, VI,VNM each occurring 128/4=32 times. We generated addi-tional INDs for 20% of the relations. We used 25% source

and target sharing for Ontology scenarios. For STB scenar-ios, we also used 25% target sharing, but 50% source sharingto compensate for sharing that naturally occurs (e.g., prim-itives such as HP generate multiple mappings sharing rela-tions, hence sharing occurs even if 0% sharing is requested).We set source relation size parameter (2-6) and create asource instance with 1,000 tuples per relation.

The target instance sizes are shown in Figure 6(a) - asexpected linear in the number of primitives. Clio (C) pro-duces the same number of constants and more nulls thanMapMerge (M). The Ontology scenarios produce more nullsand larger instances than the STB scenarios. Figure 6(b)shows that in general, the relative improvement of Map-Merge over Clio is higher for Ontology compared to STB. Animportant outcome of our experiments is that the benefits ofMapMerge w.r.t. Clio mappings are more visible in scenariosinvolving more incompleteness. We notice that the relativeimprovement remains more or less constant (the exceptionis on small STB scenarios, where Clio and MapMerge arealmost indistinguishable), because the characteristics of thescenarios are the same for different sizes.

We present the mapping generation and execution time(logscale) in Figure 6(c) and 6(d), respectively. Generat-ing and executing mappings for the Ontology scenarios takeslonger than for the STB scenarios due to the amount of nulls.Although MapMerge requires more time than Clio to gen-erate mappings, it requires less time to execute them. Thisbehavior is more visible for larger number of mappings be-cause in our experiments this implies larger target instances(up to 2M atoms, in contrast to the original MapMerge eval-uation that only considered up to 120K atoms).

Our findings extend the original MapMerge evaluation intwo ways: (i) they do not report mapping execution time(they have only the generation time), and (ii) MapMerge hasgreater benefit on the Ontology scenarios, which are definedusing iBench primitives that are not covered by previous sce-nario generators nor by the existing MapMerge evaluation.Indeed, the very flexible value invention provided by iBenchreveals another strong point about MapMerge, not consid-ered before. MapMerge not only generates smaller instancescompared to Clio, but is also more time efficient.

5.2 Impact of Constraints and SharingWe now study the impact of increasing the number of

constraints and amount of sharing for relations with (3-7)attributes and 1,000 tuples. We ran three experiments, andfor each scenario, we use 10 instantiations of each of itsprimitives. First, we vary the number of source and tar-get INDs from 10 to 40%, with no source or target sharing.We present the size of the generated target instance for bothscenarios in Figure 7(a) and the relative improvement in Fig-ure 7(b). Here the relative improvement is not constant, butimproves as the number of constraints increases. Second, wevary target sharing from 0-60% and no random constraints.Source sharing is fixed to 20% for Ontology and to 50% forSTB. We present the size of the generated target instance forboth scenarios in Figure 7(c) and the relative improvementin Figure 7(d). Third, we vary the amount of source sharingfrom 0-60%. Target sharing is fixed to 20% and there areno random constraints. Target instance sizes are shown inFigure 7(e) and the relative improvement in Figure 7(f).

Our experiments reveal that both sharing and constraintsincrease the relative improvement of MapMerge over Clio.

117

100K

200K

300K

400K

500K

0CM CM CM CM CM CM CM CM

Targ

et

Siz

e (

# A

tom

s)

Ontology STB % Constraints

Constants Nulls

4030201040302010

(a) % Constraints

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

10 20 30 40

Rel. Im

pro

vem

en

t

% Constraints

Ont-TotalOnt-NullsSTB-TotalSTB-Nulls

(b) Rel. Improv.: 7(a)

100K

200K

300K

400K

500K

0CM CM CM CM CM CM CM CM

Targ

et

Siz

e (

# A

tom

s)

Ontology STB % Target Sharing

Constants Nulls

60402006040200

(c) Target Sharing

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 20 40 60

Rel. Im

pro

vem

en

t

% Target Sharing

Ont-TotalOnt-NullsSTB-TotalSTB-Nulls

(d) Rel. Improv.: 7(c)

100K

200K

300K

400K

500K

0CM CM CM CM CM CM CM CM

Targ

et

Siz

e (

# A

tom

s)

Ontology STB % Source Sharing

Constants Nulls

60402006040200

(e) Source Sharing

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 20 40 60

Rel. Im

pro

vem

en

t

% Source Sharing

Ont-TotalOnt-NullsSTB-TotalSTB-Nulls

(f) Rel. Improv.: 7(e)

Figure 7: Evaluation of Clio (C) and MapMerge (M)

We observe that for STB, the biggest gain comes from usingconstraints, while for the Ontology scenarios the biggest gaincomes from target sharing. This suggests that the benefitsshown in Figure 6(b) (for scenarios combining constraintsand sharing) are due to both factors. MapMerge is mostuseful for mappings that share relations or contain INDs.Our results highlight the impact of MapMerge in novel sce-narios, going beyond the original evaluation.

5.3 Comparison with ++SpicyDespite producing target instances smaller than Clio, Map-

Merge may not produce core solutions. Here, we include acomparison with ++Spicy [18]. More precisely, we presenttwo experiments: (i) we first compare the core solutionsproduced by ++Spicy with the MapMerge and Clio solu-tions [19], and (ii) we independently study the impact of us-ing egds in the mapping, an important ++Spicy feature [17].For (i), we took 10 instances of each Ontology primitive, andsource relations with (3-7) attributes and 100 tuples. We fo-cus on the Ontology scenarios that have both more keys andmore nulls, making the comparison with MapMerge clearer.We used ++Spicy in its default configuration i.e., with corerewriting but without egd rewriting. In Figure 8, we reportthe generated target instance size and the relative improve-ment w.r.t. Clio for both MapMerge and ++Spicy, for threedifferent settings: we vary the amount of source and targetINDs (8(a) and 8(b)), of target sharing (8(c) and 8(d)), andsource sharing (8(e) and 8(f)), respectively (variations over

10K

20K

30K

40K

50K

0C M S C M S C M S C M S

Targ

et

Siz

e (

# A

tom

s)

% Constraints

Constants Nulls

40302010

(a) % Constraints

0

0.2

0.4

0.6

0.8

1

10 20 30 40Rel. im

pro

vem

en

t (w

.r.t

. C

lio)

% Constraints

M-TotalM-NullsS-TotalS-Nulls

(b) Rel. Improv.: 8(a)

10K

20K

30K

40K

50K

0C M S C M S C M S C M S

Targ

et

Siz

e (

# A

tom

s)

% Target Sharing

Constants Nulls

6040200

(c) Target Sharing

0

0.2

0.4

0.6

0.8

1

0 20 40 60Rel. im

pro

vem

en

t (w

.r.t

. C

lio)

% Target Sharing

M-TotalM-NullsS-TotalS-Nulls

(d) Rel. Improv.: 8(c)

10K

20K

30K

40K

50K

0C M S C M S C M S C M S

Targ

et

Siz

e (

# A

tom

s)

% Source Sharing

Constants Nulls

6040200

(e) Source Sharing

0

0.2

0.4

0.6

0.8

1

0 20 40 60Rel. im

pro

vem

en

t (w

.r.t

. C

lio)

% Source Sharing

M-TotalM-NullsS-TotalS-Nulls

(f) Rel. Improv.: 8(e)

Figure 8: Evaluation of ++Spicy (S)

the same ranges while keeping the other parameters fixedto the same values as in Section 5.2). The size of the in-stance generated by ++Spicy remains constant regardlessof the number of additional constraints (cf. 8(a)). This isin contrast to both Clio and MapMerge where the size in-creases. This indicates that the core solutions of ++Spicyare effectively removing redundancy created by chasing overthese additional INDs. The relative improvement is alwaysgreater for ++Spicy compared to MapMerge (cf. 8(b), 8(d),and 8(f)). These benefits come naturally with a time cost:while generating Clio or MapMerge mappings took up to 8sec, generating the ++Spicy mappings took up to 105 sec.

5.4 Evaluation of ++Spicy Using UDPsIn our second ++Spicy experiment, we quantified the ben-

efits of using egds in mapping creation. Neither Clio norMapMerge uses egds, hence we compared the solution givenby ++Spicy in the presence of egds with its canonical andcore solutions. The advantages of using egds can be seenin many integration scenarios generated by iBench, but tobest highlight the gains, we report on one experiment inwhich we created a specially designed UDP. The UDP con-tains source address information shred across three relationsthat needs to be combined into a single target relation. TheUDP is easy to create (the UDP code took only a few min-utes to write and is 79 lines of XML that we include in ourTR). The source has three relations Person(name, address),Address(occupant, city) and Place(occupant, zip), and two

118

0

10K

20K

30K

40K

50K

60K

70KTa

rget

Siz

e (

# A

tom

s)

% Target Relations with Keys

Constants Nulls

CaCoE100

CaCoE75

CaCoE50

CaCoE25

CaCoE0

(a) Target Instance Size

0

0.2

0.4

0.6

0.8

1

0 25 50 75 100

Rel. Im

pro

vem

ent

% Target Relations with Keys

Co/Ca-TotalCo/Ca-Nulls

E/Ca-TotalE/Ca-NullsE/Co-TotalE/Co-Nulls

(b) Rel. Improv.: 9(a)

Figure 9: ++Spicy: Canonical Solution (Ca), CoreSolution (Co), Solution in the Presence of EGDs (E)

FKs (from the two occupant attributes to Person.name).In the target, we have one relation LivesAt(name, city,zip). There is a correspondence towards each target at-tribute from the source attribute having the same name. Inthe experiment, we invoked the UDP 40 times with sourcerelations containing 100 tuples, and we varied the number oftarget relations having a primary key from 0 to 100%. Wereport the generated target size and the relative improve-ment in Figure 9(a) and 9(b), respectively. The canonicaland core solutions have constant sizes. As the number ofkeys increases, ++Spicy uses them on more and more re-lations to remove incomplete redundant tuples as expected.Notice that iBench makes the generation of such an experi-ment very easy. Once the UDP is created, iBench can applyit on data instances of different sizes and characteristics (for++Spicy this would include reuse of values) and can invokeit an arbitrary number of times with other primitives or withadditional (arbitrary) mappings. Despite the fact that thecomplexity of the mapping generation problem is hard inthe presence of UDPs, iBench makes it easy to create andapply them flexibly to quickly develop robust evaluations.

6. CONCLUSIONWe presented the first version of iBench, an open-source

metadata generator [5]. We used the system to conducta new comparative evaluation of MapMerge and ++Spicy,systems that had been evaluated individually but not com-pared directly. We also presented a new evaluation of ++Spicyto quantify the influence of target key constraints on thequality of their solution, an evaluation not done by theauthors for lack of appropriate metadata generation tools.Moreover, iBench was an essential tool in a large scale empir-ical evaluation we have conducted in previous work [7]. Ourhope is that iBench will be a catalyst for encouraging moreempirical work in data integration and a tool for researchersto use in developing, testing and sharing new quality mea-sures. In the future, we will extend the prototype with newfunctionality (with community help).

Acknowledgements. We thank the creators of STBench-mark (Alexe, Tan, Velegrakis) for sharing their code withus; the creators of MapMerge (Alexe, Hernandez, Popa andTan) for sharing MapMerge; Mecca and Papotti for helpingus to use ++Spicy; and all for verifying our experimentalconclusions.

7. REFERENCES[1] B. Alexe, M. A. Hernandez, L. Popa, and W.-C. Tan.

MapMerge: Correlating Independent Schema Mappings.VLDB J., 21(2):191–211, 2012.

[2] B. Alexe, W.-C. Tan, and Y. Velegrakis. STBenchmark:Towards a Benchmark for Mapping Systems. PVLDB,1(1):230–244, 2008.

[3] B. Alexe, B. ten Cate, P. G. Kolaitis, and W.-C. Tan.Designing and Refining Schema Mappings via DataExamples. In SIGMOD, pages 133–144, 2011.

[4] M. Arenas, J. Perez, J. L. Reutter, and C. Riveros. TheLanguage of Plain SO-tgds: Composition, Inversion andStructural Properties. J. Comput. Syst. Sci.,79(6):763–784, 2013.

[5] P. C. Arocena, R. Ciucanu, B. Glavic, and R. J. Miller.Gain Control over your Integration Evaluations. PVLDB,8(12):1960–1963, 2015.

[6] P. C. Arocena, B. Glavic, R. Ciucanu, and R. J. Miller.The iBench Integration Metadata Generator: ExtendedVersion. Technical report, U. of Toronto, 2015.https://bitbucket.org/ibencher/ibench.

[7] P. C. Arocena, B. Glavic, and R. J. Miller. Value Inventionin Data Exchange. In SIGMOD, pages 157–168, 2013.

[8] D. Barbosa, A. Mendelzon, J. Keenleyside, and K. Lyons.ToXgene: An Extensible Template-based Data Generatorfor XML. In WebDB, pages 49–54, 2002.

[9] P. A. Bernstein, T. J. Green, S. Melnik, and A. Nash.Implementing Mapping Composition. VLDB J.,17(2):333–353, 2008.

[10] I. Boneva, A. Bonifati, and R. Ciucanu. Graph DataExchange with Target Constraints. In EDBT/ICDTWorkshops–GraphQ, pages 171–176, 2015.

[11] R. Fagin. Inverting Schema Mappings. TODS, 32(4):24,2007.

[12] R. Fagin, L. M. Haas, M. A. Hernandez, R. J. Miller,L. Popa, and Y. Velegrakis. Clio: Schema MappingCreation and Data Exchange. In Conceptual Modeling:Foundations and Applications, pages 198–236, 2009.

[13] R. Fagin, P. G. Kolaitis, R. J. Miller, and L. Popa. DataExchange: Semantics and Query Answering. Theor.Comput. Sci., 336(1):89–124, 2005.

[14] R. Fagin, P. G. Kolaitis, L. Popa, and W.-C. Tan.Composing Schema Mappings: Second-Order Dependenciesto the Rescue. TODS, 30(4):994–1055, 2005.

[15] J. Hammer, M. Stonebraker, and O. Topsakal. THALIA:Test Harness for the Assessment of Legacy InformationIntegration Approaches. In ICDE, pages 485–486, 2005.

[16] M. Lenzerini. Data Integration: a Theoretical Perspective.In PODS, pages 233–246, 2002.

[17] B. Marnette, G. Mecca, and P. Papotti. Scalable DataExchange with Functional Dependencies. PVLDB,3(1):105–116, 2010.

[18] B. Marnette, G. Mecca, P. Papotti, S. Raunich, andD. Santoro. ++Spicy: an OpenSource Tool forSecond-Generation Schema Mapping and Data Exchange.PVLDB, 4(12):1438–1441, 2011.

[19] G. Mecca, P. Papotti, and S. Raunich. Core SchemaMappings. In SIGMOD, pages 655–668, 2009.

[20] R. J. Miller, D. Fisla, M. Huang, D. Kymlicka, F. Ku, andV. Lee. The Amalgam Schema and Data Integration TestSuite. www.cs.toronto.edu/~miller/amalgam, 2001.

[21] D. Patterson. Technical Perspective: For Better or Worse,Benchmarks Shape a Field. CACM, 5(7), 2012.

[22] L. Popa, Y. Velegrakis, R. J. Miller, M. A. Hernandez, andR. Fagin. Translating Web Data. In VLDB, pages 598–609,2002.

[23] A. Rajaraman and J. D. Ullman. Integrating Informationby Outerjoins and Full Disjunctions. In PODS, pages238–248, 1996.

[24] B. ten Cate and P. G. Kolaitis. StructuralCharacterizations of Schema-Mapping Languages.Commun. ACM, 53(1):101–110, 2010.

[25] C. Yu and L. Popa. Semantic Adaptation of SchemaMappings when Schemas Evolve. VLDB, pages 1006–1017,2005.

119