DataModel Evolution

Sander VermolenEelco Visser

This research is supported by NWO/JACQUARD project638.001.610, MoDSE: Model­Driven Software Evolution.

   

Data Models

   

   

   

   

User   1name bob real name Bob Johnsonemail  b.johnson@mail.com

Page   1title "The first page"isRedirect falsetext  "Hello world"

   

Count page views

Version history

   

   

User   1name bob real name Bob Johnsonemail  b.johnson@mail.com

Page   1title "The first page"isRedirect falsetext  "Hello world"

No page countNo revisions

   

   

Coupled Data Evolution

   

...

$dbh->bz_add_column('attachments', 'submitter_id', {TYPE => 'INT3', NOTNULL => 1}, 0);$dbh->bz_rename_column('bugs_activity', 'when', 'bug_when');_add_bug_vote_cache(); _update_product_name_definition(); _add_bug_keyword_cache();$dbh->bz_add_column('profiles', 'disabledtext', {TYPE => 'MEDIUMTEXT', NOTNULL => 1}, '');_populate_longdescs(); _update_bugs_activity_field_to_fieldid();

if (!$dbh->bz_column_info('bugs', 'lastdiffed')) {$dbh->bz_add_column('bugs', 'lastdiffed', {TYPE =>'DATETIME'});$dbh->do('UPDATE bugs SET lastdiffed = NOW()');

}

_add_unique_login_name_index_to_profiles(); $dbh->bz_add_column('profiles', 'mybugslink', {TYPE => 'BOOLEAN', NOTNULL => 1, DEFAULT => 'TRUE'});_update_component_user_fields_to_ids();$dbh->bz_add_column('bugs', 'everconfirmed', {TYPE => 'BOOLEAN', NOTNULL => 1}, 1);$dbh->bz_add_column('products', 'maxvotesperbug', {TYPE => 'INT2', NOTNULL => 1, DEFAULT => '10000'});$dbh->bz_add_column('products', 'votestoconfirm', {TYPE => 'INT2', NOTNULL => 1}, 0);_populate_milestones_table();$dbh->bz_alter_column('bugs', 'target_milestone', {TYPE => 'varchar(20)', NOTNULL => 1, DEFAULT => "'---'"});$dbh->bz_alter_column('milestones', 'value', {TYPE => 'varchar(20)', NOTNULL => 1});_add_products_defaultmilestone();

if (!$dbh->bz_index_info('cc', 'cc_bug_id_idx') || !$dbh->bz_index_info('cc', 'cc_bug_id_idx')->{TYPE}) {$dbh->bz_drop_index('cc', 'cc_bug_id_idx');$dbh->bz_add_index('cc', 'cc_bug_id_idx', {TYPE => 'UNIQUE', FIELDS => [qw(bug_id who)]});

}if (!$dbh->bz_index_info('keywords', 'keywords_bug_id_idx') || !$dbh->bz_index_info('keywords', 'keywords_bug_id_idx')->{TYPE}) {

$dbh->bz_drop_index('keywords', 'keywords_bug_id_idx');$dbh->bz_add_index('keywords', 'keywords_bug_id_idx', {TYPE => 'UNIQUE', FIELDS => [qw(bug_id

keywordid)]});}

...

   

Costly

High risk

Holds back the development process

Large infrequent development steps

   

Userid  : integername : varcharrealName : varcharemail : tinytext

Pageid : integertitle : varcharauthor - User *isRedirect : booleancontent : text

set of

   

Userid  : integer Uniquename : varcharrealName : varchar ?email : tinytext

Page : Mediumauthor - User min(1) max(8)content : textrefs : url * Indexed

abstract Mediumid : integer Uniquetitle : ANY_NAME

   

EvolvingData Models

   

   

   

   

   

SpecifyingData Model Evolution

   

Userid  :: integername :: varcharrealName :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor UserisRedirect :: booleancontent :: text

Userid  :: integername :: varcharrealName :: varcharemail :: tinytext

Pageid :: integertitle :: varcharcounter :: bigintegerisRedirect :: booleanrevisions set of Revision

Revisionid :: integerpage Pagecomment :: tinyblobtimestamp :: timerevisionText :: textauthor User

   

Added type revisions

Added attribute revisions

Moved content to revision text

Added attribute counter

Revisionid :: integerpage Pagecomment :: tinyblobtimestamp :: timeauthor User

Pagerevisions set of Revision

RevisionrevisionText :: text

Pagecounter :: biginteger

What happened?

   

add or remove entity

change name of entity

change type of set

8 Basic Transformations

add or remove property

change name of property

change type of property

   

move property

1 Advanced Transformation

   

addRevision

id : integerpage - Pagecomment : tinyblobtimestamp : timeauthor - User

addcounter : biginteger

   

   

At / Entity Page  /  Property Titleadd counter : biginteger

   

At / Entity[Id=''Page''] / Property[Id=''Title'']add counter : biginteger

   

at // Property [Id = ../Id]add counter :: biginteger

   

RevisionrevisionText :: text

At / Entity Revision  /  Property timeStampmove page.content to revisionText :: text

   

at Entity Page  /  Property Titleadd counter :: biginteger

at Entity Revision  /  Property timeStampadd revisionText :: textfrom page.content

;

   

EvolvingData Models

8 basic transformations

1 advanced transformation

Language to specify transformations

Positioning sub language

Specify data model evolutions

   

DataMigration

   

   

Technical Domain

WebDSL

Stratego/XT

Stratego/XTGeneric Aterm

SQL Databases (MySQL)

   

Because 

we really like program transformations

generally richer than regular data acessing languages (SQL)

data migration = model transformation

Program transformation for data migration

   

Userid  :: integername :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor User

User(id(1),name(“John”),email(“johnnyboy@mail.com”)

)

Page(id(2),title(“Hello World”),[author(1)]

)

   

User(id(1),name(“John”),email(“jb@m.com”)

)

Page(id(2),title(“Hello World”),[author(1)]

)

<user><id>1</id><name>John</name><email>jb@m.com</email>

</user>

<page><id>2</id><title>Hello World</title><authors>

<author>1</author></authors>

</page>

   

Userid  :: integername :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor User

User(id(1),name(“John”),.....

)

Page(id(2),title(“Hello World”),[author(1)]

)

Remove Attribute (1)

Signature:User := Id * Name * Email

   

Userid  :: integername :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor User

User(id(1),name(“John”),email(“johnnyboy@mail.com”)

)

Page(id(2),title(“Hello World”),[author(1)]

)

Remove Attribute (2)

   

Userid  :: integername :: varcharemail :: tinytext

Pageid :: integertitle :: varcharauthor User

User( 0,[

id(1),name(“John”),email(“johnnyboy@mail.com”)

])

Page( 1,[

id(2),title(“Hello World”),author(0)

])

Generic Aterm (GTerm)

   

User( 0,[

id(1),name(“John”),email(“jb@m.com”)

])

Page( 1,[

id(2),title(“Hello World”),author(0)

])

XMI

<user id='0'><id>1</id><name>John</name><email>jb@m.com</email>

</user>

<page id='1'><id>2</id><title>Hello World</title><author>0</author>

</page>

   

GTerm Characteristics

Explicit references

No nesting

Implicit sets/lists

Storage...

   

GTerm Transformation

Gterm libraryObject creationModifying attributes 

(add, remove, change, rename, ...)Object equivalenceObject traversals(Object graph traversals)

Data model libraryType examinationSuper/Sub type handlingAbstract type handling

   

GTerm Storage

Large quantities of data...

Storage engine:In memory ­ list based ~10KIn memory ­ hash table based ~500KIn database ~25M ­ ...

   

User( 0,[

id(1),name(“John”),email(“jb@m.com”)

])

Page( 1,[

id(2),title(“Hello World”),author(0)

])

GTerm Storage – In database (1)

0 User id 10 User name John0 User email jb@m.com1 Page id 21 Page title Hello World1 Page author 0

0 User1 Page

   

GTerm Storage – In database

CREATE TABLE  Attributes (id varchar(16),type varchar(30),name varchar(30),value text,INDEX USING HASH (id (5)),INDEX USING BTREE (v(10))

)

CREATE TABLE  Objects (id varchar(16),type varchar(30)

)

GTerm Storage – In database (2)

   

GTerm Performance

Database indexes

Stratego memory usage

Parallel execution

   

GTerm Storage – Regular database

0 User id 10 User name John0 User email jb@m.com1 Page id 21 Page title Hello World1 Page author 0

User:1 John jb@m.com

Page:2 Hello World

Page­User:2 1

   

Old Database Generic Database

SQL Script

Data model

Migration (Stratego)

SQL Script

GTerm 2 SQL

New Database

   

ResearchrDBLP

   

<dblp><incollection mdate="2002­01­03" key="books/acm/kim95/AnnevelinkACFHK95">

<author>Jurgen Annevelink</author><author>Rafiul Ahad</author><author>Amelia Carlson</author><author>Daniel H. Fishman</author><author>Michael L. Heytens</author><author>William Kent</author><title>Object SQL ­ A Language for the Design and 

Implementation of Object Databases.</title><pages>42­68</pages><year>1995</year><booktitle>Modern Database Systems</booktitle><url>db/books/collections/kim95.html#AnnevelinkACFHK95</url><crossref>books/crc/KIM95</crossref>

</incollection>

....</dblp>

   

article {key : stringtitle : string *author : string *editor : string *booktitle : string *pages : string *address : string *journal : string *volume : string *number : string *publisher : string *crossref : string *series : string *school : string *chapter : string *month : string *year : string *url : string *note : string *mdate : string *cite : string *ee : string *cdrom : string *isbn : string *

}

book {key : stringtitle : string *author : string *editor : string *booktitle : string *pages : string *address : string *journal : string *volume : string *number : string *publisher : string *crossref : string *series : string *school : string *chapter : string *month : string *year : string *url : string *note : string *mdate : string *cite : string *ee : string *cdrom : string *isbn : string *

}

....

   

~ 800,000  Authors

~ 1,200,000  Publications

~ 14,000,000  Lines of XML

~ 16,000,000 Database records

DBLP Data

   

Publication {key : string Uniquetitle : stringauthors - Author + Indmonth : stringyear : stringdblpUrl : string ?doi : stringlinks - Link *abstract : stringnote : stringannote : stringmodified : datemodifiers - User *cites - Publication *ee : string ?isbn : stringissn : stringconflicts : bool

}

abstract PrintPublication - Publication {

pages : stringpublisher : string ?firstpage : intlastpage : int ?

}

Article - PrintPublication {journalname : stringvolumenumber: stringissuenumber : string

}

Alias {name : string Unique

}

AbstractAuthor {alias - Alias

}

   

Load objects into database

9 Stages of migration

Generate SQL

Migration Approach

   

BridgingMeta levels

   

   

article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

renameType(|oldType, newName)

?Transformation(path, Substitution(newName), _)

UPDATE At SET  t=newName WHERE t = ...;

   

Proceedings - Collection {booktitle : string ?

}

Proceedings - Collection {conference : string ?

}

renameAtt(|oldName, newName, type, dmodel)

?Transformation(path, Substitution(newName), _)

UPDATE At SET  n = newName WHERE n = oldNameAND t = <getSubTypeQuery(|dmodel)> type;

   

abstract Thesis - Publication {school : string

}

abstract Thesis - Publication {school : stringtype : string ?

}

...

?Transformation(path, Addition(Att(Name(attName), PrimType(_), annotations)),_);<getAttAnn(|"MinCard")> annotations; ?0

...

   

abstract Thesis - Publication {school : string

}

abstract Thesis - Publication {school : stringtype : string

}

<addDefaultAttributesToType(|type, <make­int> low, attName, attType, mmodel

)> model;

?Transformation(path, Addition(Att(Name(attName), PrimType(_), annotations)),_

);<getAttAnn(|"MinCard")> annotations; ?low

onType(addDefaultAttributes(|type, nr, attName, attType)

| type, mmodel)

   

1. find objects of type

2. divide into chunks

3. per chunk in parallel

Load objects in chunkper object

ssave object if changed

onType

   

Publication {key : string Uniquetitle : string ?authors : string + Indexedyear : string...

}

?Transformation(path, Substitution(DeclType(Name(newTypeName))), _)...Author alias mandatoryAuthor alias not unique

Publication {key : string Uniquetitle : string ?authors - Author + Indexedyear : string...

}

Author {alias : string

}

onType(for each author

create author objectset author attribute

)

   

Author {alias : string

}

?Transformation(path, Substitution(DeclType(Name(newTypeName))), _)...Alias name mandatoryAlias name unique

Author {alias : Alias

}

Alias {name : string Unique

onType(if alias exists then

set alias attribute to existing idelse

create alias objectset alias attribute to new id

)

   

IdentityPrimitive attribute addition (3)Complex attribute additionAttribute removalAttribute name changeAttribute move (2)Primitive type changeImplicit reference resolutionAttribute wrappingType additionType removalType name changeAbstract type handlingInverse annotation handling (2)Cardinality changes (2)

Supported transformations

   

Easy to define Hard to define

Easy to optimize No need to optimize

Expressive Limited expressiveness

Easy to abstract Abstraction near impossible

Performance OK Performance great

In memory vs. Database transformations

   

DetectingEvolution

   

   

article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

diff?

   

   

article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Entity(Name(“article”),Name(“PrintPublication”),[

Att(Name(“reviewid”),PrimType(

Name(“string”)),[

MinCard(0),MaxCard(1)

]),...

])

Entity(Name(“Article”),Name(“PrintPublication”),[

Att(Name(“reviewid”),PrimType(

Name(“string”)),[

MinCard(0),MaxCard(1)

]),...

])

   

article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Entity(Name(“article”),Name(“PrintPublication”),[

Att(Name(“reviewid”),PrimType(

Name(“string”)),[

MinCard(0),MaxCard(1)

]),...

])

Entity(Name(“Article”),Name(“PrintPublication”),[

Att(Name(“reviewid”),PrimType(

Name(“string”)),[

MinCard(0),MaxCard(1)

]),...

])

diff?

   

article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string ?

}

Removed type article; Added type Article

Added type article; Removed type Article

Substituted article name with Article

What happened?

   

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string

}

Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?

}

renamed rating;  renamed number;changed rating cardinality; changed number cardinality

What happened?

   

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string

}

Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?

}

renamed rating;  renamed number;changed rating cardinality; changed number cardinality

swapped rating and number; renamed rating;  renamed number

What happened?

   

Article - PrintPublication {reviewid : string ?rating : string ?journal : string ?volume : string ?number : string

}

Article - PrintPublication {reviewid : string ?rate : stringjournal : string ?volume : string ?nr : string ?

}

renamed rating;  renamed number;changed rating cardinality; changed number cardinality

swapped rating and number; renamed rating;  renamed number

deleted Article;  added Article

What happened?

   

Addition: 0.8 * relativeSize4

Removal: 0.5 * relativeSize4

Substitution 0.4 * relativeSize6

Weighing transformations

Custom weights:Type removalType substitution  (0)Attribute substitution (0)

   

On both versions at the same time

Bound on weightIncreasing bound

Weight computation caching

Try them all!

   

   

HeterogeneousCoupled Evolution

   

   

   

   

Horizontal Generalization

   

Username :: varcharrealName :: varcharemail :: tinytext

Pagetitle :: varcharauthor UserisRedirect :: boolean

Entities

Properties

Types

Meta model / Grammar

   

   

Lists

More types

Inverse associations

Abstract types

   

Vertical Generalization

   

Heterogeneous Coupled Evolution 

ofSoftware Languages

   

The ingredients

Software Language Definition Formalism

EvolvingSoftware Language 

Software 

   

DiverseEvolution

What did we generalize?

Why did we generalize?

   

A GenericArchitecture

   

   

   

   

Input

Coupled evolution scenario

Mapping from Mi to Mi+1

   

Output

Domain Specific Transformation Language 

(DSTL)

Transformation Interpreter

Software Migration

   

Stratego

SDF SDF

   

Entity* ­> DataM  ModelId "{" Prop* "}" ­> Entity EntityId "::" Type ­> Prop  Prop"int" ­> Type   Int"bool" ­> Type   BoolId ­> Type  "set of" Type ­> Type   SetNAME ­> Id     Id

   

Entity* ­> DataM  Model

Lists

"add" Entity    ­> LocalTransformation"remove"        ­> LocalTransformation

[...]

   

NAME ­> Id     Id

Lexicals

"substitute" NAME ­> LocalTransformation

''...''

   

"int" ­> Type   Int"bool" ­> Type   BoolId ­> Type  "set of" Type ­> Type   Set

Multiple productions

"substitute" Type ­> LocalTransformation

­>*

   

"at" APath LocalTransformation ­> Transformation

Type checking

Generation of local transformation domainsAPath type derivation

Type checking

.../...

   

Transform

Constant

Larger grammars

   

   

Interpreter generation

Transformations library

Generic DSTL constructs

APath evaluation

   

A GenericArchitecture

   

Software Language Evolution

This research was supported by NWO/JACQUARD project638.001.610, MoDSE: Model­Driven Software Evolution.