Architecture Delphi SQL NoSQL ORM ODM Performance
54
April 2014 From SQL to ORM Arnaud Bouchez
-
Upload
arnaud-bouchez -
Category
Software
-
view
686 -
download
8
description
Introduces ORM code for generating SQL, and ODM approach for NoSQL / MongoDB in the context of modern Delphi applications and the Open Source Synopse mORMot framework.
Transcript of Architecture Delphi SQL NoSQL ORM ODM Performance
April 2014
From SQL to ORMArnaud Bouchez
April 2014
From SQL to ORM
SQL NoSQL ORM practice
From SQL to ORM
April 2014
SQL
De-Facto standard for data manipulation Schema-based Relational-based ACID by transactions Time proven and efficient Almost standard
From SQL to ORM
April 2014
Not Only SQL Two main families of NoSQL databases:
Aggregate-oriented databases Graph-oriented databases
Martin Fowler
From SQL to ORM
April 2014
Not Only SQL
Graph Database Store data by their relations / associations
From SQL to ORM
April 2014
Not Only SQL
Aggregate is a collection of data
that we interact withas a unit
forms the boundaries for ACID operationsin a given model
(Domain-Driven Design modeling)
From SQL to ORM
April 2014
Aggregate Database families
Document-based e.g. MongoDB, CouchDB
Key/Value e.g. Redis
Column family e.g. Cassandra
From SQL to ORM
Not Only SQL
April 2014
Not Only SQL
Are designed to scale for the web Examples: Google farms, Amazon Should not be used as scaling RDBMS
Can be schema-less For document-based and key/value
or column-driven
Can be BLOB storage
From SQL to ORM
April 2014
Not Only SQL
RBMS stores data per table JOIN the references to get the aggregate
From SQL to ORM
April 2014
Not Only SQL
NoSQL stores aggregate as documents Whole data is embedded
From SQL to ORM
April 2014
Not Only SQL
Data modeling
From SQL to ORM
April 2014
Not Only SQL
SQL > NoSQL Ubiquitous SQL Easy vertical scaling Data size (avoid duplicates and with no
schema) Data is stored once, therefore consistent Complex ACID statements Aggregation functions (depends)
From SQL to ORM
April 2014
Not Only SQL
From SQL to ORM
April 2014
Not Only SQL
NoSQL > SQL Uncoupled data: horizontal scaling Schema-less: cleaner evolution Straight-To-ODM Version management (e.g. CouchDB) Graph storage (e.g. Redis)
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
ORM gives a set of methods (CRUD) to ease high-level objects persistence into an RDBMS via mapping
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
Since decades Classes are the root of our OOP model RDBMS is (the) proven way of storage
Some kind of "glue" is needed to let class properties be saved into one or several tables
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
In fact Classes are accessed via Delphi/Java/C#
code RDBMS is accessed via SQL
SQL by itself is a full programming
language With diverse flavors (e.g. data types) Difficult to switch the logic Error prone, and difficult to maintain
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
Sometimes, there will be nothing better
than a tuned SQL statement
But most of the time, You need just to perform
some basic CRUD operations(Create Retrieve Update Delete)
From SQL to ORM
object
instance
SQL
RDBMS
objectinstance
ORM
CRUDoperations
SQL
mapping
RDBMS
April 2014
Object Relational Mapping (ORM)
Mapping may be created:
Code-first Objects are defined,
then persisted (dehydrated) Usually the best for a new project
Model-first From an (existing) database model Sometimes difficult to work with legacy
structures
From SQL to ORM
ORM
class type data model
object instance RDBMS
April 2014
Object Relational Mapping (ORM)
Mapping may be defined:
by configuration via code (attributes or fluent interface) via external files (XML/JSON)
by convention from object layout from database layout
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
Objects may be:
Any business class (PODO) Eases integration with existing code But may pollute the class (attributes)
Inherit from a common class Get a shared behavior Tend to enforce DDD’s persistence agnosticity
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
ORM advantages Compile time naming and types check
(strong types and names) One language to rule all your logic
(no mix with SQL nor LINQ syntax) Database abstraction
(the ORM knows all dialects, and can switch) Able to cache the statements and the
values You still can write hand-tuned SQL if needed
From SQL to ORM
April 2014
Object Relational Mapping (ORM)
Don't think about… tables with data types (varchar/number...),
but objects with high level types Master/Detail,
but logical units (your aggregates) writing SQL,
but writing your business code "How will I store it?",
but "Which data do I need?".
From SQL to ORM
April 2014
mORMot’s RESTful ORM
From SQL to ORM
April 2014
mORMot’s RESTful ORM
mORMot’s client-server RESTful ORM code-first or model-first ORM convention (TSQLRecord) over configuration which can be consumed via REST over HTTP, via JSON transmission or locally, on the server side, e.g. for SOA
From SQL to ORM
April 2014
mORMot’s RESTful ORM
From SQL to ORM
UI Components
DataBase
RAD
UI
Delphi classes
code mapping
DataBase
HandwrittenSQL
UI
ORM
MVCbinding
Database
GeneratedSQL
UI 1
Client 1
MVC/MVVMbinding
Server
Secureprotocol(REST)
UI 2 (web)
Client 2
MVCbinding
ORM
Persistencelayer
Database
GeneratedSQL
April 2014
mORMot’s RESTful ORM
Defining the object
From SQL to ORM
/// some enumeration // - will be written as 'Female' or 'Male' in our UI Grid // - will be stored as its ordinal value, i.e. 0 for sFemale, 1 for sMale // - as you can see, ladies come first, here TSex = (sFemale, sMale);
/// table used for the Babies queries TSQLBaby = class(TSQLRecord) private fName: RawUTF8; fAddress: RawUTF8; fBirthDate: TDateTime; fSex: TSex; published property Name: RawUTF8 read fName write fName; property Address: RawUTF8 read fAddress write fAddress; property BirthDate: TDateTime read fBirthDate write fBirthDate; property Sex: TSex read fSex write fSex; end;
April 2014
mORMot’s RESTful ORM Defining the Model and the Server:
Model := TSQLModel.Create([TSQLBaby],'rootURI');ServerDB := TSQLRestServerDB.Create(Model,'data.db'),true);ServerDB.CreateMissingTables;Server := TSQLHttpServer.Create('8080',ServerDB);
Defining the Model and the Client:
Model := TSQLModel.Create([TSQLBaby],'rootURI');Client := TSQLHttpClient.Create(Server,'8080', Model);
Both will now communicate e.g. overhttp://server:8080/rootURI/Baby
From SQL to ORM
April 2014
mORMot’s RESTful ORM
From SQL to ORM
var Baby: TSQLBaby; ID: integer; begin // create a new record, since Smith, Jr was just born Baby := TSQLBaby.Create; try Baby.Name := 'Smith'; Baby.Address := 'New York City'; Baby.BirthDate := Now; Baby.Sex := sMale; ID := Client.Add(Baby); finally Baby.Free; end; // update record data Baby := TSQLBaby.Create(Client,ID); try assert(Baby.Name='Smith'); Baby.Name := 'Smeeth'; Client.Update(Baby); finally Baby.Free; end; // retrieve record data Baby := TSQLBaby.Create; try Client.Retrieve(ID,Baby); // we may have written: Baby := TSQLBaby.Create(Client,ID); assert(Baby.Name='Smeeth'); finally Baby.Free; end; // delete the created record Client.Delete(TSQLBaby,ID); end;
April 2014
mORMot’s RESTful ORM
Returning several objects
Results are transmitted as a JSON array Results can be cached at client or server side Only one object instance is allocated and filled Full WHERE clause of the SELECT is at hand See also function TSQLRest.RetrieveList(): TObjectList
From SQL to ORM
aMale := TSQLBaby.CreateAndFillPrepare(Client, 'Name LIKE ? AND Sex = ?',['A%',ord(sMale)]); try while aMale.FillOne do DoSomethingWith(aMale); finally aMale.Free; end;
April 2014
mORMot’s RESTful ORM
JSON transmission layout Expanded / standard (AJAX)
JSON array of JSON objects [{"ID":1},{"ID":2},{"ID":3},{"ID":4},{"ID":5},{"ID":6},{"ID":7}]
Non expanded / faster (Delphi)JSON object of JSON array of values, first row being idents
{"fieldCount":1,"values":["ID",1,2,3,4,5,6,7]}
In mORMot, all JSON content is parsed and processed in-place,then directly mapped to UTF-8 structures.
From SQL to ORM
April 2014
mORMot’s RESTful ORM
« One to one » / « One to many » cardinality
From SQL to ORM
TSQLMyFileInfo = class(TSQLRecord) private FMyFileDate: TDateTime; FMyFileSize: Int64; published property MyFileDate: TDateTime read FMyFileDate write FMyFileDate; property MyFileSize: Int64 read FMyFileSize write FMyFileSize; end;
TSQLMyFile = class(TSQLRecord) private FSecondOne: TSQLMyFileInfo; FFirstOne: TSQLMyFileInfo; FMyFileName: RawUTF8; published property MyFileName: RawUTF8 read FMyFileName write FMyFileName; property FirstOne: TSQLMyFileInfo read FFirstOne write FFirstOne; property SecondOne: TSQLMyFileInfo read FSecondOne write FSecondOne; end;
April 2014
mORMot’s RESTful ORM
Automatic JOINed query At constructor level With nested instances memory management
See also CreateAndFillPrepareJoined()
From SQL to ORM
var MyFile: TSQLMyFile; begin MyFile := TSQLMyFile.CreateJoined(Client,aMyFileID); try // here MyFile.FirstOne and MyFile.SecondOne are true instances // and have already retrieved from the database by the constructor // so you can safely access MyFile.FirstOne.MyFileDate or MyFile.SecondOne.MyFileSize here! finally MyFile.Free; // will release also MyFile.FirstOne and MyFile.SecondOne end; end;
April 2014
mORMot’s RESTful ORM
Data Sharding pattern Store the whole aggregate as once Without JOIN
From ORM to ODM Object Relational Mapping (ORM) Object Document Mapping (ODM)
From SQL to ORM
April 2014
mORMot’s RESTful ODM
NoSQL storage via TDocVariant custom type
Document is indexed: by TSQLRecord.ID: integer by Name: RawUTF8
From SQL to ORM
TSQLRecordData = class(TSQLRecord) private fName: RawUTF8; fData: variant; public published property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE; property Data: variant read fData write fData; end;
April 2014
mORMot’s RESTful ODM
NoSQL storage via TDocVariant custom type
Data: variant will store any document As JSON in the RDBMS Accessed via late-binding in Delphi code
From SQL to ORM
TSQLRecordData = class(TSQLRecord) private fName: RawUTF8; fData: variant; public published property Name: RawUTF8 read fTest write fTest stored AS_UNIQUE; property Data: variant read fData write fData; end;
April 2014
mORMot’s RESTful ODM
Data: variant will store a TDocVariant
Schema-less data{ name : "Joe", x : 3.3, y : [1,2,3] }
{ name : "Kate", x : "abc" }
{ q : 456 }
Real-world evolving data{ name : "Joe", age : 30, interests : "football" }
{ name : "Kate", age : 25 }
From SQL to ORM
April 2014
mORMot’s RESTful ODM
From SQL to ORM
var aRec: TSQLRecordData; aID: integer; begin // initialization of one record aRec := TSQLRecordData.Create; aRec.Name := 'Joe'; // one unique key aRec.data := _JSONFast('{name:"Joe",age:30}'); // create a TDocVariant // or we can use this overloaded constructor for simple fields aRec := TSQLRecordData.Create(['Joe',_ObjFast(['name','Joe','age',30])]); // now we can play with the data, e.g. via late-binding: writeln(aRec.Name); // will write 'Joe' writeln(aRec.Data); // will write '{"name":"Joe","age":30}' (auto-converted to JSON string) aRec.Data.age := aRec.Data.age+1; // one year older aRec.Data.interests := 'football'; // add a property to the schema aID := aClient.Add(aRec); // will store {"name":"Joe","age":31,"interests":"footbal"} aRec.Free; // now we can retrieve the data either via the aID created integer, or via Name='Joe' end;
April 2014
mORMot’s RESTful ORM
SynDB optimized data access layer
From SQL to ORM
SynDB
ZDBC ODBC OleDB Oracle SQLite3 DB.pasTDataset
NexusDB BDE DBExpress FireDACAnyDAC UniDAC
April 2014
mORMot’s RESTful ORM SynDB classes
By-pass the DB.pas unit avoid TDataSet bottleneck, work with Starter
Edition… but can still use it (e.g. FireDAC)
Simple KISS designed APIless data types, interface-based
Statement cache Array binding
bulk insert / batch mode Direct JSON generation
From SQL to ORM
April 2014
mORMot’s RESTful ORM
SynDB integration with ORM
From SQL to ORM
TSQLRestServerDB.Add
TSQLRestServerDB.EngineAdd
internaltable
TSQLRestServerStaticExternal.EngineAdd
externaltableREST
TSQLRequest
INSERT INTO...
SQlite3 engine
internal engine
SQLite3 file
ISQLDBStatement
INSERT INTO...
External DB client
ODBC/ZDBC/OleDB...
External DB server
April 2014
mORMot’s RESTful ORM
SQLite3 Virtual Tables
From SQL to ORM
April 2014
mORMot’s RESTful ORM
SQLite3 Virtual Tables Mapping of external field names JOIN several external databases Very slight performance penalty Access TObjectList instances Access NoSQL databases By-passed if possible Full SQL-92 engine at hand
From SQL to ORM
April 2014
mORMot’s RESTful ORM
SQLite3 Virtual Tables
From SQL to ORM
mORMot
ORM
SQLite3
direct
TObjectList
direct
External DB
direct
virtual
Oracle SQLite3 ODBC OleDB ZDBC
direct
FireDAC AnyDAC UniDAC BDE DBExpress NexusDB
DB.pasTDataSet
April 2014
mORMot’s RESTful ORM
From SQL to ORM
April 2014
mORMot’s RESTful ORM
From SQL to ORM
April 2014
mORMot’s RESTful ORM
Defining the object
From SQL to ORM
type TSQLRecordPeopleExt = class(TSQLRecord) private fData: TSQLRawBlob; fFirstName: RawUTF8; fLastName: RawUTF8; fYearOfBirth: integer; fYearOfDeath: word; fLastChange: TModTime; fCreatedAt: TCreateTime; published property FirstName: RawUTF8 index 40 read fFirstName write fFirstName; property LastName: RawUTF8 index 40 read fLastName write fLastName; property Data: TSQLRawBlob read fData write fData; property YearOfBirth: integer read fYearOfBirth write fYearOfBirth; property YearOfDeath: word read fYearOfDeath write fYearOfDeath; property LastChange: TModTime read fLastChange write fLastChange; property CreatedAt: TCreateTime read fCreatedAt write fCreatedAt; end;
ID : integerData : TSQLRawBlob
FirstName : RawUTF8LastName : RawUTF8YearOfBirth : integerYearOfDeath : word
ID : INTEGERData : BLOB
FirstName : NVARCHAR(40)LastName : NVARCHAR(40)
YearOfBirth : INTEGERYearOfDeath : INTEGER
April 2014
mORMot’s RESTful ORM Defining the Model and the Server:Props := TOleDBMSSQLConnectionProperties.Create(
'.\SQLEXPRESS','AdventureWorks2008R2','','');Model := TSQLModel.Create([TSQLRecordPeopleExt],'root');VirtualTableExternalRegister(Model,TSQLRecordPeopleExt,Props,'Test.People'
);ServerDB := TSQLRestServerDB.Create(Model,'application.db'),true);ServerDB.CreateMissingTables;Server := TSQLHttpServer.Create('8080',ServerDB);
Defining the Model and the Client:Model := TSQLModel.Create([TSQLRecordPeopleExt],'root');Client := TSQLHttpClient.Create(Server,'8080', Model);
Both will now communicate e.g. overhttp://server:8080/root/PeopleExt
From SQL to ORM
April 2014
mORMot’s RESTful ORM Refining the mapping Model on the
Server:Props := TOleDBMSSQLConnectionProperties.Create(
'.\SQLEXPRESS','AdventureWorks2008R2','','');Model := TSQLModel.Create([TSQLRecordPeopleExt],'root');VirtualTableExternalRegister(Model,TSQLRecordPeopleExt,Props,'Test.People'
);
Model.Props[TSQLRecordPeopleExt].ExternalDB. MapField('ID','Key'). MapField('YearOfDeath','YOD');
ServerDB := TSQLRestServerDB.Create(Model,'application.db'),true);ServerDB.CreateMissingTables;Server := TSQLHttpServer.Create('8080',ServerDB);
From SQL to ORM
ID : integerData : TSQLRawBlob
FirstName : RawUTF8LastName : RawUTF8YearOfBirth : integerYearOfDeath : word
Key : INTEGERData : BLOB
FirstName : NVARCHAR(40)LastName : NVARCHAR(40)
YearOfBirth : INTEGERYOD : INTEGER
April 2014
mORMot’s RESTful ORM
BATCH process
From SQL to ORM
// start the transaction if ClientDist.TransactionBegin(TSQLRecordPeople) then try // start the BATCH sequence Check(ClientDist.BatchStart(TSQLRecordPeople)); // delete some elements for i := 0 to n-1 do Check(ClientDist.BatchDelete(IntArray[i])=i); // update some elements nupd := 0; for i := 0 to aStatic.Count-1 do if i and 7<>0 then begin // not yet deleted in BATCH mode Check(ClientDist.Retrieve(aStatic.ID[i],V)); V.YearOfBirth := 1800+nupd; Check(ClientDist.BatchUpdate(V)=nupd+n); inc(nupd); end; // add some elements V.LastName := 'New'; for i := 0 to 1000 do begin V.FirstName := RandomUTF8(10); V.YearOfBirth := i+1000; Check(ClientDist.BatchAdd(V,true)=n+nupd+i); end; // send the BATCH sequences to the server Check(ClientDist.BatchSend(Results)=200); // in case of success, apply the Transaction ClientDist.Commit; except // In case of error, rollback the Transaction ClientDist.RollBack; end;
ORM CRUD operation
ORM HTTP Client
In-processno latency
ORM HTTP Server
Internet100 ms latency
ORM database core
In-processno latency
April 2014
mORMot’s RESTful ORM
BATCH process Avoids slow Client-Server roundtrips Truly ACID, even with several RDBMS
backends Parameter Array Binding
e.g. Oracle, FireDAC
Multiple INSERT statementdepending on the database dialect
Re-use SQL prepared statement
From SQL to ORM
April 2014
mORMot’s RESTful ORM
Security Rooted on framework’s authentication
Secured by SHA-256 challenge to avoid MIM / replay
Users belong to groups Per-table CRUD access right for each group Optional JSON compression and encryption Over standard HTTP / HTTPS wire
From SQL to ORM
April 2014
mORMot’s RESTful ORM
RESTful Client-Server In-process
Stand-alone client, fast server-side access
Named pipes or GDI messages Stand-alone client, fast server-side access
HTTP/1.1 via kernel-mode http.sys API Part of the OS since Windows XP SP2 Kernel-mode execution, IOCP driven System-wide URI registration: share root and port Used by IIS and WCF Socket-based server is also available
From SQL to ORM
April 2014
From SQL to ORM
Thanks to Bill Meyer for his review
Any feedback is welcome athttp://synopse.info
