An Overview of a Scalable Distributed Database System: SD-SQL Server Witold LITWIN, Soror SAHRI &...
-
Upload
osborne-williams -
Category
Documents
-
view
227 -
download
0
Transcript of An Overview of a Scalable Distributed Database System: SD-SQL Server Witold LITWIN, Soror SAHRI &...
An Overview of a Scalable Distributed Database System: SD-SQL Server
Witold LITWIN, Soror SAHRI & Thomas SCHWARZ
[email protected] [email protected] [email protected]
Ceria Laboratory Comp. Eng. Dep. Paris-Dauphine University Santa Clara U.
BNCOD 2006
2BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
1. Introduction
2. Architecture
3. Command Interface
4. Processing
5. Performance
6. Conclusion & Future Work
Overview
3BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Most DBSs have distributed/parallel versions with partitioned tablesSQL Server, Oracle, DB2, MySQL, Postgres…
Partitioned Tables
4BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
BENEFITS OF PARTITIONING
BENEFITS OF PARTITIONING
Partitioning can provide tremendous benefits to a wide variety of applications byimproving manageability, performance, and availability. It is not unusual forpartitioning to improve the performance of certain queries or maintenanceoperations by an order of magnitude. Moreover, partitioning can greatly simplifycommon administration tasks.
Partitioning also enables database designers and administrators to tackle some ofthe toughest problems posed by cutting-edge applications. Partitioning is a keytool for building multi-terabyte systems or systems with extremely highavailability requirements.
Partitioning inOracle Database 10g Release 2
An Oracle White PaperMay 2005
5BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
DBSs require manual partitioning And manual repartitioning when tables scale-up
DBSs do not provide dynamically scalable tables
Hassle of Partitioning
6BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
http://ceria.dauphine.fr/CERIA-publications.html Research Report, December 2005
[Oracle Database 10g]
Facts
7BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Distributed Partitioning of Relational Tables
Scalable Distributed Database System
SD-DBS
SD-SQL Server Goal
8BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Several SDDS schemes are well-known by now: LH*, RP*, k-RP*, LH*RS…Chord, VBI & most of P2P schemes
The domain has over 20.000 references on Google An SD-DBS reuses SDDS design principles
With DB management specificity
A scalable distributed data structure Specifically designed for possibly very large
data on multi-computers or networks of WSs P2P & Grids in modern vocabulary
Why SDDS Role Model?
Role Model: an SDDS
9BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
The first and yet the only SD-DBS Implements the SD-DBS architecture
• Litwin, Schwartz & Risch (2002)
Runs on Microsoft SQL Server 2000Shared Nothing ArchitectureUp to 250 nodes at present
SD-SQL Server
10BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
LinkedSQL
Servers
D1 D2 Di Di+1
S S PC D1_T
_D1_T _D1_T
User/Application
T
sd_create_table
User/Application
sd_insert
_D1_TSplit
NDBs
SD-SQLserver
SD-SQLserver
SD-SQLclient
SD-SQL Server
Managers
SD-SQLpeer
Gross Architecture
11BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
DB1
Node1 Node2 Node3 Nodei
DB1
……
DB1
MDB
DB1 SDB DB2 SDB
DB2 DB2
Nodes, SDBs & NDBs
12BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Primary NDB First for an SDB
Client NDB• Interfaces applications & users• Carries only images
• No actual tables with application data
Server NDB
• Carries actual tables• segments
Peer NDB• Both functions
NDB Types
13BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
For the application: a table of an SDB Internally: a collection of segments behind client images A segment is an SQL table
• One per NDB of the SDB • Sharing the scalable table scheme
• Except its check constraint• Min and Max value of the partition key
• With size limit• Splitting when overflows occur
The check constraints partition the partition key space
Scalable (Distributed) Table
14BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
The primary segmentFirst allocated for a new table At some server or peer NDB of SDB
• The peer creating the table• The primary server of the client creating the table
Scalable (Distributed) Table
15BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
LinkedSQL
Servers
D1 D2 Di Di+1
S S PC D1_T
_D1_T _D1_T
User/Application
T
sd_create_table
User/Application
sd_insert
_D1_TSplit
NDBs
SD-SQLserver
SD-SQLserver
SD-SQLclient
SD-SQL Server
Managers
SD-SQLpeer
Gross Architecture
16BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
For the application, it the client image is the table The image name is the table name
Primary imageCreated during the scalable table creation
• at the client or peer NDB creating the table Secondary images
Created later on On other NDBs of the SDB For local applicationsBy a dedicated command
• sd_Create Image….
Scalable (Distributed) Table
17BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Internally, every image is a specific SQL Server view of the segments:Distributed partitioned union view
CREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_T UNION ALL SELECT * FROM N3.DB1.SD._N1_T
UNION ALL SELECT * FROM N4.DB1.SD._N1_T
Updatable• Through the check constraints
With or without Lazy Schema Validation
Scalable (Distributed) Table
18BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
LinkedSQL
Servers
D1 D2 Di Di+1
S S PC D1_T
_D1_T _D1_T
User/Application
T
sd_create_table
User/Application
sd_insert
_D1_TSplit
NDBs
SD-SQLserver
SD-SQLserver
SD-SQLclient
SD-SQL Server
Managers
SD-SQLpeer
Gross Architecture
19BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Store various SD-SQL Server meta-data In particular about each scalable table
At each server or peer NDB• SD.Size meta-table
• Segment capacity• The number of stored tuples triggering a split• Same for every segment at present
• SD.RP meta-table • the actual partitioning of the scalable table
• The location of each segment• SD.Primary table
• The location of the SD.RP table for each segment in the NDB
SD SQL Server Meta-Tables
20BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
At every client or peer NDB In SD.Image table
• All the local images• The name of the image• The type
• Primary or secondary
• The number of segments • As seen by an image• Not necessarily the actual one
SD SQL Server Meta-Tables
21BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
At every NDB SD.SDBNode points towards the primary NDB SD.MDBNode points towards the MDB
At MDB SD.Nodes indicates all the available SD-SQL Server nodes
• Over linked SQL Server nodes
SD.SDB describes all the SDBs At every primary NDB
SD.NDB points to every NDB of the SDB
SD SQL Server Meta-Tables
22BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
…….
DB1 SDB
N1.DB1 N2.DB1 N3.DB1
T Scalable Table
Ni.DB1
1000Size
N1.DB1Primary
Ni.DB1Nodes
N1.DB1
N2.DB1
N3.DB1
RP
Meta-TablesMeta-Tables
Scalable Tables: Meta-data
SDBNode
23BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
The number of segments in a scalable table may growAn overflowing segment splits
• Creating one or more new segments A split occurs when an insert overflows the
segment capacity The trigger launches the split as an
asynchronous job called splitter To avoid the application level timeout
Scalable Table Expansion
24BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
LinkedSQL
Servers
D1 D2 Di Di+1
S S PC D1_T
_D1_T _D1_T
User/Application
T
sd_create_table
User/Application
sd_insert
_D1_TSplit
NDBs
SD-SQLserver
SD-SQLserver
SD-SQLclient
SD-SQL Server
Managers
SD-SQLpeer
Gross Architecture
25BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Every new segment Is basically created at an existing NDB that does
not yet have any segments of the expanding table
• provided there is any Otherwise a new NDB is first appended to SDB
• Provided there is an available SD SQL Server nodeInherits the “father”’s schema Gets its new check constraintGets indexes as defined at the “father”
Scalable Table Expansion
26BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
S
b+1
S S1
pb+1-p
p=INT(b/2)
C( S)= { c: c < h = c (b+1-p)}
C( S1)={c: c > = c (b+1-p)}
Check Constraint?
b
SELECT TOP Pi * INTO Ni.Si FROM S ORDER BY C ASCSELECT TOP Pi * WITH TIES INTO Ni.S1 FROM S ORDER BY C ASC
Single Segment Split Single Tuple Insert
27BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Single Segment Split Bulk Insert
p = INT(b/2)C(S) = {c: l < c < h } à { c: l ≤ c < h’ = c (b+t–Np)}C(S1) = {c: c (b+t-p) < c < h}…C(SN) = {c: c (b+t-Np) ≤ c < c (b+t-(N-1)p)}
Pn
S
b
b+t
(b)
P1
b+t-np
S
b
(a)
S
b
(c)
b+t-np
P1
S1
b
Pn
SN
b
p p
Single segment split
28BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Sk
b
S
b
S1
b
S1, n1
b
p
Sk
b
Sk, nk
b
p
Multi-Segment Split Bulk Insert
Multi-segment split
29BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
SDB DB1SDB DB1
Scalable Table T
sd_insert
N1 N2 N4N3
NDBDB1
NDBDB1
NDBDB1
sd_insert
NDBDB1
Ni
sd_create_node
sd_insert
N3
NDBDB1
sd_create_node_database
NDBDB1
…….
sd_create_node_database
SDB DB1
Split with SDB Expansion
32BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Image Adjustment The splits do not modify synchronously the images Any split makes every image outdated The client or peer verifies every image dynamically when a query to the image comes in
Image checkingImage adjustment if necessary
33BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Get the number of segments presented in the image, N1
Get the number of segments of the scalable table, N2
Compare N1 and N2: If N1<N2 then Image Adjustment
Alter the partitioned view definition
Image Adjustment
34BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Image: Example
N1.DB1 N2.DB1 N3.DB1
T Scalable Table
CREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_TCREATE VIEW T AS SELECT * FROM N2.DB1.SD._N1_T UNION ALL SELECT * FROM N3.DB1.SD._N1_T UNION ALL SELECT * FROM N4.DB1.SD._N1_T
PrimaryImage
DB1 SDB
N4.DB1
T Image
35BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
The application interface manipulates scalable tables through SD-SQL Server commands
The SD-SQL Server commands start with ‘sd_’ to distinguish from SQL Server commands for static tables
INSERT sd_insert
CREATE TABLE sd_create_table
Application Interface
36BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Node Creationsd_create_node ‘Dell1’ /* Server by default */ sd_create_node ‘Ceria’, ‘client’
Node Alterationsd_alter_node ‘Ceria’, ‘ADD server’ /* Becomes peer*/
Node Removalsd_drop_node ‘Ceria’
Nodes Management
37BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
SDB Creationsd_create_scalable_database
‘SkyServer’, ‘Dell1’, ‘Server’, 2
/* Creates the primary SkyServer NDB as well at Dell1*/
SDB Alterationsd_create_node_database
‘SkyServer’, ‘Ceria’, ‘Client’ SDB Removal
sd_drop_scalable_database ‘SkyServer’
SDB & NDB Management
38BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable Table Creationsd_create_table ‘PhotoObj (objid BIGINT PRIMARY
KEY…)’, 10000• No foreign keys yet
Scalable Table Alterationsd_alter_table ‘PhotoObj ADD t INT’, 1000sd_create_index ‘run_index ON Photoobj (run)’sd_drop_index ‘PhotoObj.run_index’
Scalable Table Removalsd_drop_table ‘PhotoObj’
Scalable Tables Management
39BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Secondary Image Creationsd_create_image ‘Ceria’, ‘PhotoObj’sd_create_image ‘Dell2’, ‘PhotoObj’
Secondary Image Removalsd_drop_image 'PhotoObj’
Image Adjustment
40BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
A view of an imageInvolving perhaps static tables And perhaps static views…
Declared under SD-SQL Server by the SQL Server CREATE VIEW command
Scalable View
41BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
USE SkyServer /* SQL Server command */
Scalable Update Queriessd_insert ‘INTO PhotoObj SELECT * FROM
Ceria5.Skyserver-S.PhotoObj’ Scalable Search Queries
sd_select ‘* FROM PhotoObj’ sd_select ‘TOP 5000 * INTO PhotoObj1 FROM
PhotoObj’, 500
Scalable Queries Management
42BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Image Binding
Let Q a scalable query using the PhotoObj image: sd_select ‘COUNT (*) FROM PhotoObj’
Find Images in Q
Check PhotoObj Image for Correctness
Adjust PhotoObj Image if needed
Send Q’ to SQL Server for Execution
Command Processing
43BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Concurrency
SD-SQL Server processes every command as SQL distributed transaction at Repeatable Read isolation level Tuple level locks Shared locks Exclusive 2PL locks Much less blocking than the Serializable Level
44BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Splits use exclusive locks on segments and tuples in RP meta-table. Shared locks on other meta-tables: Primary, NDB
meta-tables
Scalable queries use basically shared locks on meta-tables and any other table involved
All the conccurent executions can be shown serializable
Concurrency
45BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Splitter sd_alter_table
Dell1.SkyServer
Dell1
Dell2
Dell3
RP
PhotoObj
Exclusive Lock Waiting
Exclusive Lock
Shared Lock
Exclusive Lock
X
X
Concurrency: Example
46BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Experimental Environment
6 Machines Pentium IV 1.7 GHz RAM: 780 Mb & 1 Gb Operating System: Windows 2K Server Ethernet Network: max bandwidth of 1 Gb/s
Use of SQL Analyzer for editing queries Use of SQL Profiler to take measurements
47BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
The SkyServer Benchmark
We use SkyServer database as benchmarkProvided and installed at Ceria by Dr. Gray
SkyServer brings the entire database of the Sloan Digital Sky Survey, SDSSWe use of the PhotoObj table as an example
scalable table In our experiments, PhotoObj has almost 159 K
tuples (about 260 MB)
• Originally, it has 14 M tuples
48BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
21.40 26.4212.157.11
37.1232.0818.9310.54
60.7359.5938.8722.42
104.15104.01
56.7946.17
66.5377.86
146.57165.11
0
50
100
150
200
2 3 4 5
Number of Segments
Spl
it Ti
me
(sec
)
10000 tuples 20000 tuples 40000 tuples
80000 tuples 160000 tuples
Splitting PhotoObj with 160 k tuples into 2…5 segments, according to segment capacity
Split Time
49BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Split Time
0
50
100
150
200
250
2 3 4 5
Number of Segments
Exe
cutio
n Ti
me
(s)
0 index 1 index 2 index 3 index
Splitting PhotoObj with 160 k tuples and indexes into 2… 5 segments according to segment capacity
50BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Split Time Analysis Longer split time may timeout a query put on wait
Future solution: Incremental Splitting The splitter moves tuples by an increment at
a time• Let us say 1000 tuples
Then ends up by calling upon itself• The query may proceed as the splitter releases
the exclusive lock on the RP tuple
The process continues for next increment etc as long as there are tuples to move
51BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
(Q) sd_select ‘COUNT (*) FROM PhotoObj’
Query (Q1) execution time
0
0,5
1
1,5
2
39500 79000 158000
Capacité de PhotoObj
Te
mp
s
d'e
xé
cu
tio
n d
e
(se
c)
Adjustment on a Peer Checking on a Peer
SQL Server Peer Adjustment on a Client
Checking on a Clientj SQL Server client
Image Adjustment
52BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
0.811 0.826 0.846 0.853
0.11 0.126 0.14 0.153
00.10.20.30.40.50.60.70.80.9
Niveau 0 Niveau 1 Niveau 2 Niveau 3
Scalable View Level
Exe
cuti
on
Tim
e (
sec)
With PhotoObj Image Adjustment With only PhotoObj Image Checking
Scalable View Processing
(Q) sd_select ‘COUNT (*) FROM Ti’
53BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
(Q): sd_select ‘COUNT (*) FROM PhotoObj’
Execution time of (Q) on SQL Server and SD-SQL Server
93156
220250
326
106
164226
256
343283
20393
356
436
220203123
76160
100
200
300
400
500
1 2 3 4 5
Nombre de segments
Tem
ps
d'e
xécu
tio
n
(sec
)SQL Server-Distr SD-SQL Server
SQL Server-Centr SD-SQL Server LSV
SD-SQL Server / SQL Server
54BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Scalable tables are now a reality with SD-SQL ServerNo more manual repartitioning
• Unlike in any other DBS we know about Performance analysis proves
Efficiency of our design Immediate utility of SD-SQL Server
Conclusion
55BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
SQL Server 2005 portage Incremental splits Virtual repository of eGov documents
SQL Server XML View Foreign keys for scalable tables More performance measurements
Skyserver & other benchmarks Error processing High availability
Parity segments Application to other DBMSs
Oracle, DB2, etc.
Future Works
56BNCOD 2006 – July 18th, 2006 An Overview of a Scalable Distributed Database System: SD-SQL Server
Thank you for your attention
Work performed between 2003 -2006
Partly founded byMsResearch
EEC Icons ProjectEEC E-Gov Project