Modernización del manejo de datos con v fabric
-
Upload
software-guru -
Category
Technology
-
view
584 -
download
0
description
Transcript of Modernización del manejo de datos con v fabric
![Page 1: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/1.jpg)
© 2011 VMware Inc. All rights reserved
#SGvFabric
![Page 2: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/2.jpg)
2
vSphere 5
vFabric: What’s in it?
Application Services vFabric
Frameworks & Tools Rich Web
Social and Mobile
Data Access
Integration Patterns
Batch Framework
Spring Tool Suite
Perf, Mgmt Hyperic / Insight
Application Srv tc Server
Web Runtime ERS
Elastic Data Grid Gemfire / SQLFire
Messaging RabbitMQ
EM4J Data Director
vCops/APM
DBaaS vPostgres
![Page 3: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/3.jpg)
3
Cloud-scale challenge…
![Page 4: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/4.jpg)
4
Challenge
Managing on-line applications on a cloud-scale is hard. As number of users grows, database becomes the bottleneck.
![Page 5: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/5.jpg)
5
DB Bottleneck
Scales…
Scales…
![Page 6: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/6.jpg)
6
Cause
Traditional databases were never designed to support thousands of concurrent users.
![Page 7: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/7.jpg)
7
Traditional DB Characteristics
§ Designed against no longer relevant constraints • Network unreliable/slow • RAM prices prohibitive
§ One size fits all • Designed for everything, optimized for nothing • Often incompatible with modern workloads
§ Centralized in nature • Data change capture an
afterthought • Lacks data partitioning
facilities
§ Obsessed with ACID • Constant contention for
resources cause locks
§ Monolithic design § Requires lots of hardware to
scale
![Page 8: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/8.jpg)
8
Traditional DB Loves IO
First write to LOG
Second write to Data files
Buffers primarily tuned for IO
![Page 9: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/9.jpg)
9
12%
8%
21%
19%
10%
30% Data Btrees keys Logging Locking Latching Buffer management
Source: Research by MIT and Brown: “OLTP Under the Looking Glass” by S. Harizopoulos, D. J. Abadi, S. Madden, M. Stonebraker, SIGMOD 2008.
Percentage of Computer cycles based on 3.5M
sample
Transaction in Traditional DB
![Page 10: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/10.jpg)
10
Cloud-scale solution…
![Page 11: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/11.jpg)
11
Apparent Choices
Build expensive database clustering solution or lengthy re-write for “big data”?
![Page 12: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/12.jpg)
12
Next generation option
SQLFire is different; it’s build for speed and scale.
Scale much?
Hablo SQL?
![Page 13: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/13.jpg)
13
New Approach
Elastic, in-memory database designed specifically for speed and low latency accessible through a familiar SQL interface.
![Page 14: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/14.jpg)
14
SQLFire Characteristics
§ Highly concurrent data structures resident in and optimized for main memory
§ Rethink ACID transactions; all state resides in distributed memory to avoid any single points of contention
§ Partition-aware DB design spreads workloads across both data set and physical nodes
§ Shared nothing logs on disk; application writes are never exposed to the disk seek latencies
§ Parallelize data access and application behavior; dynamically “shard SQL”
§ Dynamic rebalancing of data as cluster size grows/shrinks. Most efficient way of managing resources/data.
![Page 15: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/15.jpg)
15
SQLFire speed…
![Page 16: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/16.jpg)
16
SQLFire v Traditional Databases
SQLFire response times are faster and more consistent under increased database load.
![Page 17: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/17.jpg)
17
Sample Comparison
§ Spring Travel Application § Similar hardware (8 vCPU, 4GB) § Out-of-the-box configuration
SQLF R/T (ms) SQLF CPU % MySQL R/T (ms) MYSQL CPU %
14 9 25 1
8 32 23 19
5 61 172 76
6 77 fail fail
984 98 fail fail
![Page 18: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/18.jpg)
18
Response Time
0
20
40
60
80
100
120
140
160
180
200
0 500 1000 1500 2000
MySQL increased with load
SQLFire near constant much lower
R/T
Threads
![Page 19: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/19.jpg)
19
Number of Threads
0
200
400
600
800
1000
1200
0 1000 2000 3000 4000 5000 6000 7000 8000
MySQL reaches saturation
at 1850 threads
SQLFire scales to 7200 threads
with 1 second R/T
R/T
Threads
![Page 20: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/20.jpg)
20
Distributed data…
![Page 21: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/21.jpg)
21
Why Scale Horizontally?
Sub-divide system into independent data sets, eliminate distributed transactions to achieve elasticity, linear scalability and predictable latency.
![Page 22: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/22.jpg)
22
Horizontal Scalability – Throughput
0
200
400
600
800
1000
1200
1400
0
100000
200000
300000
400000
500000
600000
700000
800000
2 4 6 8 10
Clie
nt th
read
s
Que
ries
per s
econ
d
Number of servers
queriesPerSecond
client threads
![Page 23: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/23.jpg)
23
Horizontal Scalability – Consistency/HA
§ Resiliency through replication, synchronous but in parallel
§ Row updates are always atomic; no need for transactions
§ Shared nothing architecture, including storage § Instant failover at protocol level § Apps retain their connections
§ Data remains available
SQLFire
APP
SQLFire SQLFire
![Page 24: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/24.jpg)
24
Data management strategies…
![Page 25: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/25.jpg)
25
Data strategies – Partitioning
§ Balances data across SQLFire cluster
§ Delivers redundancy for high availability
SQLFire
APP
SQLFire SQLFire
Write operation (with 2 redundant copies)
Read operation
![Page 26: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/26.jpg)
26
SQLFire Hash Partitioning
§ Partition by column or primary key • Can specify multiple columns
• Uses hashCode() for single column or primary key • Uses serialized bytes for multiple columns
• Creates uniform distribution of data across the cluster
// Partition by column CREATE TABLE MY_TABLE ( . . . ) PARTITION BY COLUMN ( COLUMN_A) // Partition by primary key CREATE TABLE MY_TABLE ( . . . ) PARTITION BY PRIMARY KEY
![Page 27: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/27.jpg)
27
SQLFire Range Partitioning
§ Partition by range of column values • Can specify multiple ranges
• Colocates data in specified ranges • Used to ensure locality of data in a partition for range queries or cross table
joins
// Partition by range CREATE TABLE MY_TABLE ( . . . ) PARTITION BY RANGE ( COLUMN_A) ( VALUES BETWEEN 1 AND 10, VALUES BETWEEN 50 AND 60 )
![Page 28: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/28.jpg)
28
SQLFire List Partitioning
§ Partition by a set of column values • Can specify column value sets
• Colocates data with specified column values • Used to ensure locality of data in a partition for sets of values or cross table
joins
// Partition by list CREATE TABLE MY_TABLE ( . . . ) PARTITION BY LIST ( COLUMN_A) ( VALUES (‘VALUE_A’, ‘VALUE_B’), VALUES (‘VALUE_Y’, ‘VALUE_Z’) )
![Page 29: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/29.jpg)
29
SQLFire Expression Partitioning
§ Partition by a column expression • Expression must be valid SQL function
• Must reference only columns in the table • Hash partition with value determined by the expression
// Partition by expression CREATE TABLE MY_TABLE ( . . . ) PARTITION BY ( MONTH ( MY_DATE ) )
![Page 30: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/30.jpg)
30
SQLFire Default Partitioning
§ Default hash partitioning strategy • Start server with table-default-partitioned property set to true!
• First foreign key whose referenced primary key is also a partition column • Primary key • First unique key • SQLFire-generated row id
// No PARTITION BY clauses CREATE TABLE MY_TABLE (COLUMN_A INT NOT NULL CONSTRAINT A_PK PRIMARY KEY, . . .) CREATE TABLE MY_OTHER_TABLE (COLUMN_B INT NOT NULL CONSTRAINT B_PK PRIMARY KEY, COLUMN_C INT CONSTRAINT A_FK REFERENCES MY_TABLE (COLUMN_A), . . .)
![Page 31: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/31.jpg)
31
Data strategies – Replication
§ Copies all data across SQLFire cluster
§ Appropriate for reference data
SQLFire
APP
SQLFire SQLFire
Write operation (with replicated copies)
Read operation
![Page 32: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/32.jpg)
32
SQLFire Replicated Tables
§ Created by default with no PARTITION BY clause
§ Created with REPLICATE clause
§ Reference data or fact tables are good candidates
§ Replicates data across all peers in server group
§ Replication is parallel and synchronous
§ Automatic replication failure detection
// Replication example CREATE TABLE MY_TABLE ( . . . ) REPLICATE
![Page 33: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/33.jpg)
33
Topology strategies…
![Page 34: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/34.jpg)
34
Topology
Client-server JVM JVM
JVM JVM
APP APP
JVM
SQLFire SQLFire SQLFire
SQLFire Locator
![Page 35: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/35.jpg)
35
Topology
Embedded Peer-to-peer
JVM JVM JVM
SQLFire
APP
SQLFire
APP
SQLFire
APP
![Page 36: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/36.jpg)
36
Synchronization strategies…
![Page 37: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/37.jpg)
37
Synchronous strategy
In data-center or over private network
JVM JVM
JVM JVM JVM
APP APP
SQLFire SQLFire SQLFire
SQLFire Locator
JVM JVM
JVM JVM JVM
APP APP
SQLFire SQLFire SQLFire
SQLFire Locator
Site 1 Site 2 Redundancy Zone A Redundancy Zone B
![Page 38: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/38.jpg)
38
Asynchronous strategy
Multi-site over the Cloud
JVM JVM
JVM JVM JVM
APP APP
SQLFire SQLFire SQLFire
SQLFire Locator
JVM JVM
JVM JVM JVM
APP APP
SQLFire SQLFire SQLFire
SQLFire Locator
Site 1 Site 2
WAN Gateway
![Page 39: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/39.jpg)
39
Data strategies – Server Groups
Group 1
Group 2
SQLFire Cluster
JVM
SQLFire
JVM
SQLFire
JVM
SQLFire
JVM
SQLFire
JVM
SQLFire
JVM
SQLFire
Group 3
![Page 40: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/40.jpg)
40
data demo…
![Page 41: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/41.jpg)
41
Summary…
![Page 42: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/42.jpg)
42
Why SQLFire?
In-memory, delivers maximum speed and minimum latency Horizontally scalable, easily adopts to changing workloads, usage patterns Familiar SQL interface, accessible from Java and .NET
Speed
Scale
SQL
![Page 43: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/43.jpg)
43
If you forgot everything else…
SQLFire is better in supporting on-line applications than traditional databases.
![Page 44: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/44.jpg)
44
Sample Apps § Side-by-side comparison of SQLFire v MySQL
performance - https://github.com/vFabric/sqlf-demo
§ Demo call-center application, SQLFire configuration scripts https://github.com/vFabric/sqlf-cloud
Demo Video § Real-life performance comparison (YouTube, 3 min.)
http://youtu.be/HV-broQHJlk
SQLFire Artifacts
![Page 45: Modernización del manejo de datos con v fabric](https://reader034.fdocuments.in/reader034/viewer/2022051817/5489b560b47959e20c8b5a4e/html5/thumbnails/45.jpg)
45
http://vmware.com/go/sqlfire
@vFabricSQLFire, @_cmc
The end