Thu 1100 duncan_john_color

<Insert Picture Here>

MySQL Cluster with and without SQLJohn David DuncanSenior Software Engineer, Oracle Corp.

2


Program Agenda

•Overview of MySQL Cluster•SQL interface•NoSQL interfaces–mod_ndb–ClusterJ–Memcache

3

Copyright © 2011 Oracle Corp.

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

4


MySQL Cluster OverviewACID Compliant Relational Database

SQL & NoSQL interfaces

Write-Scalable & Real-TimeDistributed, auto-partitioning (sharding), multi-master

99.999% Availability Shared-nothing, integrated clustering & sub-second recovery, local & geographic replication, on-line operations

Low TCOOpen-source, management & monitoring tools, scale-out on commodity hardware

5


NDB Data Nodes

API Nodes

Node Group 1 Node Group 2

The Basics

6


Geographic Replication

Cluster 1

SynchronousreplicationCluster 2

MyISAM MyISAM InnoDB

Asynchronousreplication

7


MySQL Cluster – Users & ApplicationsHA, Transactional Services: Web & Telecoms

http://www.mysql.com/customers/cluster/

• Web• User profile management• Session stores• eCommerce• On-Line Gaming• Application Servers

• Telecoms• Subscriber Databases (HLR/HSS)• Service Delivery Platforms• VoIP, IPTV & VoD• Mobile Content Delivery• On-Line app stores and portals• IP Management• Payment Gateways

8




SQL interface

9

The long road to SQL JOIN performance

• Initial integration (2004)– Move the data to the query– ... one row at a time– ... one loop iteration in a nested-loop join = 1 network trip

• 2005 - 2009 – Improve efficiency ...– within the existing framework of the MySQL optimizer– e.g. Batch Key Access, push-down filters

• MySQL Cluster 7.2 (2011)– Adaptive Query Localization– Algorithmic changes to optimizer– Push the majority of the query to the data nodes

10


Results: Adaptive Query Localization in MySQL Cluster 7.2

Before: 48.68 sec

mysql> SELECT COUNT(*) FROM residents, postcodes, towns WHERE residents.postcode=postcodes.postcode AND postcodes.town=towns.town AND towns.county="Berkshire";

+----------+ | COUNT(*) | +----------+ | 40001 | +----------+

• After: 2.02 sec

11


<Insert Picture Here>NoSQL interfaces


mod_ndb

ClusterJ

Memcache

12


Node Group 1

Node Group 2

SQL and NoSQL Together

Schema creation

Reporting

Analytics

Real-Time Operations

13


<Insert Picture Here>mod_ndb

14


DataNode

Apache

NDB API

15

DataNode

mod_ndb

http://mod-ndb.googlecode.com/

• First released March 2007• Use REST and JSON to manage data

stored in MySQL Cluster• Open source project based on C++

NDBAPI


DataNode

httpd.conf

<Location /ndb/app/car> SELECT * from cars WHERE PRIMARY KEY = $id; Format JSON</Location>

Apache

NDB API

16

DataNode

mod_ndb


GET /ndb/app/car?id=371 HTTP/1.1Client

17

DataNode

Apache

NDB API

DataNode

mod_ndb


HTTP/1.1 200 OKETag: 700847d8b5b0901a2f2451efc4c4Content-type: application/jsonContent-length: 90

{ "car_id" : 371, "tag" : "807AHC", "state" : "OR", "make" : "Dodge", "year" : 1984}

Client

18

GET /ndb/app/car?id=371 HTTP/1.1

DataNode

Apache

NDB API

DataNode

mod_ndb


my_car = JSON.parse(reply);

reply = XMLHttpRequest.get();

19

Browser

The view from Javascript

DataNode

Apache

NDB API

DataNode

mod_ndb

http://mod-ndb.googlecode.com/

• Very small user community• Not officially supported by a vendor• Not as widely functional as Cluster/J• Not as fast as NDB+Memcache• Moves an unusually large portion of application logic into httpd.conf• Could use a major upgrade (maybe later this year?)

20


<Insert Picture Here>ClusterJ

21

ClusterJ

• High Performance, Light Weight, Easy to Use Direct Connection– In the style of Hibernate / JPA / JDO

• Shared Data storage with:– MySQL server– Native C++ applications– Other ClusterJ applications

• Domain Object Model DataMapper pattern– Data is represented as domain objects– Domain objects are separate from business logic– Domain objects are mapped to database tables

• Does not support relationships – Look at JDO / JPA for these modeling patterns

22


Domain Object Model Mapping

• Tables map to Persistent Interfaces / Classes• Columns map to Persistent Properties

– column names default to property name• Rows map to Persistent Instances• Annotations on Interfaces / Classes customize

mappings• User chooses to write:

– User interface (ClusterJ then generates implementation class)– Persistent class (ClusterJ provides base implementation class)

23


ClusterJ – Generated Class@PersistenceCapable(table="employee")public interface Employee {

long getId(); void setId(int id);

@Column(name="full_name") String getName(); void setName(String value);

int getSalary(); void setSalary(int value);

Integer getAge(); void setAge(Integer value);}

24


ClusterJ – Dynamic Objectpublic class Employee extends DynamicObject {

public String table() { return "employee";

long getId() { return (Long)get(0); } void setId(long value) { set(0, value); }...// other fields and behavior}

25


Numeric Column Mapping• Java boolean, Boolean

– BIT(1)• Java byte, Byte

– BIT(1) to BIT(8)– TINYINT

• Java short, Short– BIT(1) to BIT(16)– SMALLINT– YEAR

• Java int, Integer– BIT(1) to BIT(32)– INT

• Java long, Long– BIT(1) to BIT(64)– BIGINT– BIGUNSIGNED

• Java float, Float– FLOAT

• Java double, Double– DOUBLE

• Java BigDecimal– NUMERIC– DECIMAL

• Java BigInteger– NUMERIC– DECIMAL

26


Date Column Mapping

• Java util Date– DATETIME– TIMESTAMP– TIME– DATE

• Java sql Date– DATE

• Java sql Time– TIME

• Java sql Timestamp– DATETIME– TIMESTAMP

27


Variable Size Column Mapping

• Java String– CHAR– VARCHAR– TEXT

• Java byte[ ]– BINARY– VARBINARY– BLOB

28


ClusterJ Features

• Character Set Translation (all MySQL charsets)• Automatic detection of primary keys, indexes• Compound Primary Keys• Ordered (btree) indexes• Unique (hash) indexes• Automatic use of partition key• Multi-threaded applications

29


ClusterJ Limitations• No Relationships

– primitive types only• No Multi-table inheritance

– single table per persistent interface or class• No joins in queries

– column comparisons and boolean operators• No Table creation

– user needs to create tables and indexes• No Lazy Loading

– entire record is loaded at one time, including LOBs

30


ClusterJ Interfaces

• SessionFactory– Instance per connection to cluster

• Session– Instance per "user"– persist(), remove(), update(), write()– Find by ID

• Transaction– Instance per Session– begin(), commit(), rollback()

• Query– Multiple instances per Session

31


ClusterJ User View

SessionFactory Configuration Properties

Session andTransaction

Domain Object Domain

Object

Domain Object

Domain Object



Object

Domain Object

Domain Object



Object

Domain Object

Domain Object

32


ExampleSession session;void getSession() { session = sessionFactory.getSession();}Employee createEmployee(long id, String name, int salary, int age) { Employee employee = session.newInstance(Employee.class); employee.setId(id); employee.setName(name); employee.setSalary(salary); employee.setAge(age); session.persist(employee); return employee;}

33


ExampleTransaction transaction;

void getTransaction() { transaction = session.getTransaction();}

void createEmployees() { getTransaction(); transaction.begin(); createEmployee(1, "Amos", 10000, 44); createEmployee(2, "Barbara", 14000, 48); createEmployee(3, "Chuck", 78000, 61); createEmployee(4, "Dave", 3000, 22); transaction.commit();}

34

Query

• Builder pattern• Similar to JPA criteria query• Compare column values to parameters:

– equal, lessEqual, greaterEqual, lessThan, greaterThan, in– comparison with null

• Combine terms using boolean operators:– or, and, not

• Execution is optimized to use indexes– primary or unique key lookup– ordered scan for complete or partial keys– table scan if no index can be used

35


QueryDomainType qemp = builder.createQueryDefinition(Employee.class) ;

Predicate geAge = qemp.get("age") .greaterEqual(qemp.param("ageFloor"));Predicate leSalary = qemp.get("salary") .lessEqual(qemp.param("salaryCap"));

qemp.where(geAge.and(leSalary));

Query query = session.createQuery(qemp) ;

query.setParameter("ageFloor",33);query.setParameter("salaryCap", 44000);

List<Employee> results = query.getResultList() ;

Query Example

36


Performance

37


<Insert Picture Here>Memcache API

38


Memcached Overview:Two levels of hashing

Memcache

httpd memcached

memcached

memcachedmemcache key

PHP/Perl

friends:12389

hash key to find data

hash key to pick server

39


Cache hit

Memcache

httpd

memcachedPHP/Perl

friends:12389

hash key to find data

VALUE friends:12389 0 31\r\n101, 11009, 11150, 55881, 77798 \r\n


40


Cache miss: fetch from DB

Memcache

httpd

memcachePHP/Perl hash key to find data


NOT FOUND

mysql

MySQL Slave

SELECT friend_id FROM user_friends WHERE user_id = ?

memcached

41


Expected Latency & Throughput

memcache

httpd

PHP/Perl

MySQL Slave

1,000s of operations/sec.~ 2 ms round trip

10,000s of operations/sec.

~ 200 µs round trip

mysql

memcached

42


Goals • Access stored data directly from memcache clients

– Memcached perspective: • MySQL Cluster is a write-scalable, replicated data store

– with reliable in-memory storage,– plus on-disk storage when data is too big for

memory.– MySQL Cluster perspective:

• memcache is a high performance API– providing easy access to in-memory data,– plus an extra layer of caching when data is on disk.

43


Goals• Support existing schemas and all MySQL data types• Cache MySQL Cluster data inside memcached when

desired– with automatic cache management– and flexibility to fine-tune (or disable) the cache policies

• Support the whole memcache protocol• Achieve superior performance

– latency as expected from memcached– throughput as expected from memcached

44


A Key Prefix

user:1248

the prefix the database key

45


Standard Tables in ndbmemcache• meta

– stores configuration schema version (for upgrade compatibility); consider it to be read-only

• ndb_clusters• containers

– existing tables where data is stored

• cache_policies– rules describing how it can be accessed

• key_prefixes• memcache_server_roles• last_memcached_signon• demo_table

46


A key-prefix mapping

Memcache key

prefixCluster Container Cache

Policy

47


keyprefix Cluster Con-

tainerCache Policy

ServerRole

ID

A memcache server role


tainerCache Policy


tainerCache Policy


tainerCache Policy

48


Measured Latency memcachetest -t 2 -M 7000 -c 25000

49


Measured Throughput memslap

50


Limitations of Memcache API

• The size of stored values is limited to the MySQL Cluster row size (without BLOBs)– This is about 13KB (up from 8KB in 7.1)

51



Program Review

Overview of MySQL ClusterHighly available Write-scalable

SQL interfaceRedesigned in 7.2

NoSQL interfacesmod_ndb, ClusterJ, Memcacheplus: C++ NDBAPI, LDAP, etc.

52

Thu 1100 duncan_john_color

Technology

Transcript of Thu 1100 duncan_john_color