Marko Mäkelä - InnoDB

MySQL 5.7InnoDB—What’s NewSunny Bains – sunny.bains@oracle.com

Senior Engineering Manager

Marko Mäkelä marko.makela@oracle.com

Senior Principal Software Engineer

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

Program Agenda

Performance

Features

Download & Blogs

Performance : Transactions

Transaction poolFixed chunks of 4MB each

sizeof(trx_t) reduced from 1144 to 712 bytes

Ordered on address, locality of reference

Improves performance of read-write transaction list scans

Reduces malloc()/free() overhead

Transaction life cycleAll transactions are considered as read-only by default

Read only transaction start/commit mutex free

No application changes required

Read views are cached

Read view created iff a RW transaction started since the last snapshot

Reduce contention when implicit → explicit row lock conversion is done

Transaction life cycleHigh priority transactions (Replication GCS)

Cannot be rolled back

Can jump the record lock queue – prioritized

Currently not visible to end users

Allows rollback of arbitrary user transactions internally• If purge blocked due to long running transactions

Performance : Memcached Plugin

8.0 16.0 32.0 64.0 128.0 256.0 512.0 1024.00

200,000

400,000

600,000

800,000

1,000,000

1,200,000

MySQL 5.7 vs 5.6 - InnoDB & Memcached

MySQL 5.7

MySQL 5.6

Connections

Queries per Second

Performance : Sysbench OLTP Read-Only

Performance : Sysbench OLTP Read-Write

Performance : Temporary Table Optimizations

DDL changesNot stored in the data dictionary – lower mutex contention

Special shared temporary tablespace – lower IO overhead

Compressed tables done the old way, separate .ibd file

Tablespace recreated on startup

Performance : Temporary Table Optimizations

DML changesSpecial UNDO logs that are not redo logged

Undo logging required for rollback to savepoint

Changes to the temporary tablespace are not redo logged

No fsyncs() on the temporary tablespace

Configuration variablestemp_data_file_path := same format as the system tablespace

e.g., ibtmp1:12M:autoextend – default setting

Performance : Temporary Tables Benchmarks

5.6 5.70

Temporary table CREATE/DROP

Version

5.6 5.70

Insert 5M rows

Versions

5.6 5,70

Delete 5M rows

Version

5.6 5.70

Update 5M rows

Version

Performance : Buffer pool improvements

Use atomics for page reference countingBug#68079 - INNODB DOES NOT SCALE WELL ON 12 CORE

Flush list traversalFix flush and LRU list rescanning

Previously after flushing a page we would start from the tail again

Extra work and increased mutex contention

Performance : Buffer pool improvements

Multithreaded flushing5.6 introduced a separate thread for flushing

5.7 allows multiple threads

--innodb-page-cleaners := 1..64 – default is 1

Performance : Redo log

Better concurrency and faster recoveryRemove unused code

Fix read on write issue – pad the log buffer before writing to disk

Refactor and clean up the mini-transaction code

Optimize mutex acquire/release during log checkpoint

Write data file names to the redo log

No need to open clean *.ibd files on crash recovery

Performance : Memcached Plugin

Leverages the read-only transaction optimizationsFixed several bottlenecks in the Memcache and the plugin code

1.1 Million GET/s

Limiting factors were:• The network• Memcached client

Performance : Memcache Benchmarks

Performance : index->lock

Increased concurrency, improved performanceVery complex fix

Previously entire index X latched for tree structure modification

B-tree internal nodes not latched before fix

New SX lock mode – compatible with S lock mode

Increases concurrency e.g., index->lock(SX), reads can proceed

Performance : DDL & Truncate

Truncate table is now atomicPreviously DROP + CREATE

ID mismatch or .ibd missing If crash after DROP but before CREATE

More schema-only ALTER TABLE supportedRename index

VARCHAR extension

Faster ALTER TABLEBug#17657223 EXCESSIVE TEMPORARY FILE USAGE IN ALTER TABLE

WL#7277 Bulk Load for Index Creation

Features : Partitions

Native PartitioningReduced memory overhead

Allows us to easily add• Foreign key support• Full text index support

Makes it easier to plan for a parallel query infra-structure

Features : PartitionsNative Partitioning memory overhead improvement — labs

Example Table with 8K partitionsCREATE TABLE `t1` (

`a` int(10) unsigned NOT NULL AUTO_INCREMENT,`b` varchar(1024) DEFAULT NULL, PRIMARY KEY (`a`)

) ENGINE=InnoDB DEFAULT CHARSET=latin1 PARTITION BY HASH (a) PARTITIONS 8192;

Memory overhead comparison

One open instance uses 49 % less memory (111 MB vs 218 MB)

Ten open instances take 90 % less memory (113 MB vs 1166 MB)

More work to be done – stay tuned!

Features : PartitionsImport/Export support

Importing a single partition# If the table doesn't already exist, create it

mysql> CREATE TABLE partitioned_table <same as the source>;

# Discard the tablespaces for the partitions to be restored

mysql> ALTER TABLE partitioned_table DISCARD PARTITION p1,p4 TABLESPACE;

# Copy the tablespace files

$ cp /path/to/backup/db-name/partitioned_table#P#p{1,4}.{ibd,cfg} /path/to/mysql-datadir/db-name/

# Import the tablespaces

mysql> ALTER TABLE partitioned_table IMPORT PARTITION p1,p4 TABLESPACE;

Features : Partitions

DMLIndex condition push down

Limited HANDLER support for partitionsCREATE TABLE t (a int, b int, KEY (a, b)) PARTITION BY HASH (b) PARTITIONS 2;

HANDLER t READ a = (1, 2);

HANDLER t READ a NEXT;

Features : Tablespace management – labs

TablespacesSQL syntax for explicit tablespace management

Replaces legacy --innodb-file-per-table usage

• CREATE TABLESPACE Logs ADD DATAFILE 'log01.ibd';• CREATE TABLE http_req(c1 varchar) TABLESPACE=Logs ;• ALTER TABLE some_table TABLESPACE=Logs;• DROP TABLESPACE Logs; -- must be empty

Features : Tablespace management – labs

TablespacesSQL syntax for explicit tablespace management

Replaces legacy --innodb-file-per-table usage

• CREATE TABLESPACE Logs ADD DATAFILE 'log01.ibd';• CREATE TABLE http_req(c1 varchar) TABLESPACE=Logs ;• ALTER TABLE some_table TABLESPACE=Logs;• DROP TABLESPACE Logs; -- must be empty

Features : Buffer Pool

Dynamic buffer pool size re-sizeDone in a separate thread

--innodb_buffer_pool_chunk_size – resize done in chunk size

Example:

SET GLOBAL innodb_buffer_pool_size=402653184;

Features : UNDO

UNDO Log Space ManagementRequires separate UNDO tablespaces to work

• --innodb-undo_log_truncate=on (default off)• --innodb-max_undo_log_size – default 1G• --innodb-purge-rseg-truncate-frequency – default 128 - advanced

Features : Larger Page Sizes - labs

32K and 64K Page SizesLonger records can be stored “in-page”

Better compression with the new transparent page compression

Features : Memcached Multiple Get - labs

MemcachedMultiple get

Gets around Memcached client protocol bottlenecks• Shorter query string• Fetch multiple keys in range• Extension to “traditional” Memcached

Features : GIS

Spatial indexImplemented as an R-Tree

Supports all MySQL geometric types

Currently only 2D supported

Supports transactions & MVCC

Uses predicate locking to avoid phantom reads

Features : GIS

R-TreeMulti-dimension spatial data search

Queries more like:• Find object “within”, “intersects” or “touches” another object• MySQL geometric types

• POINT, LINESTRING, POLYGON, MULTIPOINT,• MULTILINESTRING, MULTIPOLYGON, GEOMETRY

Features : Mutexes

Flexible mutexesMix and match mutex types in the code – build time option only

Can use futex on Linux instead of condition variables

Futex eliminates “thundering herd” problem

Not enabled by default, build with -DMUTEX_TYPE=”futex” from source

Features : Character Sets

Adds the MySQL character set gb18030 (Chinese encoding for Unicode)

Supports the China National Standard GB 18030 character set.

The new associated collations are gb18030_bin and gb18030_chinese_ci.

Features : Full Text Search

Support for external parser

For tokenizing the document and the query

Example:

CREATE TABLE t1 (

id INT AUTO_INCREMENT PRIMARY KEY,

doc CHAR(255), FULLTEXT INDEX (doc) WITH PARSER my_parser) ENGINE=InnoDB;

ALTER TABLE articles ADD FULLTEXT INDEX (body) WITH PARSER my_parser;

CREATE FULLTEXT INDEX ft_index ON articles(body) WITH PARSER my_parser;

Features : Sandisk/FusionIO Atomic Writes

No new configuration variables – may change in GASystem wide settingDisables the doublewrite buffer if the system tablespace is on NVMFSMore to come

Features : Transparent PageIO Compression

Proof of concept patch from FusionIO – currently Linux onlyRequires sparse file support : NVMFS, XFS, EXT4, ZFS & NTFSLinux 2.6.39+ added PUNCH HOLE supportCan co-exist with current ROW_FORMAT=COMPRESSED tablesWorks on all table types, including the system tablespace

Zip compression Tested and tried, works well enough

Complicates buffer pool code

Special page format required

No IO layer changes

Algorithm supported - Zlib

Can't compress system tablespace

Can't compress UNDO tablespace

Features : Zip vs Page IO compression

PageIO compression Requires OS/FS support

Simple

Works with all file types, system tablespaces

Potential fragmentation issues

NVMFS doesn't suffer from fragmentation

Adds to the cost of IO

Current algorithms are tuned to existing assumptions

Requires multi-threaded flushing

Easy to add new algorithms – Zlib, LZ4 etc.

Features : PageIO Compression Benchmark

FusionIO – 25G BP – maxid 50 Million 64 Requesters - Linkbench

Normal PageIO compression Current compression0

Size & Operations per/sec

Ops/sec

Configuration options--innodb-compression-algorithm := 0,1,2

Where: 0 – None, 1 – Zlib and 2 – LZ4

--innodb-compression-level

--innodb-compression-punch-hole := boolean

--innodb-read-async := boolean

--innodb-read-block-size := boolean

Page IO Compression

Features : Miscellaneous

Implement update_time for InnoDB tablesImprove select count(*) performance by using handler::records();Improve recovery, redo log tablespace meta data changes

Download & Blogs

http://labs.mysql.com

http://dev.mysql.com/downloads/mysql/

http://mysqlserverteam.com/

Thank You!

Marko Mäkelä - InnoDB

Sales

Transcript of Marko Mäkelä - InnoDB

John Russell InnoDB@Oracle

OSCON2004 MySQL Innodb Performance Optimization

InnoDB: Status, Architecture, and Latest Enhancementsassets.en.oreilly.com/1/event/56/InnoDB_ Status, Architecture, and... · InnoDB: Status, Architecture, and Latest ... InnoDB-related

Research Topics Optimization Group at University of Turku Marko M. Mäkelä University of Turku Department of Mathematics.

Switching from innodb to myiasm

Data Organization in InnoDB

MySQL 和 InnoDB 性能

InnoDB Performance and Usability Patches

Innodb Performance Tuning

InnoDB Plugin: Performance Features and Benchmarks MySQL Conference and Expo, 2010 10:50 am, April 15, 2010 John Russell InnoDB@Oracle Jimmy Yang InnoDB@Oracle.

MySQL 5.7: Focus on InnoDB

Falcon for InnoDB Users

Mhonde, Ngoni; Schreithofer, Nora; Corin, Kirsten; Mäkelä ...

InnoDB Magic

Innodb Plugin 1.0 Documentation

Zheng, Jianfang; Ala-Laurinaho, Juha; Sneck, Asko; Mäkelä ...

Falcon is not InnoDB

Undrop for InnoDB

Mäkelä, Eveliina; Escobedo, José Luis González; Lindblad ...

InnoDB Performance Optimization - Percona Performance Optimization Heikki Tuuri, ... InnoDB Indexing ... dirty reads but performance improvement is limited