Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL 8.0: Atomic DDLs – Implementation and Impact
Ståle Deraas, Senior Development ManagerOracle, MySQL26 Sept 2017
Copyright © 2017, Oracle and/or its affiliates. All rights reserved.
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
2
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Agenda
Requirements for a Data Dictionary Storage Engine
Requirements for an SE implementing Atomic DDLs
Effects of new INFORMATION_SCHEMA in MySQL 8.0
1
2
3
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Requirements for a Data Dictionary Storage Engine
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Data Dictionary before MySQL 8.0
5
Data Dictionary
Files
FRM TRG OPT
System Tables (mysql.)
user procevents
InnoDB System Tables
MyISAM
File system
InnoDB
SQL
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Transactional Data Dictionary in MySQL 8.0
6
Data Dictionary
DD SESQL DD TableDD TableDD Table
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Requirements for Data Dictionary Storage Engine
• Some special care need to be taken at initialize/bootstrap
• Must support transactions
• Must support “attachable transactions”
– To open a table, the DD SE needs to open DD tables
• In MySQL 8.0 InnoDB has dual role
– MySQL Data Dictionary Storage Engine (DD SE)
– Storage Engine that supports Atomic DDL
7
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
The Data Dictionary Storage Engine API
• An addition to the Storage Engine API
• Not strictly separated from SE API
• Some examples of functionality that the DD SE must implement:
- Initialize dictionary support - Initiate recovery of dictionary transactions - Reset of the DD SE internal cache of meta data
8
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
MySQL Server
Data Dictionary Storage Engine
9
Query Executor
Optimizer
StorageEngine
SQLstatement
ClientParser
Result DataDictionary
DD StorageEngine
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Example - DROP SCHEMA at high level
10
MySQL 5.7
• Delete tables
– Metadata, TRN/TRG/FRM files
– Data, InnoDB tables
• Delete stored programs– Metadata, rows in MyISAM (non-
transactional)
• Delete schema
– Metadata, DB.OPT file
Mix of filesystem, non-transactional/transactional storage and multiple commits
MySQL 8.0
• Delete tables
– Metadata, rows in InnoDB
– Data, InnoDB tables
• Delete stored programs– Metadata, rows in InnoDB
• Delete schema
– Metadata, rows in InnoDB
Updates to transactional storage, one commit
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Requirements for an SE implementing Atomic DDLs
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Why Atomic DDL support in MySQL 8.0
• Ensure we have a consistent state after a DDL operation
– Prevent slave drift
– Prevent internal inconsistencies
• At last possible due to transactional storage of meta data
• Enables implemention of crash-safe DDL
• Enables addressing longstanding issues
12
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
User visible changes to Atomic TABLE DDL in MySQL 8.0• DROP TABLES , all tables dropped or none
• DROP SCHEMA, all entities in the schema are dropped, or none
• Note that atomic DDL statements will be rolled back or committed even in case of crash, e.g. RENAME TABLES
• CREATE TABLE would be successfully committed or rolled back (no orphan ibd left)
• TRUNCATE TABLE (including InnoDB tables with FTS AUX tables) would be successfully committed or rolled back
• RENAME TABLES, all or none
• ALTER TABLE successful or not done
13
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Storing SE private data (prerequisite)
• An SE which will support atomic DDL can store private data in SE private columns of Data Dictionary tables
• InnoDB has chosen to store key-value pairs in the SE private data
• Examples: index root page number, InnoDB table ID, tablespace ID etc.
14
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
How do we implement Atomic and Crash-Safe TABLE DDL
• At high level, create a single atomic transaction:
– Do updates to the Data Dictionary (DD SE)
– Do necessary changes in the Storage Engine
– Do necessary writes to binary log
– Since it is a single transaction, all updates to Data Dictionary can be rolled back and are crash safe
15
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
How do we implement Atomic and Crash-Safe TABLE DDL
• At more detailed level:
– Make sure there are no intermediate commits on SQL layer during DDL
– SE methods should have no intermediate commits, and register as part of trx
– SE can do redo/rollback of DDL , implemented with DDL_LOG in InnoDB
– SQL layer will invoke a post_ddl() hook, so SE can do post commit/rollback work
– As part of the transaction, write to binary log
– Introduced the SE property HTON_SUPPORTS_ATOMIC_DDL, that can be tested for in the source code
16
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
InnoDB: Features in SE for Atomic and Crash-Safe TABLE DDL• DDL_LOG
– Protected table in mysql tablespace, so no DDL and no user DML allowed
– Add trx Ids to all entries, which will be deleted by post_ddl() hook
– Changes to DDL_LOG are persisted ASAP. Will prevent data files updates and no flushing of DDL_LOG entries (exempted from innodb_flush_log_at_trx_commitcontrol for delayed flush)
• Issue with physical files and index tree:
– Before MySQL 8.0, could leave “orphan” InnoDB ibd files, “orphan” index trees/pages
– Orphan files/trees cause problems for subsequent DDLs and waste tablespace
– With MySQL 8.0, DDLs “crash safe”, no ibd files/index trees orphaned if no commit
– At commit, the old ibd files/index trees will be dropped
17
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - CREATE TABLE, SQL Perspective
18
MySQL 5.7
• At SQL layer: create table obj
• Store obj in DD tables, commit
• «Open» table, construct all internal structures
• Call to SE handler::create(name,...)
MySQL 8.0
• At SQL layer: create table obj
• Call to SE, adding SE private data
• Store obj in DD tables *
• «Open» table, construct all internal structures
• Call to SE handler::create(name,...), which can update SE private data
• Store obj in DD tables *
* = commit for non-atomic SE
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - CREATE TABLE, SQL Perspective CONT
19
MySQL 5.7
• SE creates physical files, its own artifacts such as InnoDBtablespaces. InnoDB stores data in own system tables.
• Write statement to binary log
MySQL 8.0
• Write statement to binary log
• Commit/Rollback, post_ddl() hook
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - CREATE TABLE, SE perspective
20
MySQL 5.7
• Call to SE handler::create(name,.)
• SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees
• SE starts its own transaction, insert the new table/indexes'smetadata to InnoDB own System tables
MySQL 8.0
• Call to SE handler::create(name,...)
• SE creates the physical tablespaces/files(file-per-table) or Cluster index tree and other index trees
• SE logs above physical file and indexes creation
• Note: No separate SE transaction, all info for new tablespace/indexes are passed back to server with DD objects
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - CREATE TABLE, SE perspective cont
21
MySQL 5.7
• Note: If crash happens before above trx commit, there could be orphan files or index trees left
MySQL 8.0
• SQL layer commits or rollback
• Call to SE post_ddl(). If rollback, the post_ddl() physically delete the tablespace/ibd (file-per-table) and drop the index trees for the table
• Note: If crash, SE will recover info in DDL_LOG to delete the ibd/index trees
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - DROP TABLES, SQL Perspective
22
MySQL 5.7
• For each table in table list
– Call SE handler:ha_delete_table() and if error do appropriate action
– Remove for Data Dictionary and commit
• Write up to 3 artifical DROP TABLES to binary log for dropped tables (temp tables *2 + all base tables)
MySQL 8.0
– For each table in table list• Check for 5 types of tables
– Err out if non-existing tables and no IF EXISTS clause
– Handle tables in non-atomic DDL SE• Delete Table in SE, and del in DD, commit
– Handle tables in atomic DDL SE• Delete Table in SE, and del in DD, no commit
– Write DROP TABLES statment to binary log incl all table (in «no GTID» mode)
– Commit/rollback , post_ddl() hook• Concurrent DDL on table blocked
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - DROP TABLES, SQL Perspective cont
23
MySQL 5.7 MySQL 8.0
– If GTID mode, and write to binlog postponed, write DROP TABLES stmt with all tables we dropped to binlog • To handle GTID mode from older servers
– For each temp table in non-transactional SE• Drop table, call SE handler::delete_table()
–Write DROP TEMPORARY TABLES for all temp tables above to binary log
– For each temp table in transactional SE• Drop table, call SE handler::delete_table()
– Write DROP TEMPORARY TABLES for all temp tables above to binary log
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - DROP TABLES, SE perspective
24
MySQL 5.7
• Call to SE handler:ha_delete_table()
• InnoDB starts its own transaction and delete metadata from the InnoDB own system tables
• InnoDB commit trx
• InnoDB deletes physical files/ibd(in file-per-table case)
• Note: If crash happens after InnoDB commit trx, there could be orphaned files/ibd or trees
MySQL 8.0
• Call to SE handler:ha_delete_table()
• InnoDB only logs actions to delete ibdfile or index trees in DDL_LOG, so no physical action (No InnoDB system tables needs to be updated)
• Back to SQL layer to commit - Delete metadata in DD system tables and commit
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL: Use Case - DROP TABLES, SE perspective cont
25
MySQL 5.7 MySQL 8.0
• SQL layer calls post_ddl(). If commit, InnoDB physically deletes ibd and drop index trees. If rollback, delete entries in DDL_LOG, files and trees left as they are
• Note: If crash and DROP trxcommitted, recover info from DDL_log, and delete files/trees during crash recovery
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
How do we implement Atomic and Crash-Safe non-TABLE DDL
• At slightly detailed level (this is all at SQL layer):
– Make sure there are no intermediate commits during DDL
– Write to binary log as part of the DDL transaction
– Ensure caches for Data Dictionary, routines, events and UDFs are consistent with DDL status
– Ensure user visible behaviour is atomic
– NOTE: Behaviour of DROP VIEW changes, for list of views• Pre-MySQL 8.0: Giving an error for non-existent views and removing existing views
• MySQL 8.0: Giving an error for non-existent views
26
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL Non-TABLE: Use case - CREATE ROUTINE
27
MySQL 5.7
• Create routine obj
• Store obj in DD tables, commit
• For SF, find VIEWs usage and update view cols metadata for each view. Commit for each view.
• Invalidate cache
• Write stored routine statement to binary log
MySQL 8.0
• Create routine obj
• Store obj in DD tables
• For SF, find VIEWs usage and update view cols metadata for each view
• Write stored routine event to binary log
• Commit and invalidate cache, or on error, rollback and give msg
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL for user management: Overview
28
MySQL 5.7
• Partial execution is allowed
– Make persistent changes to ACL tables/caches
– Throw error
• Error propagation to binary log– Required for HA setup
– Other nodes expect same error while executing the statement
MySQL 8.0
• Execution is atomic• A statement either fully succeeds or
is rolled back
• No error propagation to binary log• Only a successful statment makes it
way to binary log
• Simpler handling in HA setup
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL for user management: Use Case - CREATE USER
29
MySQL 5.7
• Result
– [email protected] is created
– Error is thrown for [email protected] for invalid authentication plugin
• Binary log– Statement is written to binary log
with password transformation
– Expected error for [email protected] is written
MySQL 8.0
• Result• Error because of invalid
authentication plugin for [email protected]
• [email protected] is not created
• Binary log• No statement is written to binary log
CREATE USER `foo`@`mysql.com` IDENTIFIED BY ‘haha’,`bar`@`mysql.com` IDENTIFIED WITH ‘non_existing_plugin’ BY ‘hoho’;
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL for user management: Use Case - GRANT
30
MySQL 5.7
• Result
– [email protected] get ability to SELECT from all tables/views.
– Error is thrown for non existing user
• Binary log– Statement is written to binary log
– Expected error for `non_existing`@`mysql.com`
MySQL 8.0
• Result• Error because of invalid user
• [email protected] is not granted SELECT
• Binary log• No statement is written to binary log
GRANT SELECT ON *.* TO `foo`@`mysql.com`, `[email protected]`;
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Atomic DDL for user management: Note that...
31
• IF EXISTS extensions (ALTER USER | DROP USER)
– Will continue to ignore errors related to non-existing authorization IDs
– All other errors will cause statement rollback
• IF NOT EXISTS extensions ( CREATE USER | CREATE ROLE)– Will continue to ignore errors related to existing authorization IDs
– All other errors will cause statement rollback
• New MDL type to protect access to in-memory caches for user management
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Effects of new INFORMATION_SCHEMA in MySQL 8.0
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
NEW INFORMATION_SCHEMA in MySQL 8.0
33
Uniform, simpler implemention makes it a lot faster
MySQL Client
I_S Query Results
MySQL Server
Optimizer prepares
execution plan.
Executor reads metadata from
data dictionary tables.
InnoDB storage engine
Return rows to user.
INFORMATION_SCHEMA in 8.0
MySQL Client
I_S Query Results
MySQL Server
Create temporary table.
Heuristic optimization.
Read metadata from File system or
from MyISAM/InnoDB engine.
.
TEMP TABLE
Return rows to user.
INFORMATION_SCHEMA in 5.7
File system / MyISAM
/ InnoDB engine
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION_SCHEMA Performance and Scalability
• Typically 30X performance improvements over MySQL 5.7
• More than 100X for some queries like: List all InnoDB table columns
34
I_S queries scale, both with database size and query load
0 20 40 60 80 100 120 140 160
List all InnoDB tables columns 5k tables
List all InnoDB tables columns 10k tables
MySQL 8.0
MySQL 5.7
Time in Seconds (Lower is better)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. | 35
100 schemas times 50 tables (5000 tables)
INFORMATION_SCHEMA Performance
0 0.5 1 1.5 2 2.5 3 3.5 4
Count All Schemas
Schema aggregate size stats
All Dynamic Table Info
All Static Table Info
Auto Increments Near Limit
Count All Columns
Count All Indexes
MySQL 8.0
MySQL 5.7
Time in Seconds (Lower is better)
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION_SCHEMA in MySQL 8.0, community feedback
• In MySQL 8.0 DMRs : Option “information_schema_stats” introduced
– TABLES.TABLE_ROWS, TABLES.DATA_FREE, …• Handled by opening table and retrieving data from SE in MySQL 5.7 – expensive!!!
• Bugreport: Nicolai Plum bug#83957 and Peter Brawley bug#87548
• Feedback from several MySQL ACEs, that found inital solution confusing
• For MySQL 8.0.3 RC1 We have created an improved solution
36
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
INFORMATION_SCHEMA in MySQL 8.0.3
• New option: information_schema_stats_expiry
– Default 24h
– Cache dynamic values in mysql.index_stats and mysql.table_stats
– Set to 0, always fetch data from SE and no «stats» will be stored
– ANALYZE will update mysql.index_stats and mysql.table_stats
• Improved performance fetching dynamic data from SE (InnoDB)– In 8.0.1 DMR, typically 2-3 times faster than 5.7
– In 8.0.3 RC1, typically 4-5 times faster than 5.7
• Option “information_schema_stats” removed
37
Copyright © 2017, Oracle and/or its affiliates. All rights reserved. |
Try for yourself!
• Downloadable 8.0.3 RC1
– dev.mysql.com
• Enjoy and give us your feedback!
• Thank you for listening
• http://mysqlserverteam.com
• For details on atomic DDL:
– WL#7743, WL#9173, WL#9045, WL#9175, WL#9045, WL#9536
38
Top Related