Upgrade to MySQL 5.7 and latest news planned for MySQL 8

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Upgrading to MySQL 5.7 and latest news from OOW Ted Wennmark [email protected]


Program Agenda

MySQL 5.7

MySQL 5.7 Upgrade

MySQL 8

1

2

2

3


Program Agenda

MySQL 5.7

MySQL 5.7 Upgrade

MySQL 8

1

2

3

3


MySQL 5.7 is GA!

4

Enhanced InnoDB: faster online & bulk load operations

Replication Improvements (incl. multi-source, multi-threaded slaves...)

New Optimizer Cost Model: greater user control & better query performance

Performance Schema Improvements

MySQL SYS Schema

Performance & Scalability Manageability

3 X Faster than MySQL 5.6

Improved Security: safer initialization, setup & management

Native JSON Support

And many more new features and enhancements. Learn more at: dev.mysql.com


MySQL 5.7 Sysbench Benchmark: SQL Point Selects 3x Faster than MySQL 5.6

4x Faster than MySQL 5.5 1,600,000 QPS

0

200 000

400 000

600 000

800 000

1 000 000

1 200 000

1 400 000

1 600 000

1 800 000

8 16 32 64 128 256 512 1 024

Qu

erie

s p

er S

eco

nd

Connections

MySQL 5.7: Sysbench OLTP Read Only (SQL Point Selects)

MySQL 5.7

MySQL 5.6

MySQL 5.5

Intel(R) Xeon(R) CPU E7-8890 v3 4 sockets x 18 cores-HT (144 CPU threads) 2.5 Ghz, 512GB RAM Linux kernel 3.16

5


MySQL 5.7 Sysbench Benchmark: OLTP Read Only 3x Faster than MySQL 5.6

6x Faster than MySQL 5.5

0

200 000

400 000

600 000

800 000

1 000 000

1 200 000

8 16 32 64 128 256 512 1 024

Qu

erie

s p

er S

eco

nd

Connections

MySQL 5.7: Sysbench OLTP Read Only

MySQL 5.7

MySQL 5.6

MySQL 5.5


~ 1,000,000 QPS

6


MySQL 5.7 Sysbench Benchmark: OLTP Read Write 1.5x Faster than MySQL 5.6

3x Faster than MySQL 5.5

0

100 000

200 000

300 000

400 000

500 000

600 000

700 000

8 16 32 64 128 256 512 1 024

Qu

erie

s p

er S

eco

nd

Connections

MySQL 5.7: Sysbench OLTP Read Write

MySQL 5.7

MySQL 5.6

MySQL 5.5


7


• Optimizer and Parser refactoring

– Readability, maintainability and stability

– Separate parsing, optimizing, execution stages

– Easier feature additions, with lessened risk

• New hint framework

– Easier to manage

– With support for additional new hints

• Improved JSON EXPLAIN

• EXPLAIN for running thread

• New Cost based Optimizer

– Easier to extend

– Configurable and tunable • mysql.server_cost and mysql.engine_cost tables

• API for where data resides: on disk or in cache

• InnoDB for internal temp tables

• Better ONLY_FULL_GROUP_BY mode

• Many specific new optimizations

• Generated Columns

MySQL 5.7: Optimizer Improvements

8

Queries execute faster, while using less CPU and disk space!


Optimizer Cost Model: Performance Improvements DBT-3 (Size Factor 10, CPU bound)

0

20

40

60

80

100

Q3 Q7 Q8 Q9 Q12

Exe

cuti

on

tim

e re

lati

ve to

5.6

(%)

5 out of 22 queries get a much improved query plan (others remain the same)

MySQL 5.6

MySQL 5.7

9


0

20

40

60

80

100

Q2 Q18

Exe

cuti

on

tim

e re

lati

ve to

5.6

(%)

CPU bound

5.6

5.7

Optimizer Cost Model: Performance Improvements DBT-3 (Size Factor 10)

2 out of 22 queries get a significantly improved query plan (others remain the same)

0

20

40

60

80

100

Q2 Q18

Exe

cuti

on

tim

e re

lati

ve to

5.6

(%)

Disk bound

5.6

5.7

10


MySQL 5.7: Query Rewrite Plugin

• New pre and post parse query rewrite APIs – Users can write their own plug-ins

• Provides a post-parse query plugin – Rewrite problematic queries without the need to make application changes

– Add hints

– Modify join order

–Many more …

• Improve problematic queries from ORMs, third party apps, etc

• Eliminates many legacy use cases for proxies

11


MySQL 5.7: Optimizer - Cost Info in JSON EXPLAIN

• Expanded JSON EXPLAIN – Now includes all available cost info

– Used for Visual Explain In MySQL Workbench

{ "query_block": { "select_id": 1, "cost_info": { "query_cost": "200.40" }, "table": { "table_name": "nicer_but_slower_film_list", "access_type": "ALL", "rows_examined_per_scan": 992, "rows_produced_per_join": 992, "filtered": 100, "cost_info": { "read_cost": "2.00", "eval_cost": "198.40", "prefix_cost": "200.40", "data_read_per_join": "852K" }, "used_columns": [ "FID", "title", "description", "category", "price", "length", "rating", "actors" ], ...

12


MySQL 5.7: JSON • Native JSON data type –Native internal binary format for efficient processing & storage

• Built-in JSON functions – Allowing you to efficiently store, search, update, and manipulate Documents

• JSON Comparator

– Allows for easy integration of Document data within your SQL queries

• Indexing of Documents using Generated Columns

– InnoDB supports indexes on both stored and virtual Generated Columns

– New expression analyzer automatically uses the best “functional” index available

• New inline syntax for easy SQL integration

13


MySQL 5.7: JSON and Text Datatype Comparison

# With feature column as JSON type SELECT DISTINCT feature->"$.type" as json_extract FROM features; +--------------+ | json_extract | +--------------+ | "Feature" | +--------------+ 1 row in set (1.25 sec)

Unindexed traversal of 206K documents

# With feature column as TEXT type SELECT DISTINCT feature->"$.type" as json_extract FROM features; +--------------+ | json_extract | +--------------+ | "Feature" | +--------------+ 1 row in set (12.85 sec)

Explanation: Binary format of JSON type is very efficient at searching. Storing as TEXT performs over 10x worse at traversal.

14


MySQL 5.7: Functional Indexes with JSON

15

ALTER TABLE features ADD feature_type VARCHAR(30) AS (JSON_UNQUOTE(feature->'$.type')); Query OK, 0 rows affected (0.01 sec) Records: 0 Duplicates: 0 Warnings: 0 ALTER TABLE features ADD INDEX (feature_type); Query OK, 0 rows affected (0.73 sec) Records: 0 Duplicates: 0 Warnings: 0 SELECT DISTINCT feature_type FROM features; +--------------+ | feature_type | +--------------+ | "Feature" | +--------------+ 1 row in set (0.06 sec)

From table scan on 206K documents to index scan on 206K materialized values

Meta data change only (FAST). Does not need to touch table..

Creates index only, does not touch row data.

Down from 1.25 sec to 0.06 sec

Create table features ( id int primary key, feature json )


Memory Instrumentation

• Aggregates statistics by

– Type of memory used (caches, internal buffers, …)

– Thread/account/user/host indirectly performing the memory operation

• Attributes include

– Memory used (bytes)

– Operation counts

– High/Low Water Marks

Statement Instrumentation

• Stored Procedures

• Stored Functions

• Prepared Statements

• Transactions

Additional Information

• Replication slave status

• MDL lock instrumentation

• Status and variables per thread

• Server stage tracking

• Track long running SQL

• Improved configuration and ease-of-use

• All while reducing total footprint and overhead

MySQL 5.7: Performance Schema

16


MySQL 5.7: SYS Schema Helper objects for DBAs, Developers and Operations staff

• Helps simplify DBA / Ops tasks

- Monitor server health, user, host statistics

- Spot, diagnose, and tune performance issues

• Easy to understand views with insights into

- IO hot spots, Locking, Costly SQL statements

- Schema, table and index statistics

• SYS is similar to - Oracle V$ catalog views

- Microsoft SQL DMVs (Dynamic Mgmnt Views)

17


• Replaced custom code with Boost.Geometry

– For spatial calculations

– For spatial analysis

– Enabling full OGC compliance

– We’re also Boost.Geometry contributors!

• InnoDB R-tree based spatial indexes

– Full ACID, MVCC, & transactional support

– Index records contain minimum bounding box

• GeoHash

• GeoJSON

• Helper functions such as ST_Distance_Sphere() and ST_MakeEnvelope()

MySQL 5.7: GIS Improvements

18


• Native Partitioning

– Eliminates previous limitations

– Eliminates resource usage problems

– Transportable tablespace support

• Native Full-Text Search

– Including full CJK support!

• Native Spatial Indexes

• Transparent page compression

• Support for 32K and 64K pages

– Use with transparent page compression for very high compression ratios

• General TABLESPACE support

– Store multiple tables in user defined shared tablespaces

• Support for MySQL Group Replication

– High priority transactions

• Improved support for cache preloading

– Load your hottest data loaded at startup

• Configurable fill-factor

– Allows for improvements in storage footprint

• Improved bulk-data load performance

• Resize the InnoDB Buffer Pool online

MySQL 5.7: InnoDB Improvements

19


MySQL 5.7: InnoDB Compression Thank you, SanDisk Fusion-io

• Transparent Page Level Compression

– Happens transparently in background threads

– Managed entirely within the IO layer

– Uses sparse file and "hole punching" support in OS kernels and File Systems

• Reduces IO

– Improves MySQL performance

– Improves storage efficiency

– Reduces write cycles, thus increasing SSD lifespan

• Applies to all InnoDB data, including the system tablespace and UNDO logs

20


• AES 256 Encryption now the default

• Password rotation policies

– Can be set globally, and at the user level

• Deployment: enable secure unattended install by default

– Random password set on install

– Remove anonymous accounts

– Deployment without test account, schema, demo files

• Easier instance initialization and setup: mysqld –initialize

• New detection and support for systemd

• SSL

– Enabled by default

– Auto-detection of existing keys and certs

– Auto generation of keys and certs when needed

– New helper utility: mysql_ssl_rsa_setup

– New --require_secure_transport option to prevent insecure communications

– Added SSL support to binary log clients

• Extended Proxy User Support

– Added Built-in Authentication Plugins support for Proxy Users

– Allows multiple users to share a single set of managed privileges

MySQL 5.7: Security Improvements

21


MySQL 5.7: Server-Side Statement Timeouts

Thank you Davi Arnaut!

• Server side statement timeouts

– Global for server, per session, or for individual SELECT statements

• Expanded to Windows and Solaris, restricted by removing USER option

SELECT /*+ MAX_EXECUTION_TIME(1000) */ * FROM my_table;

22


• GTID enhancements

– On-line, phased deployment of GTIDs

– Binary logging on slave now optional

• Enhanced Semi-synchronous replication

– Write guaranteed to be received by slave before being observed by clients of the master

– Option to wait on Acks from multiple slaves

• Multi-Source Replication

– Consolidate updates from multiple Masters into one Slave

• Dynamic slave filters

• 8-10x Faster slave throughput

– Often removes slave as a bottleneck; keep pace with master with 8+ slave threads

– Option to preserve Commit order

– Automatic slave transaction retries

MySQL 5.7: Replication Improvements

23

0%

50%

100%

150%

200%

250%

1 8 24 48

SlaveThreads

Slavethroughputvs.96ThreadMaster


• Multi-Source Replication – Consolidate updates from multiple

Masters into one Slave • Consolidated view of all shards

• More flexible topologies

• Centralized point for backups

– Compatible with Semi-Sync Replication & enhanced MTS

• Performance Schema tables for monitoring slave

• Online Operations: Dynamic Replication Filters, switch master

MySQL 5.7: Replication Improvements

Slave

24

Binlog

Master 1

Binlog

Master 2

…

…

Binlog

Master N

IO 1

Relay 1

Coordinator

W1 W2 … WX

IO 2

Relay 2

Coordinator

W1 W2 … WX

…

…

Coordinator

W1 W2 … WX

IO N

Relay N

Coordinator

W1 W2 … WX


MySQL Repos • Distributions

– Oracle, Red Hat, CentOS

– Fedora

– Ubuntu, Debian

– SUSE

• Official MySQL Docker Image from Oracle

• Coming Soon

– Preconfigured Containers

– Improved support for popular DevOps deployment tools

https://dev.mysql.com/downloads/repo

MySQL on GitHub • Git for MySQL Engineering

– Fast, flexible and great for a distributed team

– Great tooling

– Large and vibrant community

• GitHub for MySQL Community

– Easy and fast code availability to the community and to downstream projects

– Pull Requests

https://github.com/mysql

25



https://github.com/mysql


Program Agenda

MySQL 5.7

MySQL 5.7 Upgrade

MySQL 8

1

2

26

3


Why upgrade?

• Performance/Scalability improvements

• New features

• Stay on fully supported release

• Security improvements

27


Performance/Scalability

• MySQL Scales better: – MySQL 5.1 scales to ~4 cores

– MySQL 5.6 scales to ~36 cores

– MySQL 5.7 scales to ~64 cores

• Improvements in optimizer – Better query plans with MRR, BKA and more

–More work in storage engine with ICP

– Tunable cost model in 5.7

28

https://www.mysql.com/why-mysql/benchmarks/


Support Life Cycle

Release GA Date Premier Support End Extended Support End Sustaining Support End

MySQL 5.0 Oct 2005 Dec 2011 N/A Indefinite

MySQL 5.1 Dec 2008 Dec 2013 N/A Indefinite

MySQL 5.5 Dec 2010 Dec 2015 Dec 2018 Indefinite

MySQL 5.6 Feb 2013 Feb 2018 Feb 2021 Indefinite

MySQL 5.7 Oct 2015 Oct 2020 Oct 2023 Indefinite

29

http://www.oracle.com/us/support/library/lifetime-support-technology-069183.pdf


MySQL Security

• SSL enabled by default

• Force root password during install

• Stronger passwords

• MySQL Enterprise Edition:

–MySQL Audit

–MySQL Monitor/Backup

–MySQL Firewall

–MySQL Encryption

30


MySQL Upgrade – Supported Paths

• All minor upgrades like 5.7.9 to 5.7.10 or doing a jump like 5.7.9 to 5.7.11 is supported.

• Upgrading one major (5.6 - 5.7) release level is supported. It’s recommended upgrading to the latest 5.6 release before moving to 5.7.

• Doing long-jumps (5.1 – 5.7) is supported if you upgrade one release level at a time.

• Direct upgrades that skip a release level (for example, upgrading directly from MySQL 5.5 to 5.7) is not recommended.

31


MySQL Upgrade – Before you start!

• Review the “Release Notes” for target MySQL version

• Review the manual page on upgrade for this MySQL version, take special care of sections covering:

– “Server and Status Variables and Options Added, Deprecated, or Removed …”

– “Checking Whether Tables or Indexes Must Be Rebuilt”

– “Changes Affecting Upgrades to MySQL 5.N”

• Create a plan for your upgrade/downgrade path

• Create a backup of your database

32

http://dev.mysql.com/doc/refman/5.7/en/upgrading.html


MySQL Upgrade – In-place Upgrade

1. Do all necessary preparations/plans as explained earlier

2. Do a graceful shutdown of MySQL (innodb_fast_shutdown=0)

3. Do upgrade, replace the old binaries with the new ones

4. Start new MySQL Server using new binaries

5. Run mysql_upgrade binary to upgrade internal mysql repository

6. Done!

34

http://mysqlserverteam.com/upgrading-directly-from-mysql-5-0-to-5-7-using-an-in-place-upgrade/ http://dev.mysql.com/doc/refman/5.7/en/upgrading.html#upgrade-procedure-inplace


MySQL Upgrade – Logical Upgrade


2. Do a dump of only the user databases/schemas (skipping the mysql system schema, remember stored procedures, views and functions)

3. Do a clean installation of new MySQL version (remember to migrate important my.cnf options)

4. Load the dump file into the new MySQL server instance

5. Done!

35

http://dev.mysql.com/doc/refman/5.7/en/upgrading.html#upgrade-procedure-logical http://mysqlserverteam.com/upgrading-directly-from-mysql-5-0-to-5-7-with-mysqldump/


MySQL Upgrade – Using Replication “simplified”


2. Restore backup from production (5.6) database to new slave server (5.7)

3. Start replication from master to slave

4. Once slave server has caught up with master

5. Change application to connect to slave (5.7)

6. Done!

36

MySQL 5.6 MySQL 5.7

Application

Replication

http://dev.mysql.com/doc/refman/5.7/en/replication.html

http://dev.mysql.com/doc/refman/5.7/en/replication.html


MySQL Upgrade – Regression Problems

• Explain is you friend! – Optimizer_switch can help get back old query plans for slow queries

• Monitor recourses/MySQL before/after to spot changes – Without proper monitoring you have little chance to find root cause of problem

• Did you change more than just MySQL version?

– Normally we see people refresh HW/OS at the same time with creates many possible regression scenarios

• Use PERFORMANCE_SCHEMA and SYS schema to spot problems.

37


Upgrading from MySQL 5.1 or 5.5 to MySQL 5.7? News in MySQL 5.6

• Need version 5.1.20 of Connector/J (due to retire of “SET OPTION”)

• InnoDB “Files per table” now default

• The Query Cache is disabled by default

• 8 new reserved key words have been defined (“get” and “partition”)

• New data format of TIMESTAMP and DATETIME (microseconds)

• Password on command line causes warnings (problematic for scripts)

• New optimizer features may cause different query plans

• Old password are not allowed by default

38

https://dev.mysql.com/doc/relnotes/mysql/5.6/en/


New Servers Default’s in MySQL 5.7

• innodb_strict_mode is “ON”

• InnoDB file format is now Barracuda

• sql_mode option “NO_AUTO_CREATE_USER, STRICT_TRANS_TABLES, ERROR_FOR_DIVISION_BY_ZERO, NO_ZERO_DATE, NO_ZERO_IN_DATE” is now default

• More changes to optimizer_switch, may cause changed query plans! – Use explain to identify problem and set optimizer_switch to solve problem

• MySQL accounts expire after 360 days. The default-password-lifetime is 360

• … and many more, see link below for more details

39

http://mysqlserverteam.com/improved-server-defaults-in-5-7/


Removed Features in MySQL 5.7

• Old password are not allowed in MySQL 5.7

• Remove IGNORE for ALTER TABLE in 5.7, WL#7395

• Remove of INSERT DELAYED, WL#6073

• Remove YEAR(2) support, WL#6263

• Deprecate Unique Option Prefixes, WL#6978

• Deprecate skip-innodb in 5.6 and remove it in 5.7, WL#7976

• For complete list: http://mysqlserverteam.com/removal-and-deprecation-in-mysql-5-7/

40


MySQL 5.7 – Installation changes

• mysql_install_db is removed – Replaced by mysqld --initialize

• MySQL is more Secure by default: – TLS/SSL enabled by default and for some packages keys are created by default

–No creation of test database

– No creation of anonymous users

–Only one root user and password is set at installation time

– All user accounts will automatically have their passwords rotated after 360 days

–New mysql_ssl_rsa_setup tool

• Use mysqld --initialize-insecure to get old behavior

41


Program Agenda

MySQL 5.7

MySQL 5.7 Upgrade

MySQL 8

1

2

42

3


• Transactional (InnoDB) Data Dictionary

• SQL roles

• INVISIBLE indexes (ALTER TABLE Country ALTER INDEX c INVISIBLE)

• Persistent runtime configuration changes (SET PERSIST)

• MySQL privilege tables are now InnoDB

• Auto-increment counter value will now persist across server restarts!

• Indexing of PS tables

• UTF8MB4 as the default character set

MySQL 8 (DM1)

43

http://dev.mysql.com/doc/relnotes/mysql/8.0/en/news-8-0-0.html

http://mysqlserverteam.com/the-mysql-8-0-0-milestone-release-is-available/

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 44

MySQL 8.0: Common Table Expressions CTE

• “With queries”

• Both Recursive and Non-Recursive Forms

• Simplifies writing complex SQL: WITH t1 AS (SELECT * FROM tblA WHERE a=‘b’)

SELECT * FROM t1;

Feature Request from Developers

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | 45

MySQL 8.0: Invisible Indexes

• Indexes are “hidden” to the MySQL Optimizer – Not the same as “disabled indexes”

– Contents are fully up to date and maintained by DML

• Two use cases:

– Soft Delete (Recycle Bin)

– Staged Rollout

Feature Request from DBAs


MySQL 8.0: Security Roles

• Fully Function, Flexible, Properly Architected Roles

• Create and Drop Roles, Grant to Roles

• Grant Roles to Roles, Grant Roles to Users

• Limit Hosts that can use roles, Define Default Roles

• Decide what roles are applicable during a session

• And even visualize Roles with SQL function ROLES_GRAPHML()


• New! Source code documented with Doxygen

• New! Plugin Infrastructure!

• Expanded GIS Support

• Expanded Query Hints Support

• Improved Scan Query Performance

• Improved BLOB Storage

• Improved Memcached Interface

• Scalability Improvements

• Parser Refactoring

• New! Document Store

• Improvements to Temporary Tables

• C++11 and Toolchain Improvements

• Replication Applier Thread Progress Reports

• GTID_PURGED always settable

47

MySQL 8.0: All this and more….


MySQL: Additional Info

• http://mysqlserverteam.com/whats-new-in-mysql-5-7-first-release-candidate/

• http://mysqlserverteam.com/json-labs-release-overview/

• http://mysqlserverteam.com/category/performance/optimizer/

• http://mysqlserverteam.com/category/innodb/

• http://mysqlserverteam.com/category/mysql/performance-schema/

• http://mysqlserverteam.com/category/gis/

• http://mysqlserverteam.com/category/full-text-search/

• http://mysqlserverteam.com/category/docstore/

• http://dev.mysql.com/doc/refman/5.7/en/upgrading.html

48

http://mysqlserverteam.com/whats-new-in-mysql-5-7-first-release-candidate/

















http://mysqlserverteam.com/json-labs-release-overview/







http://mysqlserverteam.com/category/performance/optimizer/

http://mysqlserverteam.com/category/innodb/

http://mysqlserverteam.com/category/mysql/performance-schema/



http://mysqlserverteam.com/category/gis/

http://mysqlserverteam.com/category/full-text-search/





http://mysqlserverteam.com/category/docstore/

http://dev.mysql.com/doc/refman/5.7/en/upgrading.html


Thank you!

Upgrade to MySQL 5.7 and latest news planned for MySQL 8

Technology

Transcript of Upgrade to MySQL 5.7 and latest news planned for MySQL 8