Geir Høydalsvik, Sr. Director, MySQL Engineering Simon ... · PDF fileGeir...

Copyright © 2015, Oracle and/or its affiliates. All rights reserved. |

Whats New in MySQL 5.7

Geir Høydalsvik, Sr. Director, MySQL EngineeringSimon Mudd, Senior DBA, Booking.com


Safe Harbor Statement

The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

2


Agenda

First Geir will talk about Latest New “Features”

•Highlights of 5.7 Improvements and Additions

• New Server Strategy for Releases

• New MySQL Document Store

Next Simon will provide Booking’s Perspective

• Upgrading and Putting 5.7 into production

Oracle Confidential – Internal/Restricted/Highly Restricted 3


Benefit Highlights of MySQL 5.7

• Faster

– Server – quicker queries, more IO, more connections• InnoDB engine, Faster Connect/Disconnect, More Optimized Queries

– Replication – scales higher, faster slave processing• Parallel replications can be up to an order of magnitude faster

– GIS searching – quickly search geographic data • Native InnoDB Spatial indexes

– Transparent Page Compression• Leverages new advanced Operating File System capabilities

• Improves performance, saves space



Benefit Highlights of MySQL 5.7 (con’t)

Easier

• Easier Configurability – dynamic “online” changes

• More Flexible

– Native JSON support – datatype, binary storage format

– Virtual Columns/Functional indexes – index any data and find quickly

• Better Visibility– More instruments, Easy access to data using SYS schema

• More Secure

– By default and more options to secure the server



MySQL 5.7 Sysbench Benchmark: SQL Point Selects / sec

1,600,000 QPS3x Faster than MySQL 5.64x Faster than MySQL 5.5

MySQL 5.7 Sysbench Benchmark: Mixed OLTP Read Only3x Faster than MySQL 5.66x Faster than MySQL 5.5 Near 1M QPS

82% Faster than MySQL 5.6100K Connect / Sec

MySQL 5.7 Sysbench Benchmark: Connect / sec


MySQL 5.7: JSON Overview

• Native JSON data type

– Native internal binary format for efficient processing & storage

• Built-in JSON functions

– Allowing you to efficiently store, search, update, and manipulate Documents

• JSON Comparator– Allows for easy integration of Document data within your SQL queries

• Indexing of Documents using Generated Columns

– InnoDB supports indexes on both stored and virtual Generated Columns

– New expression analyzer automatically uses the best “functional” index available

7


• New Cost based Optimizer

– Configurable and tunable

– Better Join order due to condition filtering

– Better Query Plans

• New Hint Framework

– Easier to manage

– With support for additional new hints

• New Query Rewrite Plugin

• Generated Columns

• EXPLAIN for running thread

• Optimizer and Parser refactoring

– Cleanly separate parsing, optimizing, execution

• Improved JSON EXPLAIN

• InnoDB based internal temp tables

• SQL Standard compliant ONLY_FULL_GROUP_BY mode is ON by default

MySQL 5.7: Optimizer Improvements

8

Queries execute faster, while using less CPU and disk space!


SYS views make it easier to

- Monitor server, user, host

- Spot, diagnose, tune performance

- Find IO hot spots, Locking, Costly SQL statements

- Review schema, table, index stats

• SYS is similar to

- Oracle V$ catalog views

- Microsoft SQL DMVs (Dynamic MgmntViews)

Performance Schema additions

• Memory Instrumentation

• Statement Instrumentation

• Store Procedures

• Stored Functions

• Prepared Statements

• Transactions

MySQL 5.7: SYS and Performance Schema



• Replaced custom code with Boost.Geometry

– For spatial calculations

– For spatial analysis

– Enabling full OGC compliance

– We’re also Boost.Geometry contributors!

• InnoDB R-tree based spatial indexes

– Full ACID, MVCC, & transactional support

– Index records contain minimum bounding box

• GeoHash

• GeoJSON

• Helper functions such as ST_Distance_Sphere() and ST_MakeEnvelope()

MySQL 5.7: GIS Improvements

10


MySQL 5.7: InnoDB Improvements

• Most welcomed addition for many was

– Resize the InnoDB Buffer Pool online

• Others notable adds and enhancements

– Native Partitioning,

– Native Full-Text Search,

– Transparent Compression along with 32k and 64K Pages,

– User Defined Tablespaces

– Improved Cache Pre-loading

• And many more (InnoDB session will cover them all)

Many Additions - Highlights Include



• InnoDB Tablespace Encryption

• MySQL KeyRing

• AES 256 Encryption now the default

• Password rotation policies

• Deployment: enable secure unattended install by default

• Easier instance initialization and setup: mysqld –initialize

• New detection and support for systemd

• SSL

– Enabled by default

– Auto-detection of existing keys and certs

– Auto generation of keys and certs when needed

– New helper utility: mysql_ssl_rsa_setup

– New --require_secure_transport option to prevent insecure communications

– Added SSL support to binary log clients

• Extended Proxy User Support

– Added Built-in Authentication Plugins support for Proxy Users

– Allows multiple users to share a single set of managed privileges

MySQL 5.7: Security Improvements

12


• GTID enhancements

– On-line, phased deployment of GTIDs

– Binary logging on slave now optional

• Enhanced Semi-synchronous replication

– Write guaranteed to be received by slave before being observed by clients of the master

– Option to wait on Acks from multiple slaves

• Multi-Source Replication

– Consolidate updates from multiple Masters into one Slave

• Dynamic slave filters

• 8-10x Faster slave throughput

– Often removes slave as a bottleneck; keep pace with master with 8+ slave threads

– Option to preserve Commit order

– Automatic slave transaction retries

MySQL 5.7: Replication Improvements

13

0%

50%

100%

150%

200%

250%

1 8 24 48

SlaveThreads

Slavethroughputvs.96ThreadMaster


New Release Model – Accelerating Innovation

• Introducing the concept of Rapid Plugins

• A much requested feature

• Users who want more of a balance between

– Don’t break it

– Give me innovation sooner!

• Goals– Same Stability in the Core Server

– Adds Optional Server Plugins for New Functionality• New plugins come with the server distribution – but you choose whether to turn ON or OFF

• First Example MySQL Documents Store X Plugin



We <3 Schemaless

• MySQL as a Document Store (new!)

• All the existing features of MySQL

– Replication

– InnoDB

– Performance Schema

• With the addition of schemaless

– Documents using JSON

– Easy to program CRUD APIs

15


MySQL – a One Stop Shop

• Expensive to manage many data stores

• Better few databases – more flexibility

– Developers and DBAs• Can harness both NoSQL and SQL savvy

• Already know the MySQL product

– Less training – don’t need to learn many products

– Cross data store exchange – easier to move from from docs to tables etc.

– One connector/driver needed for apps

Combining Relational and Document Stores

16


Architecture

17

MySQL

Plugins

X Plugin Group Replication PluginCore

X ProtocolClassic Protocol

X Protocol33060

Classic Protocol3306

SQL API X Dev APICRUD Based

X and ClassicProtocols

MySQL ShellJavascript/Python


MySQL Document Store: What’s New?

• New Document APIs – X Dev APIs– SQL and NoSQL/Document CRUD APIs will cross all Connectors

• For BOTH Collections of Documents and Relational Tables

– Initial DMR M1 includes Node.js, Java and .NET

• New Interactive Shell – MySQL Shell– Javascript, Python, SQL modes

• New Server Features – MySQL Document Store– Native JSON datatype and storage

– Generated Columns with Indexing

– Over 20+ Native JSON functions

– New X Protocol

18


• Perspectives on MySQL 5.7- Why we follow the development versions- How we Upgrade- Our experience after 6 months GA

Simon Mudd, Senior DBA

19


About Booking.com

Booking.com is part of the Priceline Group

• World leader in booking accommodation online• Established in 1996 and based in Amsterdam– Over 855,000 properties in 220+ countries– 42 languages (website and customer service)– Supported by 174 offices in 70 countries

• We Use MySQL:– Thousands of servers: ~85% replicating– Over 110 masters: ~25 have 50+ slaves, ~8 have 100+ slaves

20


Why we tested MySQL 5.7 DMR releases

Testing of DMR releases allowed us to:

• Preview new features and give feedback

• Check performance meets expectations

• Catch unexpected behaviour early

Resources required to do this are not significant

21


What we look for when upgrading to a new major version?

Motivation for upgrades:• Longer uptime and more dynamic configuration (avoid mysqld restart)• Seamless upgrade path from previous version (via replication)• More features and flexible configuration• Better performance in replication, better concurrency and better stability

under load• Incorporation of features requested previously• Better visibility of what MySQL is doing

22


Results of our MySQL 5.7 DMR testing

• 2 years following the evolution of MySQL 5.7 DMR (5.7.1 to 5.7.8)• We provided feedback on several issues, e.g.

– SQL_MODE changes caused problems with replication (5.6 to 5.7)– ALTER, CREATE, DROP USER were not replication safe (5.6 to 5.7)– Unexpected crashes on startup in some early releases– Parallel Replication

• MySQL engineering responded to these issues– No longer an issue in the GA version– Avoided potential breakage while performing upgrades

23


Upgrading MySQL at booking.com

• Install new version into production as slaves (including DMR versions)– Adjust configuration and check behaviour– Provide real load and check behaviour– Upgrade intermediate masters to the new version– Switch write traffic to an intermediate master and promote to master– Use all new features

• Doing this allows careful control of the upgrade process allowing you to back out if necessary

• Downtime is limited to a few seconds as writes switch to new master

24


Moving MySQL 5.7 into production – observations

• The setup of a new environment is simpler and more secure– the new defaults require less cleanup

• The upgrade process for 5.7 is simpler than before– Just running mysql_upgrade is usually sufficient

• Configuration changes in /etc/my.cnf– Few required changes• log_syslog• innodb_buffer_pool_dump_pct• show_compatibility_56

– Pay attention to changes in default values between 5.6 and 5.7– We have had to add some extra settings

25


Moving MySQL 5.7 into production – observations (cont.)

• Grant management– Procedures need to be MySQL version aware

• new disable accounts vs old change password– New ALTER USER is replication safe, but can’t use on master until all servers run 5.7

• Improve monitoring• Collect 5.7 specific metrics from performance_schema / sys– Replication– Memory usage– Locks

26


MySQL 5.7 Current Deployment Status

• Production deployment of MySQL is ongoing

– ~900 servers running MySQL 5.7 on about 25 different clusters

– A few complete clusters now running 5.7 completely

• New deployments now use MySQL 5.7

27


Feedback on Production Usage

• No major problems• No crashes seen in the GA version (this is a good sign)• Performance and memory footprint under load seems much better• A few issues seen:– default_password_lifetime

– range_optimizer_max_mem_size

–GET_LOCK() change in behaviour

–Optimizer setting regression seen

28


Issues seen in Production (1)

default_password_lifetime:

• Initial period was 360 days

• We got locked out with boxes which had been running for 1 year!

• If used: requires new management procedures, easy to forget

• Oracle reverted default setting reverted to 0 in 5.7.11 so problem gone now

29



GET_LOCK() change in behaviour:

• Potential incompatible behaviour compared to MySQL 5.6 depending on

usage

• No backwards compatibility option

• Potentially requires all code to be reviewed prior to updates

• May not affect everyone, few people use GET_LOCK()?, but did affect us

30



range_optimizer_max_mem_size:

• New setting has different implicit default vs hard-coded value in 5.6

• May trigger the optimizer to revert point lookups in some queries to table

scans when used with large IN (a, b, …, z) list type expressions

• Oracle adjusted code/settings in 5.7.12 to more closely match 5.6

behaviour and be more memory efficient

31



• Optimizer setting regression– New optimizer_switch condition_fanout_filter used to determine the best

order in which tables should be joined– Looks good and sounds interesting, but there are cases where the current

implementation has to guess values and can get it wrong– Current workaround: disable this on systems which show the problem, globally or at

the session level (trickier to implement)– Future work

• Could use the new hinting infrastructure to allow hinting at the query level• Add information on optimizer usage to performance_schema

32


Conclusion

• Following MySQL 5.7 from DMR releases has been worthwhile– We hit a few bumps as might be expected– We received an excellent response to those issues we reported

• Some changes were reverted, others added

• The migration to production has been very smooth– The few issues we’ve seen have been special cases and easy to work around

• MySQL 5.7 is the best MySQL release yet– A lot of high quality work has gone into this release

• Recent announcement of optional new features in 5.7.12 looks very interesting and will require ongoing evaluation

33


Other Booking.com Sessions at Percona Live

• MySQL Scalability: Dimensions, Methods and Technologies

– Nicolai Plum |Ballroom C, Tuesday at 2:20 pm

• Bringing code to the data: from MySQL to RocksDB for high volume searches

– Ivan Kruglov | Ballroom F, Wednesday at 3:30 pm

• MySQL Parallel Replication: inventory, use-case and limitations– Jean-François Gagné | Ballroom A, Thursday 12:50 pm

34

Geir Høydalsvik, Sr. Director, MySQL Engineering Simon ... · PDF fileGeir...

Documents

Transcript of Geir Høydalsvik, Sr. Director, MySQL Engineering Simon ... · PDF fileGeir...