TR Nimble MySQL Best Practices Guide-v3 · PDF fileBEST PRACTICES GUIDE: ... The purpose of...

B E S T P R A C T I C E S G U I D E : N IMB LE S TOR AGE F OR M YS Q L 5 . 6 1

BEST PRACTICES GUIDE

Nimble Storage for MySQL 5.6 with InnoDB on Oracle Linux & RHEL 6


Document Revision

Table 1Table 1Table 1Table 1.

Date Revision Description

1/20/2014 1.0 Initial Draft

3/12/2014 1.1 Revised iSCSI Setting

6/2/2014 1.2 Logos added for Oracle and MySQL

11/17/2014 1.3 Updated iSCSI & Multipath

THIS TECHNICAL TIP IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN

TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS,

WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

Nimble Storage: All rights reserved. Reproduction of this material in any manner whatsoever

without the express written permission of Nimble is strictly prohibited.


Table of Contents

Introduction ................................................................................................................................................................................. 4

Audience ...................................................................................................................................................................................... 4

Scope ........................................................................................................................................................................................... 4

Nimble Storage Features .......................................................................................................................................................... 4

MySQL Database ........................................................................................................................................................................ 5

MySQL Enterprise Edition ......................................................................................................................................................... 6

MySQL Database 5.6 with Nimble Storage ............................................................................................................................ 6

Performance Settings ............................................................................................................................................................... 6

Nimble Recommended Settings .............................................................................................................................................. 7

Nimble Array OS ............................................................................................................................................. 7

Linux Operating System Settings ................................................................................................................... 7

Creating Nimble Volumes for MySQL Database 5.6 ...........................................................................................................10

MySQL 5.6 on EXT file system ................................................................................................................................................11

InnoDB Settings for MySQL 5.6 .............................................................................................................................................12

Using Snapshot and Zero-Copy Clones Features ................................................................................................................12

Using Nimble Storage Snapshot and Zero-Copy Clone for MySQL Backup ....................................................................13

Nimble Storage Snapshot for Backup ..................................................................................................................................13

Using Nimble Schedule ................................................................................................................................ 13

Using Nimble Command Line Interface (CLI) ............................................................................................... 17

Nimble Storage Snapshot for Clones ....................................................................................................................................17


Introduction

The purpose of this technical white paper is to describe the best practices for implementing MySQL Database

5.6 with InnoDB Storage Engine on Nimble Storage running on Oracle Linux 6 and Red Hat 6 operating system.

MySQL Cluster is not covered in this paper.

Audience

This guide is intended for MySQL database solution architects, storage engineers, system administrators and IT

managers who analyze, design and maintain a robust database environment on Nimble Storage. It is assumed

that the reader has a working knowledge of iSCSI SAN network design, and basic Nimble Storage operations.

Knowledge of Oracle Linux and Red Hat operating system, MySQL Database 5.6, and InnoDB Storage Engine

is also required.

Scope

During the design phase for a new MySQL 5.6 database implementation, DBAs and Storage Administrators

often times work together to come up with the best storage needs. They have to consider many storage

configuration options to facilitate high performance and high availability. In order to protect data against failures

of disk drives, host bus adapters (HBAs), and switches, they need to consider using different RAID levels and

multiple paths. When you have different RAID levels come into play for performance, TCO tends to increase as

well. For example, in order to sustain a certain number of IOPS with low latency for an OLTP workload, DBAs

would require a certain number of 15K disk drives with RAID 10. The higher the number of required IOPS, the

more 15K drives are needed. The reason is because mechanical disk drives have seek times and transfer rate,

therefore, you would need more of them to handle the required IOPS with acceptable latency. This will increase

the TCO tremendously over time. Not to mention that if the database is small in capacity but the required IOPS

is high, you would end up with a lot of wasted space in your SAN.

This white paper explains the Nimble technology and how it can lower the TCO of your Oracle environment and

still achieve the performance required. This paper also discusses the best practices for implementing MySQL

Database 5.6 with InnoDB Storage Engine on Nimble Storage.

Nimble Storage Features

Cache Accelerated Sequential Layout (CASL™)

Nimble Storage arrays are the industry’s first flash-optimized storage designed from the ground up to

maximize efficiency. CASL accelerates applications by using flash as a read cache coupled with a

write-optimized data layout. It offers high performance and capacity savings, integrated data protection,

and easy lifecycle management.

Flash-Based Dynamic Cache


Accelerate access to application data by caching a copy of active “hot” data and metatdata in flash for

reads. Customers benefit from high read throughput and low latency.

Write-Optimized Data Layout

Data written by a host is first aggregated or coalesced, then written sequentially as a full stripe with

checksum and RAID parity information to a pool of disk; CASL’s sweeping process also consolidates

freed up disk space for future writes. Customers benefit from fast sub-millisecond writes and very

efficient disk utilization

Inline Universal Compression

Compress all data inline before storing using an efficient variable-block compression algorithm. Store

30 to 75 percent more data with no added latency. Customers gain much more usable disk capacity

with zero performance impact.

Instantaneous Point-in-Time Snapshots

Take point-in-time copies, which do not require data to be copied on future changes (redirect-on-write).

Fast restores without copying data. Customers benefit from a single, simple storage solution for primary

and secondary data, frequent and instant backups, fast restores and significant capacity savings.

Efficient Integrated Replication

Maintain a copy of data on a secondary system by only replicating compressed changed data on a set

schedule. Reduce bandwidth costs for WAN replication and deploy a disaster recovery solution that is

affordable and easy to manage.

Zero-Copy Clones

Instantly create full functioning copies or clones of volumes. Customers get great space efficient and

performance on cloned volumes, making them ideal for test, development, and staging Oracle

databases.

MySQL Database

MySQL is the world’s most popular open source database for cost-effectively delivering reliable, high-

performance and scalable e-commerce, online transaction processing (OLTP), and embedded database

applications. It is a fully integrated transaction-safe, ACID compliant database with full commit, rollback, crash

recovery and row level locking capabilities. MySQL delivers the ease of use, scalability, and performance, as

well as a full suite of database drivers and visual tools to help developers and DBAs build and manage their

MySQL applications. Many of the world's most trafficked websites like Facebook, Google, ticketmaster, and


eBay rely on MySQL for their business critical applications. MySQL Database 5.6 includes significant

performance and high availability improvements enabling the next generation of web, embedded and Cloud

applications.

MySQL Enterprise Edition

MySQL Enterprise Edition includes the most comprehensive set of advanced features, management tools and

technical support to achieve the highest levels of MySQL scalability, security, reliability, and uptime. It reduces

the risk, cost, and complexity in developing, deploying, and managing business-critical MySQL applications. It

includes:

• MySQL DatabaseMySQL DatabaseMySQL DatabaseMySQL Database to power the most demanding Web, E-commerce and OLTP applications.

• MySQL Query AnalyzerMySQL Query AnalyzerMySQL Query AnalyzerMySQL Query Analyzer to optimize performance by visualizing query activity, pinpointing expensive queries, and

fixing problem SQL code

• MySQL Enterprise MonitorMySQL Enterprise MonitorMySQL Enterprise MonitorMySQL Enterprise Monitor to monitor MySQL performance, sessions, connections, and more

• MySQL Enterprise BackupMySQL Enterprise BackupMySQL Enterprise BackupMySQL Enterprise Backup to perform online "Hot" backups of your databases

• MySQL Enterprise High AvailabilityMySQL Enterprise High AvailabilityMySQL Enterprise High AvailabilityMySQL Enterprise High Availability to automatically detect and recover from failures

• MySQL Enterprise ScalabilityMySQL Enterprise ScalabilityMySQL Enterprise ScalabilityMySQL Enterprise Scalability to improve read/write scalability by 20x

• MySQL Enterprise SecurityMySQL Enterprise SecurityMySQL Enterprise SecurityMySQL Enterprise Security to easily integrate existing security infrastructures (PAM & Windows Active Directory)

• MySQL Enterprise AuditMySQL Enterprise AuditMySQL Enterprise AuditMySQL Enterprise Audit to add policy-based auditing compliance to new and existing applications

• MySQL RepliMySQL RepliMySQL RepliMySQL Replication Monitorcation Monitorcation Monitorcation Monitor that provides real-time information on master/slave performance and latency issues

• MySQL AdvisorsMySQL AdvisorsMySQL AdvisorsMySQL Advisors to implement security, performance, replication and administration best practices

• MySQL Workbench Enterprise EditionMySQL Workbench Enterprise EditionMySQL Workbench Enterprise EditionMySQL Workbench Enterprise Edition for visual database design, SQL development, administration and

database migration

Visit http://www.mysql.com/products/enterprise/ for more details on MySQL Enterprise Edition.and MySQL

Database.

MySQL Database 5.6 with Nimble Storage

When considering best practices for running MySQL 5.6 on Oracle Linux and Red Hat Linux, the areas to

consider include performance, data protection and efficiency –especially as it related to test and development.

This document covers the best practices including performance setting, EXT file system, and volume setup with

Logical Volume Manager (LVM).

Performance Settings

When running MySQL 5.6 on Linux, there are many operating system settings that need to be tweaked to get

the best performance and uptime. However, not all settings will make the MySQL 5.6 perform better. For an


optimal performing database, there are many factors that need to be looked at. Such factors include, but not

limited to:

• How the application was written to access the database data?

• Are the queries optimal?

• Are the logical database structures layout optimal for the workload (i.e. indexes, table partitioning)?

• What is the Server CPUs and memory profile?

• What type of IO Scheduler being used in Linux?

• What is the Queue depth setting?

• What File system is being used?

• What is the IO size chosen?

• How many Volumes/LUNs are created on storage?

• What is the number of IO paths to storage?

Nimble Recommended Settings

Nimble Array OS

• Nimble OS should be at least 1.4.7.0

Linux Operating System Settings

• iSCSIiSCSIiSCSIiSCSI

Understanding the meaning of these iSCSI timeouts allows administrators to set these timeouts appropriately. These iSCSI timeouts parameters in the /etc/iscsi/iscsi.conf file should be set as follow:

node.session.timeo.replacement_timeout = 120 node.conn[0].timeo.noop_out_interval = 5 node.conn[0].timeo.noop_out_timeout = 10 node.session.nr_sessions = 4

node.session.cmds_max = 2048

node.session.queue_depth = 1024 = = = NOP= = = NOP= = = NOP= = = NOP----Out Interval/Timeout = = =Out Interval/Timeout = = =Out Interval/Timeout = = =Out Interval/Timeout = = = node.conn[0].timeo.noop_out_timeout = [ value ] iSCSI layer sends a NOP-Out request to each target. If a NOP-Out request times out (default - 10 seconds), the iSCSI layer responds by failing any running commands and instructing the SCSI layer to requeue those commands when possible. If dm-multipath is being used, the SCSI layer will fail those running commands and defer them to the multipath layer. The mulitpath layer then retries those commands on another path. If dm-multipath is not being used, those commands are retried five times (node.conn[0].timeo.noop_out_interval) before failing altogether. node.conn[0].timeo.noop_out_interval [ value ] Once set, the iSCSI layer will send a NOP-Out request to each target every [ interval value ] seconds. = = = SCSI Error= = = SCSI Error= = = SCSI Error= = = SCSI Error Handler = = =Handler = = =Handler = = =Handler = = =


If the SCSI Error Handler is running, running commands on a path will not be failed immediately when a NOP-Out request times out on that path. Instead, those commands will be failed after replacement_timeout seconds. node.session.timeo.replacement_timeout = [ value ] ImportantImportantImportantImportant: Controls how long the iSCSI layer should wait for a timed-out path/session to reestablish itself before failing any commands on it. The recommended setting of 12The recommended setting of 12The recommended setting of 12The recommended setting of 120 seconds above 0 seconds above 0 seconds above 0 seconds above allows ample time for controller allows ample time for controller allows ample time for controller allows ample time for controller faifaifaifailoverloverloverlover. Default is 120 seconds.

NoteNoteNoteNote: If set to 120 seconds, IO will be queued for 2 minutes before it can resume. The “1 queue_if_no_path1 queue_if_no_path1 queue_if_no_path1 queue_if_no_path” option in /etc/multipath.conf sets iSCSI timers to immediately defer commands to the multipath layer. This setting prevents IO errors from propagating to the application; because of this, you can set replacement_timeout to 60-120 seconds.

NoteNoteNoteNote: Nimble Storage strongly recommends using dm-multipath for all volumes.

• MultipathMultipathMultipathMultipath cccconfigurationsonfigurationsonfigurationsonfigurations

The multipath parameters in the /etc/multipath.conf file should be set as follow in order to sustain a failover.

Use aliases for mapped LUNs

defaults { user_friendly_names yes find_multipaths yes } devices { device { vendor "Nimble" product "Server" path_grouping_policy group_by_serial path_selector "round-robin 0" features "1 queue_if_no_path" path_checker tur rr_min_io_rq 10 rr_weight priorities failback immediate } } multipaths { multipath { wwid 20694551e4841f4386c9ce900dcc2bd34 alias mysqldata1 } }

• Disk Disk Disk Disk IO SchedulerIO SchedulerIO SchedulerIO Scheduler

Use “noop” io scheduler on all Nimble volumes.

To set at boot time, add the elevator option at the kernel line in the /etc/grub.conf file: elevator=noop To manually set for a particular disk:


[root@bigdata1 ~]# echo noop > /sys/block/sd?/queue/scheduler Script to change IO Scheduler for all devices. Note: If using the script, when new devices are added, the scripts must be executed again. [root@bigdata1 ~]# multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN; do echo noop > /sys/block/${LUN}/queue/scheduler ; done

• CPU CPU CPU CPU Scaling Scaling Scaling Scaling GovernorGovernorGovernorGovernor

o Use “performance” setting for all available CPUs on the host when possible.

To change CPU governor setting to performance: [root@bigdata1 ~]# echo performance > /sys/devices/system/cpu/<cpu #>/cpufreq/scaling_governor Change all CPUs [root@bigdata1 ~]# for a in $(ls -ld /sys/devices/system/cpu/cpu[0-9]* | awk '{print $NF}') ; do echo performance > $a/cpufreq/scaling_governor ; done

NoteNoteNoteNote: To make this change persistent after reboot, add the commands in /etc/rc.local file.

• /etc/sysctl.conf /etc/sysctl.conf /etc/sysctl.conf /etc/sysctl.conf

net.core.wmem_max = 16780000

net.core.rmem_max = 16780000

net.ipv4.tcp_rmem = 10240 87380 16780000

net.ipv4.tcp_wmem = 10240 87380 16780000

• max_sectors_kb max_sectors_kb max_sectors_kb max_sectors_kb

Change max_sectors_kb on all volumes to 1024 (default 512).

To change max_sectors_kb to 1024 for a single volume: [root@bigdata1 ~]# echo 1024 > /sys/block/sd?/queue/max_sectors_kb Change all volumes: multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN do echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb done


B E S T P R A C T I C E S G U I D E : N IMB LE S TOR AGE F OR M YS Q L 5 . 6 1 0

• VMVMVMVM dirty writeback and expiredirty writeback and expiredirty writeback and expiredirty writeback and expire

Change vm dirty writeback and expire to 100 (default 500 and 3000 respectively)

To change vm dirty writeback and expire: [root@bigdata1 ~]# echo 100 > /proc/sys/vm/dirty_writeback_centisecs [root@bigdata1 ~]# echo 100 > /proc/sys/vm/dirty_expire_centisecs


• iSCSI iSCSI iSCSI iSCSI Data Data Data Data NetworkNetworkNetworkNetwork

Nimble recommends using 10GbE iSCSI for all databases.

2 separate subnets

2 x 10GbE iSCSI NICs

Use jumbo frames (MTU 9000) for iSCSI networks

Example of MTU setting for eth1: DEVICE=eth1 HWADDR=00:25:B5:00:00:BE TYPE=Ethernet UUID=31bf296f-5d6a-4caf-8858-88887e883edc ONBOOT=yes NM_CONTROLLED=no BOOTPROTO=static IPADDR=172.18.127.134 NETMASK=255.255.255.0 MTU=9000 To change MTU on an already running interface: [root@bigdata1 ~]# ifconfig eth1 mtu 9000

Creating Nimble Volumes for MySQL Database 5.6

Internal testing shows that using the Nimble 4K performance policy eliminate the IO un-alignment on the Nimble

array.

Table 1Table 1Table 1Table 1:

File TypeFile TypeFile TypeFile Type # of Volumes# of Volumes# of Volumes# of Volumes Use LVM Use LVM Use LVM Use LVM

StripingStripingStripingStriping

File System File System File System File System Nimble Nimble Nimble Nimble Storage Storage Storage Storage

CachCachCachCaching Policying Policying Policying Policy

Volume Volume Volume Volume Block SizeBlock SizeBlock SizeBlock Size

(Nimble Storage)(Nimble Storage)(Nimble Storage)(Nimble Storage)

Innodb_page_sizeInnodb_page_sizeInnodb_page_sizeInnodb_page_size

MySQL

databases/binary

logs/Innodb logs

4 - system with 8 cores

8 - system with 16 cores or

Yes EXT4 Yes 4KB 4KB


more

MySQL undo logs 4 Yes EXT4 Yes 4KB 4KB

MySQL 5.6 on EXT file system

• Use Logical Volume Manager (LVM)

• 2 volume groups (1 VG for MySQL data/binary logs/Innodb logs and 1 VG for MySQL Undo logs)

o VG01 can be mounted on /mysql or at your choosing

o VG02 can be mounted on /mysql_undo or at your choosing

• The mount options for EXT file system should be “_netdev,noatime,nodiratime,discard,barrier=0”

• Examples of creating EXT file system

Create 2 Volume Groups

[root@bigdata1 ~]# vgcreate vg01 /dev/mapper/mysqldata[1-8]

[root@bigdata1 ~]# vgcreate vg02 /dev/mapper/mysqlundo[1-4]

Create Logical Volumes

[root@bigdata1 ~]# lvcreate -l 511992 -i 8 -I 4096 -n vol1 vg01

[root@bigdata1 ~]# lvcreate -l 102396 -i 4 -I 4096 -n vol1 vg02

Create EXT4 File Systems

[root@bigdata1 ~]# mkfs.ext4 /dev/vg01/vol1 -b 4096

[root@bigdata1 ~]# mkfs.ext4 /dev/vg02/vol1 -b 4096

Add to /etc/fstab

/dev/vg01/vol1 /mysql ext4 _netdev,noatime,nodiratime,discard,barrier=0 0 0

/dev/vg02/vol1 /mysql_undo ext4 _netdev,noatime,nodiratime,discard,barrier=0 0 0

• Here is the performance policy for MySQL.

NoteNoteNoteNote: As with any workload, be sure to evaluate the active data set and the cache hit rate through the Nimble

Storage management tool to ensure optimal performance of the transactional database.


NoteNoteNoteNote: Consider turning on aggressive caching when migrating databases from a legacy storage system. This can

ensure faster response times as some of the active data may already be loaded into the flash cache.

InnoDB Settings for MySQL 5.6

Internal testing of a single instance shows that setting these InnoDB parameters in the /etc/my.cnf file yields

better performance compare to the defaults.

# Data & Log directories innodb_data_home_dir = /mysql/data # User defined. innodb_log_group_home_dir = /mysql/innodblog # User defined # InnoDB logs section innodb_log_files_in_group = 4 # User defined innodb_log_file_size = 32G # User defined. Higher is better but recovery is longer # InnoDB log buffer section innodb_log_buffer_size = 8590000000 # User defined. Larger is better for transactional workload. # InnoDB buffer pool section # This setting can be set at or up to 80% of the machine physical memory size innodb_buffer_pool_size = 40G # User defined # InnoDB file section innodb_file_per_table = on # Nimble recommended. This separates data & index from system tablespace # InnoDB flush section innodb_flush_method = O_DIRECT # Nimble required # InnoDB page size section innodb_page_size = 4096 # Nimble required to eliminate IO un-alignment # Undo section innodb_undo_directory = /mysql_undo/innodb_undo # User defined but needs to be on separate VG innodb_undo_tablespaces = 5 # Nimble required on separate VG. Number is user defined innodb_undo_logs = 20 # Nimble required on separate VG. Number is user defined # IO Capacity section Innodb_io_capacity = 200 (default) # User defined - depending on workload and IO pattern. Internal testing of OLTP workload shows increasing this number does not benefit.

NoteNoteNoteNote: The parameters above specified with “User defined” need to be tested to suit the environment. Any other

parameters for InnoDB related to disk I/O not mentioned above have been tested with OLTP workload but did not have any impact.

Using Snapshot and Zero-Copy Clones Features

Nimble Storage recommends using the native Nimble Storage snapshot feature to protect MySQL databases.

Below are some business requirements that can be achieved quickly and efficiently using snapshot.


• Whole database backup

• Database refreshes for Test/Dev/QA

• Database replication

• Test MySQL and Operating System patches

Using Nimble Storage Snapshot and Zero-Copy Clone for MySQL Backup

In addition to backing up MySQL with mysqldump, mysqlhotcopy or MySQL Enterprise Backup, leveraging the

Nimble snapshot feature can also be done. The Nimble snapshot feature is instantaneous and space efficient.

It is very useful to spin up an instance for testing or development purposes.

Nimble Storage Snapshot for Backup

There are 2 ways to make a backup of MySQL using Nimble snapshot.

• Using Nimble Schedule

• Using Nimble CLI via scripting

Using Nimble Schedule

After creating Nimble volumes for MySQL database, a Volume Collection can be created and applied to all MySQL

volumes.

1. Connect to the Nimble GUI and click on “Manage”, “Protection”, “Volume Collections”.

Data, innodb log,

binlog

Innodb undo logs

Data

Volume

Undo

Volume

MySQL

Volume

Collection


2. Click on “New Volume Collection” and enter a name. Select the “Blank schedule” radio button and click

“Next”

3. Select “None” under the Synchronization window and click “Next”.


4. Enter a name for the “Schedule Name”. In this example, an hourly snapshot schedule is created and

retained for 25 snapshot. Multiple schedules can be created for the same Volume Collection. Click “Next”.

5. Select the MySQL volumes and click on the “Add” button. Multiple volumes can be added in a single step by

highlighting all the volumes. Click “Finish” when complete.


Using Nimble Command Line Interface (CLI)

When using the Nimble CLI to create a snapshot, a Volume Collection must exist. Here is an example of creating a

snapshot of the Volume Collection created above.

[root@bigdata1 ~]# ssh admin@<nimble-array> volcoll - -snap <VolumeCollection> - -snapcoll_name <name of snap>

NoteNoteNoteNote: The snapshots created by the CLI are not managed by the schedule, meaning manual retention is required.

Nimble Storage Snapshot for Clones

Creating a replica of a production database in MySQL is simple. The steps below show how to create a test

database on another server using one of the available production snapshots.

1. Create the latest snapshot of production volumes

2. Create clones from the snapshot and map the cloned volumes to a test/dev server

3. Run iSCSI discovery and login

4. Scan for LVM Volume Groups. Note: the cloned volumes contain the exact VGID and PVID of the

production volumes. Hence, the test/dev server should not have the same VGID. For example, if

MySQL database on production server uses VG01 and VG02, then the test/dev server should not have

VG01 and VG02 already defined.

5. Mount the logical volumes on the same mount points as production.

Nimble Storage, Inc.

211 River Oaks Parkway, San Jose, CA 95134

Tel: 877-364-6253) | www.nimblestorage.com | [email protected]

© 2014 Nimble Storage, Inc. Nimble Storage, InfoSight, SmartStack, NimbleConnect, and CASL are trademarks or registered trademarks of Nimble Storage, Inc. All other trademarks are the property of their respective owners. BPG-MySQL-1114

TR Nimble MySQL Best Practices Guide-v3 · PDF fileBEST PRACTICES GUIDE: ... The purpose of...

Documents

Transcript of TR Nimble MySQL Best Practices Guide-v3 · PDF fileBEST PRACTICES GUIDE: ... The purpose of...