Advanced MySQLQuery OptimizationPHP World November 16th, 2017

Safe Harbor Agreement

THE FOLLOWING IS INTENDED TO OUTLINE OUR GENERAL PRODUCT DIRECTION. IT IS INTENDED FOR INFORMATION PURPOSES ONLY, AND MAY NOT BE INCORPORATED INTO ANY CONTRACT. IT IS NOT A COMMITMENT TO DELIVER ANY MATERIAL, CODE, OR FUNCTIONALITY, AND SHOULD NOT BE RELIED UPON IN MAKING PURCHASING DECISIONS. THE DEVELOPMENT, RELEASE, AND TIMING OF ANY FEATURES OR FUNCTIONALITY DESCRIBED FOR ORACLE'S PRODUCTS REMAINS AT THE SOLE DISCRETION OF ORACLE.

2

IntroSo you know how to add indexes to speed queries and maybe can use EXPLAIN. But why do your queries still stink? This session covers how the MySQL Optimizer looks at your SQL statements, where it has to throw up its hands in frustration, and how to tame it. Plus, there are some minor data architecture tweaks you need to know.

3

Ground Rules

The only dumb question is the one you don’t ask!

This is your time and I hope this is a wise investment!

Many people are needlessly frightened of databases. They do not bite …

..but they will chew your butt if not careful!

Practice good programming practices & stress clarity4

Seven P’s

5

Some Prologue

6

7

COST MODEL

Hueristics

Storage Engine

Optimzer

Index Info

MySQL Server

SELECT City.name, Country.nameFROM cityJOIN (city.countryCode = country.code)

MySQL Server Architecture

IndexesIndexes greatly speed the lookup of records

But there is overhead

Insertion of new records

Updates of existing records

Removal of existing records

Disk space & Memory

The speed of record lookup should faster that the speed of the overhead

8

9

General IdeaOf cost based query

optimization

1. Assign Costs to operations2. Compute costs of alternative

plans3. Search for lowest cost plan

Cost Based Optimizations:Access MethodJoin OrderSubquery Strategy

Cost InputsWhat goes into the cost model

IO CostCost of number pages to read to

get data -- both INDEX and DATA

SchemaLength of records & keysUniqueness of indexesNULLable

StatisticsNumber or rows in tableNumber of records in index rangeKey DistributionIndex Cardinality

(Avg number of records per key)

10

Example Query

SELECT City.Name, Country.NameFROM CityJOIN CountryON (City.Countrycode = Country.code);

Read the City file, print the name of the city and print the corresponding country name.

Quiz: Does having Country.Code a ‘PRI’ help in this query’s speed?

11

DESC City;

DESC Country;

Example Query

SELECT city.name as 'City',country.name as 'Country'

FROM cityJOIN country ON (city.countryCode=country.Code)

12

EXPLAIN

What is this saying?

1. We have to read ALL (100 %) of the City table, 4079 rows 2. For each of the rows (loop) in #1 we have to read one row in the Country table,

Will need to read 100% of that table (but one line at a time),There is a key to use,The PRIMARY will be used with eq_ref on world_x.City.CountryCode

Quiz: Explain the key_len length13

Example Query

SELECT city.name as 'City',country.name as

'Country'FROM cityJOIN country ON (city.countryCode=country.Code)

14

15

Query Trace{ "query_block": { "select_id": 1, "cost_info": { "query_cost": "5231.03" }, "nested_loop": [ { "table": { "table_name": "country", "access_type": "ALL", "possible_keys": [ "PRIMARY" ], "rows_examined_per_scan": 239, "rows_produced_per_join": 239, "filtered": "100.00", "cost_info": { "read_cost": "6.00", "eval_cost": "47.80", "prefix_cost": "53.80", "data_read_per_join": "61K" }, "used_columns": [ "Code",

"Name" ] } }, { "table": { "table_name": "city", "access_type": "ref", "possible_keys": [ "CountryCode" ], "key": "CountryCode", "used_key_parts": [ "CountryCode" ], "key_length": "3", "ref": [ "world.country.Code" ], "rows_examined_per_scan": 18, "rows_produced_per_join": 4314, "filtered": "100.00", "cost_info": { "read_cost": "4314.36", "eval_cost": "862.87", "prefix_cost": "5231.03", "data_read_per_join": "303K" }, "used_columns": [ "Name", "CountryCode" ] } } ] }}

Cost Examples

TABLE SCAN

IO-Cost -- Number of Pages in table

CPU Cost -- Number of ROWS * ROW_EVALUATE_COST

16

RANGE SCAN

IO-Cost -- Number of Pages in Index + Number of Rows in Range

CPU Cost -- Number Rows in range * ROW_EVALUATE_COST

Slightly More Complex Example

select a,b from abwhere a > 10 and a < 25 and a NOT IN (11,19) and (b < 5 or b > 10);

{ "query_block": { "select_id": 1, "cost_info": { "query_cost": "2.40" }, "table": { "table_name": "ab", "access_type": "ALL", "rows_examined_per_scan": 7, "rows_produced_per_join": 0, "filtered": "14.29", "cost_info": { "read_cost": "2.20", "eval_cost": "0.20", "prefix_cost": "2.40", "data_read_per_join": "15" }, "used_columns": [ "a", "b" ], "attached_condition": "((`preso`.`ab`.`a` > 10) and (`preso`.`ab`.`a` < 25) and (`preso`.`ab`.`a` not in (11,19)) and ((`preso`.`ab`.`b` < 5) or (`preso`.`ab`.`b` > 10)))" } }}

17

RTFMAll you will here today is in the MySQL manual. I am just pointing out some of the stuff

that should be more obvious or need to be pointed out.

18

Optimizing Querieswith EXPLAIN

https://dev.mysql.com/doc/refman/5.7/en/using-explain.html

19

Things we could speed a week on easily!!

READ THIS LATER FOR BACKGROUND!

Chapter 1 -- Index Use

20

Use Only Index

In some cases, a query can be optimized to retrieve values without consulting the data rows. (An index that provides all the necessary results for a query is called a covering index.) If a query uses from a table only columns that are included in some index, the selected values can be retrieved from the index tree for greater speedhttps://dev.mysql.com/doc/refman/5.7/en/mysql-indexes.html

21

ExampleCREATE TABLE ionly (a int, b int, c int);

CREATE INDEX idxonly ON ionly(a,c,b);

INSERT INTO ionly VALUES (1,10,100),(2,20,200),(3,30,300),(4,40,400) ….

SELECT c FROM ionly where a=1;

22

FasterThis allows the server to ONLY read the index, not the index THEN the data

23

Note:

To eliminate rows from consideration. If there is a choice between multiple indexes, MySQL normally uses the index that finds the smallest number of rows (the most selective index).

24

This may bite you where you least want to be bitten

Multiple Column Index

If the table has a multiple-column index, any leftmost prefix of the index can be used by the optimizer to look up rows. For example, if you have a three-column index on (col1, col2, col3), you have indexed search capabilities on (col1), (col1, col2), and (col1, col2, col3). For more information, see Section 8.3.5, “Multiple-Column Indexes”.

25

Where to be careful!

Hashed Indexes

As an alternative to a composite index, you can introduce a column that is “hashed” based on information from other columns.

If this column is short, reasonably unique, and indexed, it might be faster than a “wide” index on many columns. In MySQL, it is very easy to use this extra column

26

SELECT * FROM tbl_name WHERE hash_col=MD5(CONCAT(val1,val2)) AND col1=val1 AND col2=val2;

Compare Apples and OrangesMySQL can use indexes on columns more efficiently if they are declared as the same type and size. In this context, VARCHAR and CHAR are considered the same if they are declared as the same size. For example, VARCHAR(10) and CHAR(10) are the same size, but VARCHAR(10) and CHAR(15) are not.

For comparisons between nonbinary string columns, both columns should use the same character set. For example, comparing a utf8 column with a latin1 column precludes use of an index.

Comparison of dissimilar columns (comparing a string column to a temporal or numeric column, for example) may prevent use of indexes if values cannot be compared directly without conversion. For a given value such as 1 in the numeric column, it might compare equal to any number of values in the string column such as '1', ' 1', '00001', or '01.e1'. This rules out use of any indexes for the string column.

https://dev.mysql.com/doc/refman/5.7/en/mysql-indexes.html

27

Where Indexes won’t work

● Indexed column used as argument to a function YEAR(birthdate) < 2007

● Suffix searches name LIKE ‘%ington’

● Mismatched type last_name = 10

28

Chapter 2 Avoid Full Table Scans (if you can)

29

Remember?

30

Example w/WHERESELECT city.name as 'City',

country.name as 'Country'FROM cityJOIN country ON (city.countryCode=country.Code)WHERE country.Code = 'USA'

31

Finishing Full Table ScansFor small tables, a table scan often is appropriate and the performance impact is negligible. For large tables, try the following techniques to avoid having the optimizer incorrectly choose a table scan:

● Use ANALYZE TABLE tbl_name to update the key distributions for the scanned table. See Section 13.7.2.1, “ANALYZE TABLE Syntax”.

● Use FORCE INDEX for the scanned table to tell MySQL that table scans are very expensive compared to using the given index:

● SELECT * FROM t1, t2 FORCE INDEX (index_for_column) WHERE t1.col_name=t2.col_name;

32

Query PlanSome databases let you ‘lock down’ the query plan. MySQL ain’t one of them.

33

Index HintsIndex hints give the optimizer information about how to choose indexes during query processing. Index hints are specified following a table name.

SELECT * FROM t USE INDEX (index1) IGNORE INDEX (index1) FOR ORDER BY IGNORE INDEX (index1) FOR GROUP BY WHERE ... IN BOOLEAN MODE ... ;

SELECT * FROM t USE INDEX (index1) WHERE ... IN BOOLEAN MODE ... ; 34

Optimizer Hintsoptimizer hints apply on a per-statement basis

SELECT /*+ NO_RANGE_OPTIMIZATION(t3 PRIMARY, f2_idx) */ f1 FROM t3 WHERE f1 > 30 AND f1 < 33;SELECT /*+ BKA(t1) NO_BKA(t2) */ * FROM t1 INNER JOIN t2 WHERE ...;SELECT /*+ NO_ICP(t1, t2) */ * FROM t1 INNER JOIN t2 WHERE ...;SELECT /*+ SEMIJOIN(FIRSTMATCH, LOOSESCAN) */ * FROM t1 ...;EXPLAIN SELECT /*+ NO_ICP(t1) */ * FROM t1 WHERE ...;

35

Foreign Keys

36

Why Use Foreign Keys

37

If a table has many columns, and you query many different combinations of columns, it might be efficient to split the less-frequently used data into separate tables with a few columns each, and relate them back to the main table by duplicating the numeric ID column from the main table. That way, each small table can have a primary key for fast lookups of its data, and you can query just the set of columns that you need using a join operation.

Depending on how the data is distributed, the queries might perform less I/O and take up less cache memory because the relevant columns are packed together on disk. (To maximize performance, queries try to read as few data blocks as possible from disk; tables with only a few columns can fit more rows in each data block.)

https://dev.mysql.com/doc/refman/5.7/en/foreign-key-optimization.html

FKs

Foreign keys let you cross-reference related data across tables, and foreign key constraints help keep this spread-out data consistent.

MySQL supports ON UPDATE and ON DELETE foreign key references in CREATE TABLE and ALTER TABLE statements. The available referential actions are RESTRICT (the default), CASCADE, SET NULL, and NO ACTION.

38

MySQL supports foreign keys, which let you cross-reference related data across tables, and

foreign key constraints, which help keep this spread-out data consistent.

Use FKsTo automatically delete or update child tables.

39

Example

CREATE TABLE parent ( id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB;

40

CREATE TABLE child ( id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB;

PK Example

select parent.id as 'parent',child.id as 'child'from parentjoin child on (parent.id=child.parent_id);

delete from parent where id=20;

The PK definition removed the corresponding child record -- you do not have to remove it! 41

MySQL 8MySQL 8 fixes a lot of legacy issues (Bug 199, ALTER TABLE, etc.) and introduces some cool features!

42

A lot of cool stuff is on the way

NOWAITNOWAIT informs the server to return immediately IF the desired rows can not be locked.

How To Use NOWAITStart a connection to your MySQL 8 server, start a transaction, and make a query to lock up part of the data.mysql>START TRANSACTION;mysql>SELECT * FROM city WHERE countryCode = 'USA' FOR UPDATE;

On a second connection, start a transaction and issue a SELECT query with NOWAIT.

mysql>START TRANSACTION;mysql>SELECT * FROM city WHERE countryCode = 'USA' FOR UPDATE NOWAIT;

The second connection will get a message saying 'statement aborted because lock(s) could not be acquired immediately and NOWAIT is SET

43

SKIP LOCKEDSKIP LOCKED allows you to not wait about for locked records and work on what is available -- the unlocked records.

The MySQL world database has 274 records in the city table where the countryCode field equals 'USA' there are 274 records. From past interactions, we somehow know there are some records with the ID field greater than 4000.

On MySQL shell number 1, start a transaction and lock some recordsmysql>START TRANSACTION;mysql>SELECT * FROM city WHERE ID > 4000 and countryCode = 'USA';There will be 66 records.

On MySQL shell number number 2, start a transaction and lets try to lock the records starting at IS 3990 and up.mysql>START TRANSACTION;mysql>SELECT FROM city WHERE id > 3990 and countryCode='USA' FOR UPDATE SKIPPED LOCKED;There will be 10 records. The records 4000 and up are locked by the transaction on the other shell.

44

More in 8MySQL is a major revolution, as is the new Document Store. Oracle is working very hard

to provide a much improved product with each release.

45

Another Option?The MySQL Document Store allows you to use MySQL as a Document Database -- No schemas, no tables, no data normalization -- Store data in JSON

46

MySQL without the SQLOh My!!

MySQL Document Store

47

Built on the new X DevAPI and new Protocol

Allows for async calls and used for InnoDB Cluster management

Write data to collections

But can still talk to ol’ SQL style relational tables

JavaScript, Node.JS, Java, Python, C/C++ and PHP

PHP PECL mysql_xdevapi #!/usr/bin/php<?PHP// Connection Parameters $user = 'root'; $passwd = 'S3cret#'; $host = 'localhost'; $port = '33060'; $connection_uri = 'mysqlx://'.$user.':'.$passwd.'@'.$host.':'.$port;

$nodeSession = mysql_xdevapi\getNodeSession($connection_uri); $schema = $nodeSession->getSchema("world_x"); $collection = $schema->getCollection("countryinfo"); $result = $collection->find('_id = "USA"')->execute(); $data = $result->fetchAll(); var_dump($data);?> 48

Final pointToo much in this subject for a

mere hour!

49

“Now it’s computers and more computersand soon everybody will have one,

3-year-olds will have computersand everybody will know everything

about everybody elselong before they meet them.

nobody will want to meet anybodyelse ever again

and everybody will bea recluse

like I am now.”

- Charles BukowskiAugust 16, 1920 – March 9, 1994

50

MySQL Optimization Details https://www.slideshare.net/oysteing/how-to-analyze-and-tune-sql-queries-for-better-performance-percona15?qid=598b4222-0f0d-4a01-9661-d679093e4310&v=&b=&from_search=3

MySQL SYS Schemahttps://dev.mysql.com/doc/refman/5.7/en/sys-schema.html

MySQL Blogshttps://planet.mysql.com/

51

Thanks!david.stokes@oracle.com

@Stoker

https://elephantdolphin.blogspot.co.uk/

https://joind.in/talk/688b0

52