MySQL Query Optimization
-
Upload
morgan-tocker -
Category
Software
-
view
516 -
download
1
Transcript of MySQL Query Optimization
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Query Optimization
Morgan Tocker MySQL Community Manager October, 2014
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |4
Today’s Agenda
Introduction
World Schema
IMDB Schema
Advanced Topics
1
2
3
4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Introduction
• SQL is declarative. • You say what you want, not how to process. • Can’t sight check a query to understand how it executes. • Database management chooses the best possible way -‐ think like a GPS navigator.
5
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Diagnostic Commands
• EXPLAIN (all versions) • EXPLAIN FORMAT=JSON (MySQL 5.6+) • Workbench supports visual format.
• OPTIMIZER TRACE (MySQL 5.6+)
8
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |9
Today’s Agenda
Introduction
World Schema
IMDB Schema
Advanced Topics
1
2
3
4
2
1
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
The World Schema
• Contains cities, countries and language statistics. • Download From: • http://dev.mysql.com/doc/index-‐other.html
• Very small data set -‐ good for learning. • Not perfect for explaining performance differences
10
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Find Countries in Asia with Population >5M
11
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 239 filtered: 4.76 Extra: Using where1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 12
EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "53.80" }, "table": { "table_name": "Country", "access_type": "ALL", "rows_examined_per_scan": 239, "rows_produced_per_join": 11, "filtered": 4.7614, "cost_info": { "read_cost": "51.52", "eval_cost": "2.28", "prefix_cost": "53.80", "data_read_per_join": "2K" }, "used_columns": [ "Code", .. "Capital", "Code2" ], "attached_condition": "((`world`.`country`.`Continent` = 'Asia') and (`world`.`country`.`Population` > 5000000))" } } }
FORMAT=JSONAvailable since MySQL 5.6. Expanded in 5.7 to add cost information.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
What indexes will make this query faster?
• Some suggestions: • Index on (population) • Index on (continent) • Index on (population, continent) • Index on (continent, population)
13
SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000;
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Next Question
• Given so many choices, which is the best choice? • A good GPS navigator understands traffic and finds the fastest route. • A good query optimizer does similar.
14
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Query Optimizer Strategy
• Pick the plan that consumes the least amount of resources (CPU, IO).
15
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Query Optimizer Strategy (cont.)
• Use statistics available to identify good and bad plans. • Decisions may include: • Which indexes are good/bad. • Which order to join tables in. • If indexes should be read in a particular order to avoid sorting, or if sorting is not expensive.
16
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Using single column indexes first….
• p (population) • c (continent)
17
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 18
mysql> ALTER TABLE Country ADD INDEX p (population);Query OK, 0 rows affected (0.04 sec)Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G************************ 1. row ************************ id: 1 select_type: SIMPLE table: Country partitions: NULL type: ALLpossible_keys: p key: NULL key_len: NULL ref: NULL rows: 239 filtered: 6.46 Extra: Using where1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Why would an index not be used?
19
mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country FORCE INDEX (p) WHERE Continent = 'Asia' AND population > 5000000\G******* 1. row *******EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "152.21" }, "table": { "table_name": "Country", "access_type": "range", "possible_keys": [ "p" ],
mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G******* 1. row *******EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "53.80" }, "table": { "table_name": "Country", "access_type": "ALL", "possible_keys": [ "p" ],
Using MySQL 5.7+
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Modifying the query range: >50M
20
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: p key: p key_len: 4 ref: NULL rows: 24 filtered: 14.29 Extra: Using index condition; Using where1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 21
mysql> ALTER TABLE Country ADD INDEX c (continent);Query OK, 0 rows affected (0.04 sec)Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 5000000\G*********************** 1. row *********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: refpossible_keys: p,c key: c key_len: 1 ref: const rows: 51 filtered: 45.19 Extra: Using where1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 22
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 500000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: p,c key: p key_len: 4 ref: NULL rows: 2 filtered: 21.34 Extra: Using index condition; Using where1 row in set, 1 warning (0.00 sec)
Modifying the query range: >500M
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Composite Indexes
• On >50M which is better? • (population, continent) • (continent, population)
23
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 24
mysql> ALTER TABLE Country ADD INDEX pc (pop..on, co..nt);Query OK, 0 rows affected (0.02 sec)Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: refpossible_keys: p,c,pc key: c key_len: 1 ref: const rows: 51 filtered: 10.04 Extra: Using where1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 25
mysql> EXPLAIN SELECT * FROM Country FORCE INDEX (pc) WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: pc key: pc key_len: 4 ref: NULL rows: 24 filtered: 14.29 Extra: Using index condition1 row in set, 1 warning (0.00 sec)
FORCE INDEX
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 26
mysql> EXPLAIN FORMAT=JSON SELECT * FROM Country FORCE INDEX (pc) WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row **********************EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "34.61" }, "table": {.. ], "key": "pc", "used_key_parts": [ "Population" ],..
Easier to see with JSON EXPLAIN
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Rule of Thumb
• Index on (const, range) instead of (range, const). • Applies to all databases.
27
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 28
mysql> ALTER TABLE Country ADD INDEX cp (co..nt, pop..on);Query OK, 0 rows affected (0.02 sec)Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT * FROM Country WHERE Continent = 'Asia' AND population > 50000000\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: Country partitions: NULL type: rangepossible_keys: p,c,pc,cp key: cp key_len: 5 ref: NULL rows: 11 filtered: 100.00 Extra: Using index condition1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Cost Estimates
29
5M 50M 500M
Table Scan 53.80 53.80 53.80
(Population) 152.21 34.61 3.81
(Continent) 28.20 28.20 28.20
(Population, Continent)
152.21 34.61 3.81
(Continent, Population)
24.83 16.41 3.81
Where Population >N. Lower is better.
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |30
Today’s Agenda
Introduction
World Schema
IMDB Schema
Advanced Topics
1
2
4
1
3
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Examples Using IMDB
• Larger than world database schema. • Better to understand performance differences. • License requires data to be regenerated: • http://imdbpy.sourceforge.net/
31
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 32
mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 2387523 filtered: 0.50 Extra: Using where1 row in set, 1 warning (0.00 sec)
Ignore EXPLAIN time Actual run time is 2.9 seconds (Not shown)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 33
mysql> ALTER TABLE title ADD INDEX (title(100));Query OK, 0 rows affected (4.73 sec)Records: 0 Duplicates: 0 Warnings: 0
mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: refpossible_keys: title key: title key_len: 102 ref: const rows: 4 filtered: 100.00 Extra: Using where1 row in set, 1 warning (0.01 sec)
From 2.9 seconds to 0.0 seconds (!!!)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
How does LIKE work?
34
mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'The Lion %'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: rangepossible_keys: title key: title key_len: 102 ref: NULL rows: 109 filtered: 100.00 Extra: Using where1 row in set, 1 warning (0.01 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
LIKE (cont.)
35
mysql> EXPLAIN SELECT * FROM title WHERE title LIKE 'The%'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: rangepossible_keys: title key: title key_len: 102 ref: NULL rows: 325548 filtered: 100.00 Extra: Using where1 row in set, 1 warning (0.00 sec)
VS
Very Close. Estimate is that it is almost as fast to table scan. Run Time is 1.9 sec
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Sorting
36
mysql> EXPLAIN SELECT * FROM title WHERE title = 'The Lion King' ORDER BY production_year\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title partitions: NULL type: refpossible_keys: title key: title key_len: 102 ref: const rows: 4 filtered: 100.00 Extra: Using where; Using filesort1 row in set, 1 warning (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 37
SELECT movie_info.* FROM title INNER JOIN movie_info ON title.id=movie_info.movie_id WHERE title=‘The Lion King’\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: movie_info type: ALLpossible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 1212187 Extra: NULL********************** 2. row ********************** id: 1 select_type: SIMPLE table: title type: eq_refpossible_keys: PRIMARY,title key: PRIMARY key_len: 4 ref: imdb.movie_info.movie_id rows: 1 Extra: Using where
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 38
mysql> ALTER TABLE movie_info ADD INDEX mi (movie_id);Query OK, 0 rows affected (1.95 sec)Records: 0 Duplicates: 0 Warnings: 0
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 39
EXPLAIN SELECT movie_info.* FROM title INNER .. WHERE title='The Lion King'\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title type: refpossible_keys: PRIMARY,title key: title key_len: 102 ref: const rows: 4 Extra: Using where********************** 2. row ********************** id: 1 select_type: SIMPLE table: movie_info type: refpossible_keys: mi key: mi key_len: 4 ref: imdb.title.id rows: 7 Extra: NULL
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. | 40
mysql> EXPLAIN SELECT * FROM movie_info WHERE movie_id IN (SELECT id FROM title WHERE title = 'The Lion King')\G********************** 1. row ********************** id: 1 select_type: SIMPLE table: title type: refpossible_keys: PRIMARY,title key: title key_len: 102 ref: const rows: 4 Extra: Using where********************** 2. row ********************** id: 1 select_type: SIMPLE table: movie_info type: refpossible_keys: mi key: mi key_len: 4 ref: imdb.title.id rows: 7 Extra: NULL
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |41
Today’s Agenda
Introduction
World Schema
IMDB Schema
Advanced Topics
1
2
3
1
4
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizer Hints
• USE INDEX • FORCE INDEX • IGNORE INDEX • STRAIGHT JOIN
42
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizer Hints (cont.)
43
Expanded in MySQL 5.6
mysql> select @@optimizer_switch\G********************** 1. row **********************@@optimizer_switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on,condition_fanout_filter=on1 row in set (0.00 sec)
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Statistics Sampling
• Some decisions based on pre-‐computed statistics. • For example, which order to join tables.
• Accuracy of statistics greatly improved in MySQL 5.6. • Feature is called “InnoDB persistent statistics”.
44
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizer Decision Visibility
• Intended for MySQL developer usage, and reporting optimizer bugs. • Yet totally practical for advanced usage :)
45
New to MySQL 5.6
SET optimizer_trace="enabled=on";SELECT /* query here */;SELECT * FROM INFORMATION_SCHEMA.OPTIMIZER_TRACE;
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Optimizer Trace Output
• Includes why decisions were not made:
46
.. "index": "pc", "ranges": [ "50000000 < Population" ], "index_dives_for_eq_ranges": true, "rowid_ordered": false, "using_mrr": false, "index_only": false, "rows": 24, "cost": 29.81, "chosen": false, "cause": "cost" },
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Too many indexes
• Slows down modifications due to maintenance cost. • i.e. Slower INSERT/UPDATE/DELETE
• May also lead to worse decisions being made by optimizer. • Too many similar choices. • Only limited time window to evaluate suitability.
• Space requirements in memory and storage.
47
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
MySQL 5.7 and beyond
• We are working on a new cost model. • Better adapt to new hardware with different costs (i.e. SSDs have cheap random IO). • Memory buffer aware cost estimates.
48
Copyright © 2014 Oracle and/or its affiliates. All rights reserved. |
Cost constants configurable per engine in 5.7
• Previously hard-‐coded constants. • Better adapts to differences in hardware.
49