1. Mastering InnoDB Diagnostics Harrison Fisk Senior Principal
Technical Support Engineer
2. Overview
Lots of information
Hard to understand
3. Easy to get overwhelmed
Hard to figure out what is useful
4. Two methods for troubleshooting
Symptom based
What we will be doing today
Data based
5. Information Sources
6. Sources
MySQL info
SHOW ENGINE INNODB STATUS
7. InnoDB monitors
8. SHOW GLOBAL STATUS
9. INFORMATION_SCHEMAtables
SHOW PROCESSLIST
10. SHOW MUTEX STATUS
OS info
vmstat / iostator Performance Monitor
11. Gdb ( Poor mans profiler (PMP))
12. OS specific tools
13. SHOW INNODB STATUS
Be careful of truncation
innodb_status_file
14. Table monitors
Ensure enough time has passed
15. Exact format varies
InnoDB built-in
16. InnoDB plugin
17. XtraDB
18. Data Dictionary
19. Problems
InnoDB: Error: table test/parent already exists in InnoDB
internal InnoDB: data dictionary. Have you deleted the .frm file
InnoDB: and not used DROP TABLE? InnoDB: Cannot find table
test/child2 from the internal data dictionary InnoDB: of InnoDB
though the .frm file for the table exists. Maybe you InnoDB: have
deleted and recreated InnoDB data files but have forgotten InnoDB:
to delete the corresponding .frm files of InnoDB tables?
Various errors about not being in sync
20. Can't drop and/or create tables or databases
21. Example errors:
22. Diagnostics
InnoDB table monitor
Dumps InnoDB data dictionary
23. Output to error log
24. CREATE TABLE innodb_table_monitor (id int)
ENGINE=innodb;
25. Interpreting
Ignore system tables
NamedSYS_
Ignore hidden fields from InnoDB
DB_ROW_ID
26. DB_TRX_ID
27. DB_ROLL_PTR
Ignore hidden indexes
GEN_CLUST_INDEX
28. Solutions
Ensure.frmmatches InnoDB
Create.frmas needed
Create another table with same structure
29. Copy.frmto the name of the table
Drop orphaned.frmfiles
If not in InnoDB
30. Remove them
31. Crashing
32. Problem
mysqld Crash
Signal 11
Segfault
Signal 6
Abort
33. InnoDB loves this one
May not actually notice immediately
mysqld_safe restarts
34. Diagnostics
Error log
Hopefully it has info in it
Core file
Not enabled by default
35. OS often restricts setuid cores
/proc/sys/fs/suid_dumpable
36. coreadm
General log
When all else fails
37. Interpreting Error log
Error log
Look for signal type
Signal 11
38. Signal 6
Backtrace
May or may not be present
OS
39. Version
In 5.0 and before you need to resolve it
resolve_stack_dump
Search bug system for function names
40. Interpreting Cores
Must be enabled
41. Memory image produced
42. Requires exactly the same binary and libraries
43. With rpm ensure debuginfo is installed
44. Non-stripped binaries
45. Load with gdb
thread apply all bt
46. Use PMP to summarize if needed
Solaris:pstack
47. Windows: Minidump/DrWatson/Windbg
48. Solutions
Find cause of the crash
InnoDB doing on purpose
49. Real crash
Find query causing problem
Stop doing it
50. Report bug to us
Upgrade
Lots of bugs fixed each release
51. Locking
52. Problems
Deadlocks
Real deadlocks
ERROR 1213 (40001): Deadlock found when trying to get lock; try
restarting transaction
Reading and writing your actual tables and indexes
78. More common with very large tables
Compare to buffer pool size
Sometimes caused by read ahead in InnoDB
79. Diagnostics
SHOW INNODB STATUS
Buffer pool hit rate
Watch absolute values too
FILE I/O
Reads/s
Avg bytes/read shows reada head
Pending reads
Read ahead
Pages read ahead/evicted without access
iostat
avgqu-sz
80. await
81. Solutions
Optimize queries
82. Increaseinnodb_buffer_pool_size
83. New LRU algorithm in InnoDB plugin
84. Tune/disable readahead
In plugin, can be tuned
85. In non-plugin
Use gdb hack
86. set srv_startup_is_before_trx_rollback_phase=1
Buy better disks
87. CPU: Lots of Rows
Bad queries can read lots of rows
Or lots and lots of smaller queries
Data fully cached
Lots of buffer pool reading
88. Often smaller dataset
Eventually will bottleneck InnoDB on CPU
89. Diagnostics
SHOW INNODB STATUS
ROW OPERATIONS
Reads/s
SHOW GLOBAL STATUS LIKE 'Handler%'
Type of row operations going on
Slow query log
log-queries-not-using-indexes
90. Solutions
Fix your queries
Ensure proper index usage
Do less queries
External caching
Scale reads
Replication
91. CPU: Thread Thrashing
InnoDB can have issues at high concurrency
92. Later releases help reduce this dramatically
InnoDB plugin
93. Further versions
XtraDB
94. MySQL 5.5
95. Diagnostics
SHOW INNODB STATUS
Lots of queued threads
TRANSACTIONS
96. Queries in queue
ROW OPERATIONS
reads/s is low
vmstat
Very high CPU usage
97. Lots of cs context switches
SHOW MUTEX STATUS
98. Solutions
innodb_thread_concurrency
Very high or 0 works best until it doesn't
Sometimes very low works best
99. A bit of a black art in tuning
innodb_concurrency_tickets
Size based on number of rows being touched
Use newer version
Lots of work to help this in InnoDB Plugin
100. Questions?
101. The presentation is intended to outline our general
product direction. It is intended for information purposes only,
and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and
should not be relied upon in making purchasing decisions. The
development, release, and timing of any features or functionality
described for Oracles products remains at the sole discretion of
Oracle.