Intro to Deadlocks

Intro to Deadlocks

February 4th, 2013Panupan Sriautharawong

Monday, February 11, 13

What are Deadlocks?

A situation that occurs in concurrent systems.

Happens when two or more competing actions get stuck because they’re waiting on one another to finish.

A bug or design flaw in most cases.

Preventable!


InnoDB Storage Engine



TransactionsAre used to group of database operations

Are atomic, meaning they can’t broken into smaller pieces. Either everything goes through or nothing does.

## Bank Transfer ExampleBEGIN;UPDATE account SET balance=balance+100 WHERE id=1;UPDATE account SET balance=balance-100 WHERE id=2;COMMIT;



Shared vs Exclusive LocksA lock must be acquired (either implicitly or explicitly) before retrieving or modifying any data. There are two types of locks: - Shared (S) = Read only - Exclusive (X) = Read/Write

Table vs Row-Level LockingProperly defined indexes prevent toomany rows from unintentionally getting locked and also ensures optimal performance.


Deadlocked Doggies

Tony starts with 5 bones

Jake starts with 1 bone

Action #1: Jake steals a bone from Tony

Action #2: Tony, being a good doggie, gives Jake a bone so they’ll both have 3

Jake

Tony


Deadlocked DoggiesJake

Tony

## INITIAL SETUPCREATE TABLE `doggies` (`id` INT NOT NULL AUTO_INCREMENT,`name` VARCHAR(255) NOT NULL,`bones` INT NOT NULL,PRIMARY KEY (`id`)) ENGINE=InnoDB;

INSERT INTO doggies (id, name, bones) VALUES (1, "Tony", 5);INSERT INTO doggies (id, name, bones) VALUES (2, "Jake", 1);

## ACTION #1 # Jake steals a boneBEGIN;UPDATE doggies SET bones=bones-1 WHERE id=1; # X lock acquired on 1

## ACTION #2# Tony gives Jake a boneBEGIN;UPDATE doggies SET bones=bones+1 WHERE id=2; # X lock acquired on 2UPDATE doggies SET bones=bones-1 WHERE id=1;! # LOCK WAIT!COMMIT;

## ACTION #1 (continued)UPDATE doggies SET bones=bones+1 WHERE id=2;! # DEADLOCK!COMMIT;


How to Debug’emInspecting MySQL Status

Grepping Rails Logs

$ cat log/production.log | grep -B 40 Deadlock

mysql> SHOW ENGINE INNODB STATUS;

mysql> SHOW PROCESSLIST;

mysql> EXPLAIN SELECT ...;


How to Debug’em------------------------LATEST DETECTED DEADLOCK------------------------130204 13:10:32*** (1) TRANSACTION:TRANSACTION 214FC, ACTIVE 12 sec starting index readmysql tables in use 1, locked 1LOCK WAIT 3 lock struct(s), heap size 376, 2 row lock(s), undo log entries 1MySQL thread id 182, OS thread handle 0x111c8c000, query id 120976 localhost 127.0.0.1 root UpdatingUPDATE doggies SET bones=bones-1 WHERE id=1*** (1) WAITING FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FC lock_mode X locks rec but not gap waitingRecord lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000001; asc ;; 1: len 6; hex 0000000214fb; asc ;; 2: len 7; hex 7900000de72774; asc y 't;; 3: len 4; hex 546f6e79; asc Tony;; 4: len 4; hex 80000004; asc ;;

*** (2) TRANSACTION:TRANSACTION 214FB, ACTIVE 44 sec starting index readmysql tables in use 1, locked 13 lock struct(s), heap size 376, 2 row lock(s), undo log entries 1MySQL thread id 184, OS thread handle 0x111e2b000, query id 120977 localhost 127.0.0.1 root Updating## CONNECTION #1# Jake steals a bone (cont..)UPDATE doggies SET bones=bones+1 WHERE id=2*** (2) HOLDS THE LOCK(S):RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FB lock_mode X locks rec but not gapRecord lock, heap no 2 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000001; asc ;; 1: len 6; hex 0000000214fb; asc ;; 2: len 7; hex 7900000de72774; asc y 't;; 3: len 4; hex 546f6e79; asc Tony;; 4: len 4; hex 80000004; asc ;;

*** (2) WAITING FOR THIS LOCK TO BE GRANTED:RECORD LOCKS space id 0 page no 1130 n bits 72 index `PRIMARY` of table `deadlocks`.`doggies` trx id 214FB lock_mode X locks rec but not gap waitingRecord lock, heap no 3 PHYSICAL RECORD: n_fields 5; compact format; info bits 0 0: len 4; hex 80000002; asc ;; 1: len 6; hex 0000000214fc; asc ;; 2: len 7; hex 7a00000cf122d8; asc z \" ;; 3: len 4; hex 4a616b65; asc Jake;; 4: len 4; hex 80000002; asc ;;

*** WE ROLL BACK TRANSACTION (2)


Best PracticesDefine proper indexes and use EXPLAIN SELECT to verify. For example, make sure to define multi-column indexes across [id, type] on our polymorphic models.

Most common method of deadlock exception handling is to retry but this should only be used a last resort.

Easiest fix is to rearrange queries to make sure updates occur before selects (not always possible).

You can also lock records manually at the beginning of a transaction when you know they’ll need updating.


Helpful Resources

http://dev.mysql.com/doc/refman/5.0/en/innodb-lock-modes.html

http://rhnh.net/2010/07/01/debugging-deadlocks-in-rails

http://www.chriscalender.com/?p=426












Intro to Deadlocks

Documents

Transcript of Intro to Deadlocks