The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam...
Transcript of The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam...
![Page 1: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/1.jpg)
The Database Designer’s Modern-Day Cookbook
Preetam JinkaSoftware Engineer
Percona Live 2017
![Page 2: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/2.jpg)
VividCortex’s database monitoring application is the best way to improve your database performance, efficiency, and uptime. Supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora, VividCortex uses patented algorithms to reveal key insights, helping users fix performance problems before they impact customers. Say hello and see a demo, Booth #205.
We’re hiring!
![Page 3: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/3.jpg)
Topics
● Immutability
● Transactions & ACID
● Replication
3
![Page 4: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/4.jpg)
Topics
● Immutability
● Transactions & ACID
● Replication
But mainly about trade-offs.
4
![Page 5: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/5.jpg)
Immutability
5
![Page 6: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/6.jpg)
6
Immutability
Not changing something once it’s created.
![Page 7: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/7.jpg)
An immutable database?
Just use a log.
● Write optimized
● Transactional
● Everything else is just a read optimization, right?
● Space might become a problem...
7
![Page 8: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/8.jpg)
Something more realistic
● Two ways to update data.
○ In-place
○ Copy-on-write (the immutable approach)
8
In-place Copy-on-write
![Page 9: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/9.jpg)
Concurrency
Databases need to handle multiple readers and writers.
And they need to provide certain guarantees
(transactions).
9
![Page 10: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/10.jpg)
In-place updates with ARIES
Algorithms for Recovery and Isolation Exploiting Semantics
● Write-ahead logging
● Redo logs
● Undo logs
Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to
manage transactions.
10
![Page 11: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/11.jpg)
Copy-on-write with row versioning
● Systems like PostgreSQL create a copy of data when it needs to be changed.
● Immutability is inherently free of data races!
● But you need to get rid of old versions through vacuuming.
● You also need to manage the overhead of multiple versions.
● This is why systems like PostgreSQL don’t have an “undo log.”
11
![Page 12: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/12.jpg)
> … PG must do more work at commit time, right?
No. Commit and abort are both O(1). Where we pay the piper is in
having to run VACUUM to clean up no-longer-needed row versions.
This is a better design in principle, because the necessary maintenance
can be done in background processes rather than making clients wait
for transactions to finish. In practice, it's still pretty annoying,
just in different ways than Oracle's UNDO.
http://www.postgresql-archive.org/PG-and-undo-logging-td5850789.html
12
![Page 13: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/13.jpg)
Secondary indexes
● Secondary indexes need to point to the original row.
● For MySQL, you just need the primary key.
● For PostgreSQL, you need the primary key and a version.
○ Primary keys aren’t unique because there could be different row versions!
13
![Page 14: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/14.jpg)
What’s better?
It’s a trade-off!
14
![Page 15: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/15.jpg)
Uber’s Migration to MySQL
Uber migrated from PostgreSQL to MySQL. Their reasons:● Inefficient architecture for writes
● Inefficient data replication
● Issues with table corruption
● Poor replica MVCC support
● Difficulty upgrading to newer releases
https://eng.uber.com/mysql-migration/In other words: PostgreSQL probably didn’t have a set of trade-offs that worked well for them.
15
![Page 16: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/16.jpg)
Transactions & ACID
16
![Page 17: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/17.jpg)
17
Transactions are complicated!
MVCC
ARIES
UNDO logs REDO logs
Row locks
Isolation levelsACID
Write skew
Snapshot isolation
Write-ahead log
Consistency
Commit
![Page 18: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/18.jpg)
This is about making them simple.
18
![Page 19: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/19.jpg)
ACID transactions
19
● Atomicity
● Consistency
● Isolation
● Durability
![Page 20: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/20.jpg)
What do you need for ACID?
1. A snapshot view of the data
2. Durable, atomic writes
● Immutability makes #1 easier.
● Single writer makes #2 easier.
You can get both from a log.
20
![Page 21: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/21.jpg)
...but transactions & ACID in the real world tend to be much more complicated...
...because not everything uses immutability, and most systems are not single writer.
21
![Page 22: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/22.jpg)
Replication
22
![Page 23: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/23.jpg)
23
Replication
Copying data to several places.
![Page 24: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/24.jpg)
24
Replication choices
● Asynchronous
● Synchronous
○ Semi-synchronous is another option with MySQL
As usual… trade-offs.
![Page 25: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/25.jpg)
Replication spectrum
25
Synchronous AsynchronousSemi-sync
Most guarantees
Leastflexible
Least guarantees
Mostflexible
![Page 26: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/26.jpg)
● Requires coordination at the master○ Coordination can get complicated...
● There’s waiting involved○ Replica lag doesn’t exist
● Safe
Synchronous
26
![Page 27: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/27.jpg)
● You need some sort of “master” or “leader” server handling coordination.
● Leader election and consensus are ways of selecting a master automatically○ Paxos is a consensus algorithm that’s widely used. MySQL Group Replication uses a variant in
their multi-master approach.
Synchronous
27
![Page 28: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/28.jpg)
Synchronous
28
Master
Replica Replica
Master pushes to replicas
![Page 29: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/29.jpg)
Asynchronous
29
Master
Replica Replica
Replicas pull from the master
![Page 30: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/30.jpg)
Asynchronous
30
● Less coordination
● Pro: Master doesn’t wait for replicas.○ It’s faster because there’s no waiting.
● Con: Master doesn’t wait for replicas.○ Replicas can fall behind.
● Delayed replicas can be really useful when things go wrong!
![Page 31: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/31.jpg)
Disaster recovery with a delayed replica
31
DigitalOcean’s April 2017 Outage:
“Within three minutes of the initial alerts, we discovered that our primary
database had been deleted. Four minutes later we commenced the recovery
process, using one of our time-delayed database replicas. Over the next four
hours, we copied and restored the data to our primary and secondary replicas.”https://www.digitalocean.com/company/blog/update-on-the-april-5th-2017-outage/
![Page 32: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/32.jpg)
The diagrams look similar but they’re very different.
32
Master
Replica Replica
Replicas pull from the master
Master
Replica Replica
Master pushes to replicas
![Page 33: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/33.jpg)
Use the right tool for the job.
33
![Page 34: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/34.jpg)
Final thoughts
34
● There are trade-offs everywhere.
● You’re not limited to a single technology or implementation.
● Things keep getting more exciting.
![Page 35: The Database Designer’s Modern-Day Cookbook - · PDF fileModern-Day Cookbook Preetam Jinka ... Systems like MySQL, Oracle, SQL Server, DB2 use something like ARIES to manage transactions.](https://reader031.fdocuments.in/reader031/viewer/2022022501/5aa637c37f8b9a2f048e7182/html5/thumbnails/35.jpg)
Questions?
35