Appnexus scalability
-
Upload
sean-hull -
Category
Technology
-
view
8.002 -
download
0
description
Transcript of Appnexus scalability
1
Database ScalabilitySweep the Query Log & Keep it Clean
2
What Is Scalability?
Webserver tier - Apache, Nginx
Object caches - Memcache
Search servers - Sphinx
Queueing servers - SQS, RabbitMQ
3
Why is DB hard to Scale?
Relational Databases
Single authoritative master
Scaling a write db is hard
Durability
SQL code weight non-obvious
4
ORM Advantages
Middleware, multi-database support
Reusable code, encapsulation & abstraction
Time saving - auto generated model objects
Built for wide audience, reduce need for SQL
Sacrifice a bit of efficiency for code agility
5
ORMs + ScalabilityNegates standard optimization techniques
Difficult to divorce after you are wedded
Limit query tuning at the bare metal
Wide audience not high-scale applications
Bad at deducing JOINs
Fetch 20 cols when you need only 2?
“death by a thousand queries” - Laurie Voss
6
ORM ExampleEx. from http://mattiasgeniar.be
<?php
// First, get all the companies in your database
$companies = $this->getAllCompanies();
$totalValue = 0;
foreach ($companies as $company) {
// For each company there is, retrieve the total value of all their orders
$orders = $company->getOrders();
foreach ($orders as $order) {
$totalValue += (float) $order->getValue();
}
echo "This company made us ". $totalValue ." euro already.";
}
?>
7
ORM Example<?php
// First, get all the companies in your database
SELECT * FROM "company";
$totalValue = 0;
foreach (...) {
// For each company there is, retrieve the total value of all their orders
SELECT * FROM "order" WHERE companyid = $companyid;
foreach (...) {
$totalValue += $row->value;
}
echo "This company made us ". $totalValue ." euro already.";
}
?>
8
ORM Example as SQL
SELECT SUM(o.VALUE) AS TotalValue, c.name AS CompanyName
FROM company AS c
LEFT JOIN "order" AS o ON o.companyid = c.companyid
GROUP BY o.companyid;
9
ORM Articles
ORM is an anti-pattern - Laurie Voss
Bad ORM is infinitely worse than bad SQL - Mattias Geniar
Case against ORM frameworks - Todd Hoff
5 things toxic to scalability - Sean Hull
10
Database Tuning Process
Identify heavy queries in slow log
Find queries in code
Rewrite, run explain & profile
Benchmark with large dataset
Deploy fixed query
11
Being Reactive?
Sweep out the slow query log
Rewrite queries
12
Being Proactive
Keep slow query log quiet
With new code deploys, new slow queries will pop up
Tune early, tune often
13
Use Profiling
set profiling=1;
<run query>
show profile for query 1;
14
Enable Session Profiling(sean@localhost:mysql.sock) [sakila]> set profiling = 1;
Query OK, 0 rows affected (0.00 sec)
(sean@localhost:mysql.sock) [sakila]> select customer_id, last_name from customer where customer_id = 5;
+-------------+-----------+
| customer_id | last_name |
+-------------+-----------+
| 5 | BROWN |
+-------------+-----------+
1 row in set (0.00 sec)
(sean@localhost:mysql.sock) [sakila]> select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5;
+-------------+-----------+
| customer_id | last_name |
+-------------+-----------+
| 5 | BROWN |
+-------------+-----------+
1 row in set (0.00 sec)
15
Anatomy of a Profile(sean@localhost:mysql.sock) [sakila]> show profiles;
+----------+------------+--------------------------------------------------------------------------------+
| Query_ID | Duration | Query |
+----------+------------+--------------------------------------------------------------------------------+
| 1 | 0.00015500 | select customer_id, last_name from customer where customer_id = 5 |
| 2 | 0.00054700 | select SQL_NO_CACHE customer_id, last_name from customer where customer_id = 5 |
+----------+------------+--------------------------------------------------------------------------------+
2 rows in set (0.00 sec)
(sean@localhost:mysql.sock) [sakila]> show profile for query 1;
+--------------------------------+----------+
| Status | Duration |
+--------------------------------+----------+
| starting | 0.000062 |
| checking query cache for query | 0.000020 |
| checking privileges on cached | 0.000019 |
| sending cached result to clien | 0.000033 |
| logging slow query | 0.000010 |
| cleaning up | 0.000011 |
+--------------------------------+----------+
6 rows in set (0.00 sec)
16
Non Cached Query Profile
(sean@localhost:mysql.sock) [sakila]> show profile for query 2;
+--------------------------------+----------+
| Status | Duration |
+--------------------------------+----------+
| starting | 0.000052 |
| checking query cache for query | 0.000103 |
| Opening tables | 0.000025 |
| System lock | 0.000014 |
| Table lock | 0.000022 |
| init | 0.000043 |
| optimizing | 0.000022 |
| statistics | 0.000095 |
| preparing | 0.000030 |
| executing | 0.000011 |
| Sending data | 0.000045 |
| end | 0.000013 |
| end | 0.000010 |
| query end | 0.000012 |
| freeing items | 0.000015 |
| closing tables | 0.000015 |
| logging slow query | 0.000010 |
| cleaning up | 0.000010 |
+--------------------------------+----------+
18 rows in set (0.00 sec)
17
Single Column Index
create index d_idx on a (d);
select email from a where d=99;
18
Multi-column Index
create index def_idx on a (d, e, f);
SELECT email FROM a WHERE d=1 and e=2 and f=3;
19
Left-most Prefix Index
create index abc_idx on a (d, e, f);
SELECT name FROM a where d = 1 and e = 2;
SELECT name FROM a WHERE d=1 and e=2 and f=3;
SELECT name FROM a WHERE e=2;
20
Covering Index
create index cover_id on (id, name);
SELECT name FROM a WHERE id = 5;
21
Use EXPLAIN
Illustrates database engine’s path to the data
Shows logical & physical I/Os
Shows sorting, temp tables, index usage
22
EXPLAIN - no index
(sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = '[email protected]';
+----+-------------+----------+------+---------------+------+---------+------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------+------+---------+------+------+-------------+
| 1 | SIMPLE | customer | ALL | NULL | NULL | NULL | NULL | 541 | Using where |
+----+-------------+----------+------+---------------+------+---------+------+------+-------------+
1 row in set (0.00 sec)
23
EXPLAIN - basic index
(sean@localhost:mysql.sock) [sakila]> create index cust_email on customer(email);
(sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = '[email protected]';
+----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+
| 1 | SIMPLE | customer | ref | cust_email | cust_email | 153 | const | 1 | Using where |
+----+-------------+----------+------+---------------+------------+---------+-------+------+-------------+
1 row in set (0.02 sec)
24
EXPLAIN - covering index
(sean@localhost:mysql.sock) [sakila]> create index cust_email_ln on customer(email, last_name);
(sean@localhost:mysql.sock) [sakila]> explain select last_name from customer where email = '[email protected]';
+----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+
| 1 | SIMPLE | customer | ref | cust_email_ln | cust_email_ln | 153 | const | 1 | Using where; Using index |
+----+-------------+----------+------+---------------+---------------+---------+-------+------+--------------------------+
1 row in set (0.00 sec)
25
Test With Large Datasets
As data grows, indexing more crucial
Test with million row tables
Slow desktop or test boxes?
26
Can Devops Help?
Teamwork - way of working
Less siloing of departments
DBAs helping identify queries
DBAs helping rewrite queries
Devs on top of new code deploys
Clean slow-log as new queries popup
27
About Sean Hull
Join 8000 & please follow me: @hullsean
www.iheavy.com/blog
www.iheavy.com/signup-scalable-startups-newsletter
mobile: +1-917-442-3939