Large Scale NoSQL DB Migration Under Fire - Aerospike
Transcript of Large Scale NoSQL DB Migration Under Fire - Aerospike
![Page 1: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/1.jpg)
A E RO S P I K E US E R S UM M I T 2018
Large Scale NoSQL DB Migration Under Fire
Ido Barkan
Software Architect
AppsFlyer
![Page 2: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/2.jpg)
2 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Ag
en
da
Motivation and limitations
What is AppsFlyer?
What have we learned?
War plan
![Page 3: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/3.jpg)
3 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
What is AppsFlyer?
~25B events per day
![Page 4: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/4.jpg)
4 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Installs DB- Zoom in
app installs writer
2K write IOPS
app events enrichments 180k read
IOPS
![Page 5: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/5.jpg)
5 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
We need to replace CouchBase
● Scale out is out of control(45 r3.4xl instances)
● XDCR can’t keep up● Daily backups take more than a day● Paid support is not good enough
Chosen solution: Aerospike
![Page 6: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/6.jpg)
6 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Starting point
● # of records: ~ 2,000,000,000 (repl. factor 3)● Read intensive: 180K read + 2K write IOPS● # of machines: 45 r3.4xlarge● Cost: ~ 30,000 $ per month● Time frame: a few months.Mission critical
![Page 7: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/7.jpg)
7 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Why is data migration harder than in a relational DB?
Relational DB
NoSql DB
Data types conventional
blob, JSON, specialized bins
Vendors a few 100+
Query Language
SQL N1QL, SQL, JavaScript, HTTP/REST-- creating views/ MapReduce style
Schema Yes Maybe
Migration tools
Yes DIY
![Page 8: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/8.jpg)
8 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Rules of the game
No DowntimeNo data loss
![Page 9: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/9.jpg)
9 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
War plan- easy part
● Write new installs/updates to BOTH● Continue reading from the OLD DB Application
![Page 10: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/10.jpg)
10 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Breathing (testing) time
test test test● Sizing RAM/Disk● Expected load● Metrics● Alerting● Logs● Backups
![Page 11: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/11.jpg)
11 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Easy part (Cont.)● Write new installs/updates to NEW DB● Read from BOTH DBs
● A fraction of records (not VIP)
● Do NOT delete yet!● Have metrics on
misses
![Page 12: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/12.jpg)
12 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Breathing time
Test Test Test● Sizing (again)● Expected load
(with reads)● Metrics (again)
![Page 13: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/13.jpg)
13 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Is that it?
If your data is short lived- you are mostly done.
Otherwise...
![Page 14: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/14.jpg)
14 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Dumperview (query)dump keys/records
Kafka as a buffer
Loader-> write(do not overwrite)
values
‘Manual’ partMigrating the long tail
![Page 15: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/15.jpg)
15 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
Detour: DB views
function (doc, meta) {emit(meta.expiration, doc.id);}
key (sorting) value
Map function (JS)
● Record itself is NOT a part of the view
![Page 16: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/16.jpg)
16 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
‘Manual’ partMigrating the long tail
Dumperview (query)dump keys/records
Kafka as a buffer
Loader-> write(do not overwrite)
values
![Page 17: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/17.jpg)
17 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
New Cluster
● # of records: ~ 2,000,000,000 ● 2AZ- repl. factor 2 in each one● IOPS, same and growing● # of machines: 2 x 5 i3.4xlarge● Cost: ~ 10,000 $ per month (⅓)
![Page 18: Large Scale NoSQL DB Migration Under Fire - Aerospike](https://reader031.fdocuments.in/reader031/viewer/2022012915/61c5367c361488371d1297a5/html5/thumbnails/18.jpg)
18 AEROSPIKE USER SUMMIT | Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc
What have we learned?
1. Plan, Plan, Plan.2. Go easy: test a sample before you start
(1 app / API / Customer).3. Breath (read: test) between stages.4. Do NOT delete until you have to.5. slower -> safer (-> $$$).