2015-08-22 Stream billions of data to Google Big Query
13
-
Upload
aftership -
Category
Technology
-
view
323 -
download
0
Transcript of 2015-08-22 Stream billions of data to Google Big Query
Shipment tracking APISupporting 270 290+ carriers
Free Shipping APIsSingle API for 5 carriers
How do we find you?
Google BigQuery is great
- Charge only when you do query
- Super cheap in storage
- Super cheap in sending the data
Perfect for log everything
3 billions records / month
1,200 records / sec
API Log BigQuery
Goal
API Log BigQuery
It does not work!
- http RESTful- BigQuery API call limit
API Log Redis Streamer BigQuery
How do we solve?
Why Redis?
- TCP persistent connection- Fast, really FAST
What does the STREAMER do?
- Read data from redis- Send BATCH data to BQ- Pre / Post process data
Demo
http://www.slideshare.net/aftership
FAQ
OPEN SOURCEhttps://github.com/AfterShip/demo_streamer2bigquery
