2 one spot redshift bigdatacamp 1.02

Copyright © 2013 OneSpot, Proprietary & Confidential 1

Amazon Redshift:How we managed 300 billion rows with no DBA

Matt Cohen

Founder & [email protected]

December 10th, 2013

What is OneSpot?

• OneSpot is a content advertising platform that distributes content as ads that people want to click on.– Fortune 2000 clients

– Realtimead exchange bidding

– Adaptive machine learning

– Seed funded until $5.3M Series A last month

• Big data, big analysis


What is Redshift?

1. When light from a receding object appears

shifted to the red end of the spectrum

– A consequence of the expanding universe.

2. A cheap, fast, Petabyte-scale, managed

SQL data warehouse service from Amazon

Web Services

– A consequence of the expanding cloud ecosystem


Why Redshift?

• Cheap

• Fast

• Petabyte-scale

• Managed Service

• SQL

• Data Warehouse

• From AWS


SQL Data Warehouse

• Based on the commercial ParAccel database– Which is based on Postgres

• Standards-based tools and knowledge

• Built for data warehousing– Column-oriented

– Cluster architecture

– Read optimized

– No relational integrity

– Almost no SQL extensions


SQL Data Warehouse

• Column-oriented


SQL Data Warehouse

• Column-oriented

• 11 different compression techniques


SQL Data Warehouse

• Cluster architecture


SQL Data Warehouse

• Read optimized

– Large block size (1MB)

– Data replication

• 2x live, 1x S3

• No relational integrity

– No indexes:

sort and distribution keys

• Almost no SQL

extensions


Fast = Cheap

• Starts with 1 XL node

– 85¢ an hour ($620/month) on demand

– 50¢ an hour ($365) 1 year reserved

• Benchmarks say:

– Scales linearly

– 5-10x faster than Hadoop/Hive


Petabyte scale

• Up to

– 32 XL nodes (64 Terabytes)

– 100 8XL nodes (1.6 Petabytes)


Managed Service from AWS

• Managed Service

– Incredibly easy

– Nice UI

– Most SQL tools

• From AWS

– Free data transfer

– Easy load from S3

– Use AWS Data Pipeline


The TL;DR

• Pros

– Standard SQL

– Super easy

– Very fast

– Affordable

– Integrates with AWS

– No DBA

– No Sysadmin

• Cons

– Standard SQL

– Almost no SQL

extensions

– Best with Star Schema

• Big joins can be slow

– No MapReduce

– Fixed columns

– Consistency

– 1.6 Pbyte limit



Amazon Redshift:How we managed 300 billion rows with no DBA

Matt Cohen

Founder & [email protected]

December 10th, 2013

2 one spot redshift bigdatacamp 1.02

Technology

Transcript of 2 one spot redshift bigdatacamp 1.02