The JOURNEY

Adform explores Hadoop

Ramunas Urbonas @ Adform

Developer first

Development as a journey

High perspective talk

3 aspects

The JOURNEY

Direction / Planning / Equipment

It starts with a goal

Vision

Born of need

Maintenance costs

Time to market

Licensing costs

Hadoop

main data storage

alternative reporting

Attack your vision

why storage?

HDFS

Distributed

x 3

Auto-balancing

Can store big files

Resilient

50% 3%

Files vs Database

Multiple engines on same data

that brings us to...

Alternative reporting

Rich eco-system

HDFS

map-reduce

hive pig hbase

yarn & mr2

spark shark

impala

druid etc

Different purpose

Different SLAs

Emerging tools

Big community

Beta products

UI

Automation

Confident in vision

main data storage

alternative reporting

Building vision vs Maintaining

When left unattended

Narrow focus

Keep it fresh

The JOURNEY

Direction / Planning / Equipment

Travelling light

Climbing a mountain

Even harder for Adform

Linux courses

Java

Java backend administration

Memory management

Profiling

Garbage collection

Know Hadoop principles

More climbing ahead

Test everything

Consultants

Abstract

Technical calls

Time consuming

How > What

POC vs Production

The JOURNEY

Direction / Planning / Equipment

Commodity hardware

“Commodity hardware”

No SSD / Raids

Desktop cluster?

Our cluster - leftovers

6-7 years old

Still server machines

Electricity

Burning fuses

Outdated notion?

Tricky question

Best sport shoes?

Basketball?

Football / Jogging / Climbing

Common hardware

8Gb vs 300Gb

Our bottlenecks

Common bottlenecks

The JOURNEY

Continues...

Releasing products

Data imports

Growing team

Upgrading cluster

Impala / Shark

More business areas

Clear vision

Plan your learning

Careful hardware decissions

Thanks!