The Multiple Uses of HBase
description
Transcript of The Multiple Uses of HBase
![Page 1: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/1.jpg)
The Multiple Uses of HBase
Jean-Daniel Cryans, DB Engineer @ SU@jdcryans, [email protected]
Berlin Buzzwords, Germany, June 7th, 2011
![Page 2: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/2.jpg)
Overview
1. Why HBase2. How to X in HBase
LOLcat to keep you awake, thanks to http://icanhascheezburger.com/
![Page 3: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/3.jpg)
Why HBase
1. Big Data™HBase scales as you add machines
2. Affinity with HadoopSame configuration files, scripts, language
3. About to write something similar anywayFiles in HDFS are immutable, then it turtles all the way
down4. Simple concepts
Master-slave, row-level ACID
![Page 4: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/4.jpg)
How to Use HBase
1. General concerns:1. Query patterns (direct key access, joins, etc)
duplicate data, join at the application level, embed in families
2. Read/Write proportionsUsually dictates the amount of RAM given to the MemStores and the block cache
3. Working dataset sizeIf it doesn’t fit in the block cache, it’s usually better to skip it. Hi random workloads!
![Page 5: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/5.jpg)
How to: CRUD
1. Straight up Create, Read, Update, Delete2. HBase becomes a general store, one table
per class, usually one family3. When crafting row keys, consider
well distributed keys (UUID) VS incrementing keys
![Page 6: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/6.jpg)
How to: CRUD
1. stumbleupon.com1. About 100 tables used for our products.2. Families usually called “d”, saves on memory and
disk.3. DAO layer is the same for MySQL and HBase.4. Access done through Thrift, one per region
server. Topology stored in ZK.2. yfrog.com
1. Whole site served out of HBase through Thrift
![Page 7: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/7.jpg)
How to: Big Objects
1. Storing web pages, images, documents.2. The default configuration is usually not
suitable, memstore and region sizes are too small.
3. If possible, compress the data before sending into HBase. Most of the time that’s already done with images.
![Page 8: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/8.jpg)
How to: Big Objects
1. yfrog.com / imageshack.us1. Every yfrog image and some imageshack images end
up in a heterogeneous cluster of >50 desktop-class machines.
2. Serving done through REST servers2. stumbleupon.com
1. We crawl every website that we recommend and store it in HBase for later processing.
2. About to migrate from storing thumbnails into Netapp to HBase more cost effective.
![Page 9: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/9.jpg)
How to: Counters
1. Count page views, accesses, actions, etc.2. HBase supports atomic “compare-and-swap”
since 2009, incrementColumnValue is one.3. Pre-split regions in order to have a few per
region server, it should not split
![Page 10: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/10.jpg)
How to: Counters
1. facebook.com1. Facebook Insights is a new product that offers real-
time analytics for developers and website owners.2. Massive amounts of counters are incremented per
second. See Jonathan Gray’s talk tomorrow for more!2. stumbleupon.com
1. Counters used to keep track of everything our users do and AB testing.
2. Mix of “sloppy” counters applied asynchronously and synchronously.
![Page 11: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/11.jpg)
How to: Archive
1. Storing logs, time series, events.2. Only the most recent data is accessed.3. Regions should be big, try to keep row key
distribution even.4. Often impossible/impractical to MapReduce
archive tables, requires skipping rows.
![Page 12: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/12.jpg)
How to: Archive
1. stumbleupon.com1. Using OpenTSDB to monitor all our machine and
systems.2. Storing 2.5B data points per week, more are
added on a daily basis.2. mozilla.com’s Socorro
1. Firefox crashes are stored in HBase for processing.
![Page 13: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/13.jpg)
How to: Batch
1. Good old MapReduce2. 1 region = 1 map, can’t go lower without
knowledge of the key space3. Tools you can use: Hive, Pig, Cascading4. Speculative execution should be disabled
when reading/writing to HBase5. Block caching is often useless when scanning,
either disable completely or on the Scan
![Page 14: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/14.jpg)
How to: Batch1. stumbleupon.com
1. Hive is usually used by business analysts to combine MySQL, logs and HBase data and by developers seeking fast answers about their big data.
2. Cascading is used by engineers to write data pipelines for the ad system.
3. Pure MR are used by the research team for complicated machine learning jobs for our recommendation engine.
2. twitter.com1. A copy of the tweets table is stored in HBase and
processed, loaded via Elephant Bird.
![Page 15: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/15.jpg)
How to: Batch & Real-time
1. Respecting SLAs versus trying to feed MR jobs with IO, best to avoid.
2. One option is to have 2 classes of clusters, one live and one MR.
3. Else, configure to have a few slots as possible. If scanning data that’s served live, better to avoid block caching.
![Page 16: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/16.jpg)
Infrastructure EngineerDatabase AdministratorSite Reliability EngineerSenior Software Engineer
(and more)http://www.stumbleupon.com/jobs/
Mandatory “We’re Hiring!” slide
Help us building the best recommendation engine!
![Page 17: The Multiple Uses of HBase](https://reader036.fdocuments.in/reader036/viewer/2022062521/568168b4550346895ddf8515/html5/thumbnails/17.jpg)
Questions?