YCSB++ benchmarking toolwtantisi/files/tablebenchmark-pdl11-talk.pdf · YCSB++ benchmarking tool...

YCSB++ benchmarking tool Performance debugging

advanced features of scalable table stores

Swapnil Patil M. Polte, W. Tantisiriroj, K. Ren, L.Xiao,

J. Lopez, G.Gibson, A. Fuchs *, B. Rinaldi * Carnegie Mellon University * National Security Agency

Swapnil Patil © November 11 http://www.pdl.cmu.edu/ 2

Importance of scalable table stores

•  For data processing and analysis •  For systems services (e.g., metadata in Colossus)


Growing complexity of table stores

Simple, lightweight complex, feature-rich stores •  Supports a broader range of applications •  Hard to debug performance issue and complex

component interactions

Growing set of HBase features

2008 2009 2010 2011+

RangeRowFilters Batch updates

Bulk load tools RegEx !ltering

Scan optimizations

HBASE release

Co-processors Access Control

⏏ ⏏

State of table store benchmarking YCSB: Yahoo Cloud Serving Benchmark[Cooper2010]

•  Modular design to test different table stores •  Great for CRUD (create-read-update-delete)

benchmarking, but not for sophisticated features


⏏ ⏏ Need richer tools for understanding advanced features in table stores …

This talk: YCSB++ tool


NEW EXTENSIONS IN YCSB++ Distributed, coordinated and multi-phase testing

Fine-grained, correlated monitoring using OTUS[Ren2011]

TABLE STORE FEATURES TESTED BY YCSB++ Batch writing Table pre-splitting Bulk loading

Weak consistency Server-side !ltering Fine-grained security

Tool released at http://www.pdl.cmu.edu/ycsb++

Talk Outline •  Motivation •  YCSB++ architecture •  Illustrative examples of using YCSB++ •  Summary and ongoing work



Original YCSB framework

Configurable workload generation to test stores •  API adaptor converts read(K) to hbase_get(K)

Storage Servers

HBASE

Workload Executor

Threads

Stats

API A

dapt

or

Workload Parameters

OTHER DBS


YCSB++ supports new table store

New DB adaptor for Apache Accumulo table store •  New parameters and workload executor extensions

Storage Servers

HBASE

Workload Executor

Threads

Stats

API A

dapt

or

Workload Parameters

ACCUMULO EXTENSIONS EXTENSIONS


Coordinated & multi-phase tests

ZooKeeper-based coordination & synchronization •  Enables heavy workloads and asymmetric testing

Storage Servers

HBASE

Workload Executor

Threads

Stats

API A

dapt

or

Workload Parameters

EXTENSIONS EXTENSIONS ACCUMULO

MULTI-PHASE

YCSB clients COORDINATION

Coordinated & multi-phase tests •  Distributed, multi-client tests using YCSB++

•  Allows clients to co-ordinate their test actions •  Rely on shared data structures in ZooKeeper •  Useful for testing weak data consistency

•  Multi-phase tests in YCSB++ •  Can construct tests comprising of different phases •  Built on ZooKeeper-based barrier-synchronization •  Used for understanding high-speed ingest features



Collective monitoring in YCSB++

Fine-grained resource monitoring using Otus[Ren2011]

•  Collects from YCSB, table stores, HDFS and /proc

Storage Servers

HBASE

Workload Executor

Threads

Stats

API A

dapt

or

Workload Parameters

EXTENSIONS EXTENSIONS

MULTI-PHASE

ACCUMULO

YCSB clients COORDINATION OTUS MONITORING

Talk Outline •  Motivation •  YCSB++ architecture •  Illustrative examples of using YCSB++

•  Case study: HBase and Accumulo •  Both are Bigtable-like table stores

•  Summary and ongoing work


Primer on Bigtable-like stores


Tablet Servers

HDFS nodes

Tablet TN

Memtable

Write Ahead

Log

Data Insertion 1

Sorted Indexed

Store Files

2

(Fewer) Store Files 3

(1) Incoming mutation logged in memory (unsorted order)

(2) MINOR COMPACTION Memtables written to sorted,

indexed store !les in HDFS

(3) MAJOR COMPACTION LSM-tree based !le

merging (in background)

Accumulo table store •  Started at NSA; now an Apache project

•  Built for high-speed ingest and scan workloads •  http://incubator.apache.org/projects/accumulo.html

•  New features in Accumulo •  Iterator framework for user-specified programs

placed in different stages of DB pipeline – E.g., Supports joins and stream processing

•  Also provides fine-grained cell-level access control


Before I talk about examples … YCSB++ provides

•  Abstractions to construct distributed, parallel tests – Has in-built tests that use these abstractions

•  Monitoring that collects and correlates system (store/FS/OS) state with observed performance

YCSB++ does not provide •  Root cause diagnosis of performance problems

– Merely points you to where you should look …


FEATURES TESTED BY YCSB++

Table bulk loading Batch writing

Weak consistency Table pre-splitting

Server-side !ltering Access control


ILLUSTRATIVE EXAMPLE

Table bulk loading

High-speed ingestion through minimal data migration

Need careful tuning and con!guration [Sasha2002]

⏏

⏏

Table bulk loading in action


Data !les

HFile HFile HFile HFile HDFS cluster

Tablet servers Hbase Hbase Hbase Hbase

(1) FORMAT … … existing data !les to store-!le

speci!c format using Hadoop

(2) IMPORT … … store !les into table

stores to make data available for users

Hadoop tool

8-phase bulk load test in YCSB++


Load data •  Load 48M rows

in existing table

Pre-load data •  Insert 6M rows

in empty table

Pf Pi M S M M Lf Li

Sleep •  Let servers !nish

balancing work

Measurement phase •  Light mix of Read/Update operations •  Interleaved to study performance over time

Multi-phase tests show variation


Pf Pi M S M Lf Li M

1

10

100

1000

0 60 120 180 240 300

ACCU

MULO

Read

Laten

cy (m

s)

Measurement phase time (s) 0 60 120 180 240 300 0 60 120 180 240 300

Uniformly low latency after store is steady (no inserts)

10x latency variation; lasts for a long time!

Monitoring rebalancing at servers


Pf Pi M S M Lf Li M

1

10

100

1000

0 300 600 900 1200 1500 Running time of the 8-phase test (sec)

StoreFiles

Tablets

Compactions

Let’s take a closer look at correlating performance with server-side state

Effect of server-side work on latency


1

10

100

1000

0 60 120 180 240 300

ACCU

MULO

Read

Laten

cy (m

s)

1

10

100

1000

0 60 120 180 240 300 360 Experiment Running Time (sec)

StoreFiles

Tablets

Compactions

StoreFiles and Tablets increase with splitting

Background compactions reduce number of store !les

YCSB++ helps study different policies


1

10

100

1000

10000

0 60 120 180 240 300

ACCU

MUL

O Rea

d Lat

ency

(ms)

Measurement Phase Running Time (Seconds)

R/U 1 (Phase 3) R/U 2 (Phase 6) R/U 3 (Phase 8)

1

10

100

1000

0 300 600 900 1200 1500 1800 ACCUMULO Experiment Running Time (sec)

StoreFiles

Tablets

Compactions

1

10

100

1000

10000

0 60 120 180 240 300

HBAS

E Rea

d Lat

ency

(ms)

Measurement Phase Running Time (Seconds)

1

10

100

1000

0 300 600 900 1200 1500 1800 HBASE Experiment Running Time (sec)


Table bulk loading Batch writing

Weak consistency Table pre-splitting



ILLUSTRATIVE EXAMPLE

Batching writes at clients

Improves insert throughput and latency Newly inserted data may not be immediately visible to others

⏏

⏏


Batching improves throughput

6 clients create 9 million 1-KB records on 6 servers •  Small batches: high client CPU utilization limits work •  Large batches: servers are saturated, limits benefit

0 10 20 30 40 50 60

10 KB 100 KB 1 MB 10 MB

Insert

s per

secon

d (10

00s)

Batch size

Hbase Accumulo


Weak consistency test in YCSB++

ZooKeeper-based multi-client coordination •  Clients use a shared producer-consumer queue to

communicate keys to be tested

Table store servers

ZooKeeper

YCSB++

Store client

Batch

YCSB++

Store client

1 2 3 4

CLIENT #1 CLIENT #2

Insert {K:V}

(106 records)

Enqueue K (sample 1% records)

Poll and dequeue K

Read {K}

Test setup details •  YCSB++ tests on 1% of keys inserted by C1

•  C1 inserts 1 million keys, C2 reads 10K keys •  Sampling avoids overloading ZooKeeper

•  R-W lag for key K = time required by C2 to read K successfully •  If C2 can’t read K in the first attempt, tries again •  Report the time lag for fraction of keys that need

multiple read()s



Batch writing causes time lag

Delayed writes may not be seen for ~100 seconds •  Batching implementations affect latency; YCSB++

helps understand differences

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 10 100 1000 10000 100000

Fra

ctio

n o

f re

qu

est

s

read-after-write time lag (ms)

(a) HBase: Time lag for different buffer sizes

10 KB ( <1%)100 KB (7.4%)

1 MB ( 17%)10 MB ( 23%)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 10 100 1000 10000 100000

Fra

ctio

n o

f re

quest

s

read-after-write time lag (ms)

(b) Accumulo: Time lag for different buffer sizes

10 KB ( <1%)100 KB (1.2%)

1 MB ( 14%)10 MB ( 33%)

0.001 0.01 100 10 1 0.1 Read-after-Write time lag (sec)

0.001 0.01 100 10 1 0.1 Read-after-Write time lag (sec)

Fracti

on of

requ

ests

Fracti

on of

requ

ests

HBASE lag for different buffer sizes ACCUUMULO lag for different buffer sizes


Batch writing Weak consistency Table bulk loading Table pre-splitting



OTHER DETAILS

ACM SOCC 2011 paper available

Poster session(s)

Future work •  Evolving YCSB++

•  Study additional table stores (Cassandra, MongoDB and CouchDB)

•  Test more features: Iterators and co-processors

•  Understanding table store features •  Cost-benefit tradeoff of different heuristics for

compactions on tablet servers •  Dynamo-style eventual consistency


Summary: YCSB++ tool •  For performance debugging & benchmarking

advanced features using extensions to YCSB

•  Two case-studies: HBase & Accumulo •  Download at http://www.pdl.cmu.edu/ycsb++


Weak consistency semantics Distributed clients using ZooKeeper

Fast insertion (pre-splits, bulk loads) Multi-phase testing (with Hadoop)

Server-side !ltering New workload generators and database client API extensions Fine-grained access control

YCSB++ benchmarking toolwtantisi/files/tablebenchmark-pdl11-talk.pdf · YCSB++ benchmarking tool...

Documents

Transcript of YCSB++ benchmarking toolwtantisi/files/tablebenchmark-pdl11-talk.pdf · YCSB++ benchmarking tool...