HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer...

29
HBase

Transcript of HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer...

Page 1: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

HBase

Page 2: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

2

OUTLINE

• Basic• Data Model• Implementation

– Architecture of HDFS• Hbase Server• HRegionServer

Page 3: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Basic

• HBase directly uses or subclasses the parent Hadoop implementation

Page 4: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

4

Basic

Linux Linux LinuxLinux

Page 5: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Basic

• DataBase of problem:– Grown of Data – Complexity of install and maintain

• Mutil-RDBMS of poblem:(for nodes )– JOIN– not effective– rebalance

Solution : Relational DataBase Management System(RDBMS)

Solution : NoSQL DataBase

Page 6: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Basic

• NoSQL DataBase :– Distributed– Scalability– Easy to use (EX:put, get ,alter etc.)

Page 7: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Basic

• List of NoSQL:– OpenSource

• HBase (Yahoo!)• Cassandra (Facebook)• SimpleDB (Amazon)

– Commercial• BigTable (Google)

Page 8: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Basic• Hbase:

– Hadoop’s DataBase.– Reversion of 0.20.6 released– Usage with Map/Reduce

Page 9: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

9

OUTLINE

• Basic• Data Model• Implementation

– Architecture of HDFS• Hbase Server• HRegionServer

Page 10: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Table• member : Row , Column, TimeStamp

Row key TimeStamep Column”Contents”

“com.yahoo.news.tw”

t3 “ 我研發水下 6 千公尺機器人”

t2 “ 蚊子怎麼搜尋人肉”

t1 “… Wang 40…”

“com.cnn.www” t1 “ 用腦波「發聲」 ”

Page 11: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Table• Add column

”Anchor”Row key

TimeStamep ”Contents”

“com.yahoo.news.tw”

t3“ 我研發水下 6 千

公尺機器人”

t2 “ 蚊子怎麼搜尋人肉”

t1 “… Wang 40…”

“com.cnn.www” t1 “ 用腦波「發聲」

• Add< Family, Label>

Page 12: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Table

Row key TimeStamep ”Contents” ‘’ Anchor’’

“com.yahoo.news.tw”

t5“Anchor:tech” “Silvia”

t4 “Anchor:sports” “Eric”

t3 “ 我研發水下 6 千公尺機器人”

t2 “ 蚊子怎麼搜尋人肉”

t1 “… Wang 40…”

“com.cnn.www” t1 “ 用腦波「發聲」 ”

‘’Anchor_tech’’ ‘’ Anchor_sports’’

Silva Eric

Page 13: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Region

Row key TimeStamep ”Contents” ‘’ Anchor’’

“com.yahoo.new

s.tw”

t5 “Anchor:tech” “Silvia”

t4 “Anchor:sports” “Eric”

t3 “ 我研發水下 6 千公尺機器人”

t2 “ 蚊子怎麼搜尋人肉”

t1 “… Wang 40…”

“com.cnn.www” t1 “ 用腦波「發聲」

“com.abc.www”

“com.def.www”

region1

region2

Region1(com.yahoo.new.tw,com.def.www>,ID

Express: Region(start row key, end row key>& identifier

Page 14: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Sort

• Sort by row key – byte-ordered

• Add label on family column

Page 15: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

LockingRow key Time

Stamep ”Contents” ‘’ Anchor’’

“com.yahoo.new

s.tw”

t5 “Anchor:tech” “Silvia”

t4 “Anchor:sports” “Eric”

t3 “ 我研發水下 6 千公尺機器人”

t2 “ 蚊子怎麼搜尋人肉”

t1 “… Wang 40…”

“com.cnn.www” t1 “ 用腦波「發聲」

“com.abc.www”

“com.def.www”

User1

update

User2

update

User3

update

User4

update

Page 16: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

16

OUTLINE

• Basic• Data Model• Implementation

– Architecture of Hbase• Hbase Server• HRegionServer

Page 17: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Architecture of Hbase

NN: NameNodeDN: DataNodeHM: HmasterHR:HRegion

Cluster

HDFSClient

NN DN DN

DN DN DN

HM HR HR

HRHR HR

ZooKeeper

Page 18: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

rebalance

• a single host grows the regions– it split a row into two new regions of

approximately equal size.• Until not across threshold

• automatic

Page 19: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Hbase Master

• The master node is lightly loaded.• assignment of the replacement daughters• Recovering regionserver failures.

Page 20: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

RegionServer

• carry zero or more regions • client read/write/scan requests

– Random access• Automatic split regions• Send HeartBeat to Master

Page 21: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

RegionInfo.

• Region of metadata– the current list, state, recent history, and location

of all regions afloat on the cluster.

{NAME => ‘docs’, FAMILIES => [{NAME => ‘cache’, COMPRESSION => ‘NONE’, VERSIONS => ’3′, TTL => ′ 2147483647′, BLOCKSIZE => ’65536′, IN_MEMORY => ‘false’, BLOCKCACHE => ‘false’}

Page 22: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

.MATA.

Page 23: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

HBase in operation• memory size of 256MB and each row is 1KB size

-ROOT-

.META.

useregion

• 1.8 x 1019 (264) bytes of user data

2.6 x 105 META regions

6.9 x 1010 user regions

Page 24: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

HBase in operation

NN: NameNodeDN: DataNodeHM: HmasterHR:Regionsever

Cluster

HBase Client NN DN DN

DN DN DN

HM R R

RR R

ZooKeeper

ROOT META

Request

consult

Step 3.User region

Step 1.

Step 2

• Read Requests - Step 1.location of -ROOT- - Step 2.location of the .META. Region - Step3.user region space

Page 25: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

HBase in operation

NN: NameNodeDN: DataNodeHM: HmasterHR:Regionsever

Cluster

HBase Client

NN DN DN

DN DN DN

HM R R

RR R

ZooKeeper

Interacts with RegionServer

• Read Requests -clients cache save information of ROOT , META and User Region

Page 26: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

HBase in operation

Interacts with RegionServer

HBase Client

HLog

• table Region server of state

Region Serser

Hstore RegionRegion

Hstore

Hstore

HFileHFileHfile

MemStore

Page 27: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

HBase in operation

RegionServer

HBase Client

HLog

• Client request to save data in table

Region Serser

Hstore RegionRegion

Hstore

Hstore

HFileHFileHfile

MemStore

Page 28: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.
Page 29: HBase. OUTLINE Basic Data Model Implementation – Architecture of HDFS Hbase Server HRegionServer 2.

Hbase of characteristic

• Fault tolerance• Batch processing• Automatic partitioning• Scale linearly with new nodes