Building hadoop based big data environment

Building Hadoop Based Big Data Environment

Evans Ye @ TWHUG

2013/12/14

• Evans Ye @

• Dumbo Team

• http://dumbointaiwan.blogspot.tw/

Who am I

• Building your own Hadoop version

• Hadoop Deployment

• Hadoop release engineering

• The development environment

• Bigtop puppet

Agenda

• Add your own patch at any time– From community perspective, they need to take care about

backward complicity,which need much more time and effort on it.

• Fetch official patches in to current adopted version– You may not upgrade your Hadoop version frequently,

But there’s a specific need for that patch.

• Flexibility, Business needed features

Why Build our own version

As a Beginner

What’s your work?Build Hadoop Infrastructure

I thought you just need to yum install Hadoop.….

• git clone

• Make some changes

• Builde binary tarball

Brute force

core-site.xmlhdfs-site.xml

mapred-site.xml…

How to do version control?

Bigtop

• Apache Hadoop App developers: – Run pseudo-distributed Hadoop cluster to test your code on.

• Vendors: – Build your own Apache Hadoop distribution, customized from

Apache Bigtop bits.

• Packaging, Deployment, Integration Testing

How bigtop helps you

• Ubuntu 10.10

• CentOS 5/6

• Fedora 18

• Mageia 1

• openSUSE 12.2

Supported Linux Distro

• Build hadoop-common (see BUILDING.txt)

– hadoop-common$ mvn package –Pdist,docs,src,native -Dtar

• Prepare your src tar in bigtop

• Bigtop$ make hadoop-rpm

Hadoop Deployment

• Hadoop related config– core-site.xml– hdfs-site.xml– mapred-site.xml– log4j.properties– hadoop-env.sh– fair-scheduler.xml– rack-topology– hadoop-metrics.properties– taskcontroller.cfg

Configuration files

• Hadoop related file and directory– Namenode metadata

• /name/1, /name/2– Datanode

• /data/1, /data/2 , /data/3 , /data/4– Tasktracker

• /mapred/1/local, /mapred/2/local– …

Local Directories

More hadoop ecosystem

• Lots of nodes need to be configured

• Less human involved, less mistake made

• Configuration changed quite often– adjust fair scheduler– enable/disable short circuit– try more performance improvement configurations

Problems to solve

Hadooppet

• A IT automation tool to help system administrators automate the many repetitive tasks

• You need to only define the desired state

What is puppet ?

• A general hadoop cluster deployment tool based on puppet

• Kerberos / ldap auto configured

• A set of hadoop / kerberos management tool

• A set of sanity check scripts for trend hadoop related services

• Manage configuration on puppetmaster

What is Hadooppet ?

• Abstract environment specific configurations in a single configuration file

• setup.sh– namenode_fqdns=(“dev1.example.com” “dev2.example.com”)– namenode_dirs=(“/name/1” “/name/2”)– namenode_heap=32g– map_slots=5– reduce_slots=3– …

Design

• Can be used to setup any kind of hadoop cluster

• When doing main version upgarade, minimal the downtime– hadoop1 hadoop2

Namenode Active/Standby NamenodeSecondarynamenode Journalnodes ZKFC

Benifits

Release Engineering

• Build src tarball in hadoop-common

• Build rpms in bigtop

• submit build to release yum repo

• yum update on hadoop cluster…

Manually

• Setup hadoop-common daily build

• Setup Bigtop release Build – should be manually triggered

• Setup Hadooppet daily build– Run sanity checks on a REAL CLUSTER

Continuous Integration

• Build a Xen Server Cluster

Virtualization

• Pycon 2012– Small Python Tools for Software Release Engineering

• An automation tool to manageVM lifecycle

• Use Python XenAPI

• Create temporary VM for testingby self service

• Destroy it when the testingis finished

give-me-vm

• ./give_me_vm.py

• setup passphraseless ssh between each VM

• set hostname

• Install Hadooppet on master

• run deployment

• run sanity checks

• ./destroy_vm.py

Build auto deployment on Hadooppet

Development Environment

For hadoop service developers…

• No enough hadoop client for each developers

• Developer can not reach server side while developing hadoop related services

• Can not experiment new technology like impala spark flume

• CI on Hadoop related services

give-me-vm + Hadoop all-in-one VM

• Use Hadooppet to setup a peudo-distributed hadoop VM as Xenserver template

• get a Hadoop all-in-one VM via give-me-vm

• Services integrate its CI test with hadoop all-in-one VM

Bigtop

puppet

Bigtop puppet

• Bigtop also has a set of puppet scripts to deploy Hadoop ecosystem

Bigtop puppet

• Preparation:– A VM with jdk, puppet installed– mkdir –p /data/{1,2}– git clone https://github.com/apache/bigtop.git

• There’re many great deployment tool exist– Ambari, CM, ETU appliance– Choose suitable distribution by your business need

• If you want to do it by yourself– Bigtop can do packaging for you easily– Leverage bigtop puppet module for your deployment

Conclusion

Questions?

Thank you !

Building hadoop based big data environment

Technology

Transcript of Building hadoop based big data environment

Hadoop @ Sara & BiG Grid

Hadoop and big data

BIG DATA: Apache Hadoop

Big Data Hadoop Insight

Big data with hadoop

Hortonworks hadoop big data_retail__white_paper

HPE GreenLake Solutionsit.expandi-web.com/.../pdf/...Webinar_19Aprile2018.pdf · Hadoop in a nutshell • Hadoop is a complete SW environment –Hadoop Ecosystem • Big Data Storage

Big Data & Hadoop

Big data&hadoop

Big data hadoop rdbms

Big data hadoop FAQ's

Big Data - Hadoop/MapReduce

Hadoop Big Data A big picture

BIG DATA HADOOP FULL - themeslearning.com · Learning Big Data and Hadoop This module will help you understand Big Data Common Hadoop ecosystem components Hadoop Architecture HDFS

Hadoop Development - Leader In Software Training · HADOOP DEVELOPMENT 1. Understanding Big Data and Hadoop Introduction to Big Data & Big Data Challenges Limitations & Solutions

Intel 10GBASE-T in TACC Dynamic Hadoop Environment · Intel 10GBASE-T in TACC Dynamic Hadoop Environment ... and $34 billion in 2013 in information technology to handle big data.

Big Data Hadoop (Overview)

Big Data Hadoop Training

Big data Hadoop

BIG DATA HADOOP