buildoop

Open source collaboration project that provides templates and tools to help you create custom Linux-based systems based on Hadoop ecosystem.

“

The Buildoop Project

Software Development life-cycle

Buildoop dissection

● “SIT” the individual software modules are combined and tested as a group by means of virtualized infrastructure.

● “Configuration artifacts” set of configuration management files for operating system deploy and software configuration.

● “buildoop recipes” specify how a particular package is built. It includes all the package dependencies, source code locations, configuration, compilation, build, install and remove instructions.

Buildoop: Build package 1/4

● List of available BOM – Bill Of Materials. Set of components to manufacture an end product.

$ buildoop -bomAvailable BOM targets:

test.bomopenbus-0.0.1.bomstable.bombigtop-0.7.0.bom

conf/bom/├── bigtop-0.7.0.bom├── openbus-0.0.1.bom├── stable.bom└── test.bom

TARGET=CentOS-6.x

HADOOP_VERSION=2.2.0_openbus-0.0.1-r1ZOOKEEPER_VERSION=3.4.5_openbus-0.0.1-r1HBASE_VERSION=0.94.16_openbus-0.0.1-r1PIG_VERSION=0.11.1_openbus-0.0.1-r1HIVE_VERSION=0.11.0_openbus-0.0.1-r1SQOOP_VERSION=1.4.4_openbus-0.0.1-r1OOZIE_VERSION=4.0.0_openbus-0.0.1-r1FLUME_VERSION=1.4.0_openbus-0.0.1-r1STORM_VERSION=0.9.0.1_openbus-0.0.1-r1KAFKA_VERSION=0.8.0_openbus-0.0.1-r1HUE_VERSION=3.5.0_openbus-0.0.1-r1PROTOBUF_VERSION=2.5.0_r1

Buildoop: Build package 2/4

$ buildoop openbus-0.0.1 -infoTarget Platform:

CentOS-6.x

Ecosystem versions:

HADOOP: 2.2.0_openbus-0.0.1-r1ZOOKEEPER: 3.4.5_openbus-0.0.1-r1HBASE: 0.94.16_openbus-0.0.1-r1PIG: 0.11.1_openbus-0.0.1-r1HIVE: 0.11.0_openbus-0.0.1-r1SQOOP: 1.4.4_openbus-0.0.1-r1OOZIE: 4.0.0_openbus-0.0.1-r1FLUME: 1.4.0_openbus-0.0.1-r1STORM: 0.9.0.1_openbus-0.0.1-r1KAFKA: 0.8.0_openbus-0.0.1-r1HUE: 3.5.0_openbus-0.0.1-r1PROTOBUF: 2.5.0_r1

Buildoop: Build package 3/4

$ buildoop openbus-0.0.1 flume -info

Recipe name : flume-1.4.0_openbus-0.0.1-r1.bdDescription : Apache FlumeHome site : http://flume.apache.org/License : Apache-2.0URL base : http://www.eu.apache.org/dist/flume/1.4.0/apache-flume-1.4.0-src.tar.gzMD5SUM hash : a932da09b9edb95b0c4bf52e6480fe21

{"do_info": {

"description": "Apache Flume", "homepage": "http://flume.apache.org/","license": "Apache-2.0","filename": "flume-1.4.0_openbus-0.0.1-r1.bd"

},"do_download": {

"src_uri": "http://www.eu.apache.org/dist/flume/1.4.0/apache-flume-1.4.0-src.tar.gz","src_md5sum": "a932da09b9edb95b0c4bf52e6480fe21"

}, "do_fetch": {

"download_cmd": "wget"}

]

JSON recipe

Buildoop: Build package 4/4

$ buildoop openbus-0.0.1 flume -build$ buildoop openbus-0.0.1 hadoop -build

$ buildoop openbus-0.0.1 -build

build/deploy/├── bin│ ├── flume-1.4.0-openbus0.0.1.noarch.rpm│ ├── flume-agent-1.4.0-openbus0.0.1.noarch.rpm│ ├── hadoop-2.2.0-openbus0.0.1.x86_64.rpm│ ├── hadoop-hdfs-2.2.0-openbus0.0.1.x86_64.rpm│ ├── hadoop-hdfs-zkfc-2.2.0-openbus0.0.1.x86_64.rpm│ └── repodata│ ├── filelists.sqlite.bz2│ ├── filelists.xml.gz│ ├── other.sqlite.bz2│ ├── other.xml.gz│ └── repomd.xml└── src ├── flume-1.4.0-openbus0.0.1.src.rpm └── hadoop-2.2.0-openbus0.0.1.src.rpm

SIT: System Integration Tests

$ buildoop openbus-tests -infoAvailable tests for openbus set:

hdfs-testzookeeper-testFlume-test

$ buildoop openbus-tests hdfs-test -info$ buildoop openbus-tests hdfs-test -run

$ buildoop openbus-tests all-test -run

Test scope:

● Hadoop package repository deployment.● Puppet configuration deployment.● Smoke testing for Hadoop ecosystem.

https://github.com/buildoop

buildoop