Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data...
Transcript of Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data...
![Page 1: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/1.jpg)
Building Cutting Edge Big Data Platform with Apache Bigtop
PRESENTED BY Evans Ye| December 5, 2016
Big Data Innovation Summit
![Page 2: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/2.jpg)
Yahoo Confidential & Proprietary
Who am I
2
▪Software Engineer @ Yahoo! APAC Data Team
▪Building personalized data products for...
▪Apache Bigtop PMC member
![Page 3: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/3.jpg)
Yahoo Confidential & Proprietary
Outline
3
▪Quick Intro to Apache Bigtop
▪Bigtop Provisioner
▪Bigtop Sandbox
▪Big Data Landscape
▪Open Source Adoption Strategy
▪Release Timeline
![Page 4: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/4.jpg)
Quick Intro to Apache Bigtop
![Page 5: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/5.jpg)
Yahoo Confidential & Proprietary
Linux Distributions
5
![Page 6: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/6.jpg)
Yahoo Confidential & Proprietary
Hadoop Distributions
6
![Page 7: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/7.jpg)
Yahoo Confidential & Proprietary7
There're some other great Hadoop ecosystem components..
![Page 8: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/8.jpg)
Yahoo Confidential & Proprietary8
How do I add patches?
![Page 9: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/9.jpg)
Yahoo Confidential & Proprietary9
![Page 10: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/10.jpg)
Yahoo Confidential & Proprietary
From source code to packages
10
BigtopPackaging
![Page 11: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/11.jpg)
Yahoo Confidential & Proprietary
Bigtop feature set
11
Packaging Testing Deployment Virtualization
for you to easily build your own Big Data Stack
![Page 12: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/12.jpg)
Yahoo Confidential & Proprietary
Supported components
12
![Page 13: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/13.jpg)
13
Addressdependency Issues
![Page 14: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/14.jpg)
Yahoo Confidential & Proprietary
Bigtop early mission accomplished
14
Leveraged by app providers…
![Page 15: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/15.jpg)
Yahoo Confidential & Proprietary15
What now?
![Page 16: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/16.jpg)
Yahoo Confidential & Proprietary16
Get out from the Apache dome
![Page 17: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/17.jpg)
Yahoo Confidential & Proprietary
New focus and target end users
17
▪Data engineers vs Distro. builders
▪Reference implementation & stacks
▪Solution diversity:
▪Streaming: Flink, Apex
▪ In-memory cache: Alluxio, Ignite
▪User/developer tools:
▪Bigtop Provisioner
▪Bigtop Sandbox
![Page 18: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/18.jpg)
Bigtop Provisioner
![Page 19: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/19.jpg)
Yahoo Confidential & Proprietary
Bigtop Provisioner
19
▪A tool to demonstrate full life cycle of Bigtop
Packaging TestingDeploymentVirtualization
Create resources Run Bigtop Puppet Run Bigtop Tests
Bigtop Provisioner
![Page 20: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/20.jpg)
Yahoo Confidential & Proprietary
One click Hadoop provisioning(Bigtop 1.0.0)
20
bigtop/deploy image on Docker hub
./docker-hadoop.sh -c 3
puppet apply
puppet apply
puppet apply
![Page 21: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/21.jpg)
Yahoo Confidential & Proprietary
What’s the problem with Vagrant’s Docker Provider?
21
▪Need to add vagrant public key into docker images
▪Too many issues with auto-created boot2docker VM
▪A bug for docker provider keep opening for almost 2y
▪Waiting for machine to boot' hangs infinitely
▪Can not share same code for different providers anyway
▪Not all the docker options supported in Vagrantfile
▪^#?& slow
![Page 22: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/22.jpg)
Yahoo Confidential & Proprietary
Impl. replaced by docker-compose (1.2.0-SNAPSHOT)
22
bigtop/deploy image on Docker hub
./docker-hadoop.sh -c 3
puppet apply
puppet apply
puppet apply
![Page 23: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/23.jpg)
Yahoo Confidential & Proprietary
Advantages
23
▪No need to create customized image beforehand
▪Better compatibility with Docker’s native solutions
▪Clear, simple yaml file for orchestration settings
▪Supports new features such as overlay network
and named volume
▪Fast —> better user experience
![Page 24: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/24.jpg)
Bigtop Sandbox
![Page 25: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/25.jpg)
Yahoo Confidential & Proprietary
Introducing Bigtop Docker Sandbox
25
▪Docker images that has Bigtop stacks installed and configured
▪Pseudo cluster up & running w/ zero
installation/configuration
▪Command-line tool to build your own stack
![Page 26: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/26.jpg)
Yahoo Confidential & Proprietary
Docker Image layer Interface
26
Customizedbigdatastack
Deploy&managementtool
Baseimage(OS)
![Page 27: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/27.jpg)
Yahoo Confidential & Proprietary
Docker Image layer Concrete implementation
27
Hadoop+HBase+Spark
BigtopPuppet
CentOS
![Page 28: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/28.jpg)
Yahoo Confidential & Proprietary
Building images
28
CentOS
BigtopPuppet
Hadoop+Hbase+Spark
+site.yaml
$ puppet apply
![Page 29: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/29.jpg)
Yahoo Confidential & Proprietary
Running images
29
Hadoop+Hbase+Spark
$ puppet apply
![Page 30: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/30.jpg)
Yahoo Confidential & Proprietary
Support integration tests in CI/CD
30
![Page 31: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/31.jpg)
Yahoo Confidential & Proprietary31
Bigtop Provisioner Bigtop Sandbox
Data engineers Create multi-node cluster for testing
Build/use sandboxes for dev/test
Ops Create multi-node cluster for testing -
ContributorsTest packages, puppet recipes,
test cases-
Distro. BuildersTest packages, puppet recipes,
test casesProvide Sandboxes
![Page 32: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/32.jpg)
Big Data Landscape
![Page 33: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/33.jpg)
Yahoo Confidential & Proprietary
Hot topics in Apache conferences
33
▪Spark dominates the big data world and the
research area
▪26/27 talks with Spark on the title
▪Streaming is still a hot topic
▪8/8 talks with steaming on the title
![Page 34: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/34.jpg)
Yahoo Confidential & Proprietary34
▪New features and major version upgrade in key projects
▪Spark 2.0
▪Hadoop 3.0
▪HBase 2.0
▪Cassandra 3.X
▪Kafka Streams & Connect
▪Still, there’re so many new projects
▪Helix, Calcite, Unomi, Samoa, Kudu, Streams, OODT, Tinkerpop,
Kerby, Yetus, HTrace, REEF, Bahir, ...
Innovations
![Page 35: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/35.jpg)
Apache HBase
![Page 36: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/36.jpg)
Yahoo Confidential & Proprietary
Region Replicas for better Availability
36
▪Multiple Region Servers host each region
▪Primary + N read replicas (usually N=2)
▪Primary is authority on writes
▪Replicas tail replicate edits, offer TIMELINE view
▪Client's choice
▪Read primary only for "classic" strong consistency
▪Fan-out reads for faster, potentially TIMELINE results
▪Ref:
▪Apache HBase: State of the Database, Nick Dimiduk
![Page 37: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/37.jpg)
Yahoo Confidential & Proprietary
Off-heap Caching
37
▪On-heap LRU cache limited by Java heap size
▪Off-heap bucket cache - solution for GC pause
▪Serve data directly from off-heap cache since cells
are backed by byte arrays
▪Ref:
▪Recent Development in HBase, Zhihong Yu
![Page 38: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/38.jpg)
Apache Cassandra
![Page 39: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/39.jpg)
Yahoo Confidential & Proprietary
SStable Attached Secondary Index(SASI)
39
▪Attach B+ tree on SSTable
▪Range scan on 2nd index made possible
▪Ref:
▪Cassandra 3.4 and Beyond, Jon Haddad
![Page 40: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/40.jpg)
Apache Kafka
![Page 41: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/41.jpg)
Yahoo Confidential & Proprietary
Kafka Connect
41
▪Focus on copying
▪Standardize
▪Parallelism
▪Scale
▪Ref:
▪Kafka Connect: Real-time Data Integration at Scale
with Apache Kafka, Ewen Cheslack-Postava
![Page 42: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/42.jpg)
Yahoo Confidential & Proprietary
Kafka Streams
42
▪Available in Kafka since 0.10, May 20, 2016
▪Powerful yet easy-to-use stream processing library
▪Event-at-a-time, Stateful
▪Windowing with out-of-order handling
▪Highly scalable, distributed, fault tolerant
▪Ref:
▪ Introduction to Kafka Streams, Guozhang Wang
![Page 43: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/43.jpg)
Yahoo Confidential & Proprietary
Summary
43
▪HBase
▪ region replicas in 1.2, off-heap read in 2.0
▪Cassandra
▪SASI in 3.4
▪Kafka
▪Kafka Connectors available on Confluent
▪Kafka Streams in Kafka 0.10
![Page 44: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/44.jpg)
Open Source Adoption Strategy
![Page 45: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/45.jpg)
Yahoo Confidential & Proprietary
3 key aspects
45
▪Functionality
▪Community
▪Maturity
![Page 46: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/46.jpg)
Yahoo Confidential & Proprietary
Functionality
46
▪Survey! Only you knows your requirement
▪Requires an architect /senior data engineer who has deep
understanding as well as broader view
▪Streaming for instance:
▪Exactly-once
▪Out-of-order handling
▪CEP
▪<1s SLA
![Page 47: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/47.jpg)
Yahoo Confidential & Proprietary
Community
47
▪How many contributors? (see on github)
▪Employers of committers?
▪When's the last release?
▪What's the last code updated time?
▪Subscribe to the mailing list and see how many
discussions daily/weekly
![Page 48: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/48.jpg)
Yahoo Confidential & Proprietary
Maturity
48
▪Who already on-boarded?
▪Founded by major big data players?
▪Cloudera, Hortonworks, MapR, Pivotal, AMPLab
▪Donated by big tech companies?
▪Yahoo!, Facebook, Twitter
▪Previously a commercial software?
▪For example: Apex, Ignite, Geode
▪Backed by a company?
▪Databricks, dataArtisans, Confluent, DataTorrent, Gridgain
![Page 49: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/49.jpg)
Release Timeline
![Page 50: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/50.jpg)
▪Released Feb, 2016:
▪Zeppelin 0.5.6
▪Tachyon 0.6.0
▪Hama 0.7.0
Yahoo Confidential & Proprietary
New components Bigtop 1.1.0
50
![Page 51: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/51.jpg)
▪Ported 22 out of 24 Apache Bigtop stacks to
POWER in two week
▪Apache Bigtop has Dockerized the entire build
environmentYahoo Confidential & Proprietary
New Arch in Bigtop 1.1.0
51
![Page 52: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/52.jpg)
Yahoo Confidential & Proprietary52
What's coming in 1.2 release?
Expected to be out in Feb, 2017
![Page 53: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/53.jpg)
▪Featured upgrade:
▪Hadoop 2.7.3
▪Spark 2.0.2
▪Kafka 0.10.0
▪HBase 1.2.4
Yahoo Confidential & Proprietary
What's coming in 1.2 release?
53
![Page 54: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/54.jpg)
▪New components:
▪Apache Flink 1.1.3
▪Apache Apex 3.4.0
▪Greenplum Database 4.3.99.0
▪Quantcast File System 1.1.4
Yahoo Confidential & Proprietary
What's coming in 1.2 release?
54
![Page 55: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/55.jpg)
▪New features:
▪Juju bigtop charms
▪Bigtop Docker Sandbox
▪ Improvement:
▪Bigtop Docker Provisioner made faster
Yahoo Confidential & Proprietary
What's coming in 1.2 release?
55
![Page 56: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/56.jpg)
Yahoo Confidential & Proprietary
Reference
56
▪ Home page: http://bigtop.apache.org/
▪ Document: https://cwiki.apache.org/confluence/display/BIGTOP/Index
▪ Source code: http://www.apache.org/dist/bigtop/bigtop-1.1.0/
▪ Packages: https://www.apache.org/dist/bigtop/bigtop-1.1.0/repos/
▪ JIRA: https://issues.apache.org/jira/browse/BIGTOP
▪ Sandbox Preview: https://youtu.be/yvmZu7Jbtag
![Page 57: Building Cutting Edge Big Data Platform with Apache Bigtop pub · Building Cutting Edge Big Data Platform with Apache Bigtop PRESENTED BY Evans Ye December 5, ... Introduction to](https://reader030.fdocuments.in/reader030/viewer/2022041014/5ec5449edb52844997415297/html5/thumbnails/57.jpg)
Yahoo Confidential & Proprietary57
Thank you !
Questions?