Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015
-
Upload
sangjin-lee -
Category
Software
-
view
165 -
download
5
Transcript of Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015
![Page 1: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/1.jpg)
T I M E L I N E S E R V I C E N E X T G E N ( YA R N - 2 9 2 8 )
![Page 2: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/2.jpg)
WHY NEXT GEN?
Scalability Single global instance of writer/reader v.1 uses a local-disk-based LevelDB storage instance
Usability Handle flows as first-class concept and model aggregation Elevate configuration and metrics to first-class members
Existing external tooling: hRaven, Finch, Dr. Elephant, etc.
![Page 3: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/3.jpg)
KEY DESIGN POINTS
Distributed writer architecture Scalable storage backend (HBase) Reimagined object model API with flows built into it Separated reader instances Aggregation
![Page 4: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/4.jpg)
DISTRIBUTED WRITERS & READERS
!melinereader
!melinereader
Storage
!melinereader
AM !melinewriter
NM
!meline reader pool
app metrics/events
container events/metrics
RM!meline writer
app/container events
user queries
![Page 5: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/5.jpg)
STATUS
[DONE] timeline writers (per-app and per-node) as aux service [DONE] RM companion writer [DONE] first iteration of the object model API [DONE] file-based test writer [DONE] NM writing container events [DONE] RM writing app/container entities [DONE] AMs writing framework-specific events and metrics [DONE] first versions of Phoenix and HBase writer impls [DONE] performance benchmarking evaluation of writers
![Page 6: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/6.jpg)
STATUS
[WIP] timeline readers [WIP] aggregation UI enhancements Stand-alone timeline writer (per-node and per-app) Finalize implementation of supported queries Security Migration/compatibility story …
![Page 7: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/7.jpg)
TEAM
This is a true community collaboration! Sangjin, Vrushali and Joep (Twitter) Zhijie, Li, Junping and Vinod (Hortonworks) Naga and Varun (Huawei) Robert and Karthik (Cloudera) Input from LinkedIn, Yahoo! and Altiscale
![Page 8: Timeline Service Next Gen (YARN-2928): YARN BOF @ Hadoop Summit 2015](https://reader036.fdocuments.in/reader036/viewer/2022071902/55c2c81fbb61ebac768b467d/html5/thumbnails/8.jpg)
QUESTIONS?