URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg...
Transcript of URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg...
![Page 1: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/1.jpg)
Proprietary and Confidential. Copyright 2016, The HDF Group.Proprietary and Confidential. Copyright 2016, The HDF Group.
HDF Group Past, Present & Future
1
![Page 2: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/2.jpg)
Outline
2
Who we are and where we come from
Mission and commitments to community
Future directions for HDF Group & HDF R&D
![Page 3: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/3.jpg)
Who we are and where we come from
![Page 4: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/4.jpg)
National Center for Supercomputing Applications - 19874
Big simulations
Desktop visualization
![Page 5: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/5.jpg)
The birth of HDF 5
![Page 6: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/6.jpg)
6
![Page 7: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/7.jpg)
Version 1.0
7
![Page 8: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/8.jpg)
NASA Earth Observing System
• 6,700 Data Products • 12 Data Archive Centers
• 16 terabytes per day • Distributed to 1.7 million
end users worldwide
![Page 9: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/9.jpg)
Non-destructive testing - TRAPPIST
9
![Page 10: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/10.jpg)
Shortcomings of HDF (1996)
Limits on object & file size (<2GB)
Limited number of objects (<20K)
Rigid data models
I/O performance
Code complexity
![Page 11: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/11.jpg)
Accelerated Strategic Computing Initiative
11
How to maintain a nuclear stockpile without testing?
![Page 12: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/12.jpg)
Data can be complicated
12
![Page 13: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/13.jpg)
The birth of HDF5
13
![Page 14: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/14.jpg)
Long-term data
preservation
14
![Page 15: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/15.jpg)
HDF keys to success
15
NeedsHDF
Group
Software for the long term.
API, library, utilities, documentation
User support
Institutional support
Long term storage
Complex formatting
Complex uses
Open source durability
![Page 16: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/16.jpg)
Spinning off
•Business model options•For-Profit or not?•How to make money?• Intellectual property
16
![Page 17: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/17.jpg)
Who is the HDF Group?
Small company ~ 40 employeesfocus on High
Performance Computing and Scientific Data
Offices in Champaign, IL + 7 other locations
Our flagship platform –HDF5 – is at the heart of our
open source ecosystem.
Tens of thousands use HDF5 every day.
1500+ projects on Github
Work with industry, research instituitons,
government to improve HDF5,
build HDF5 solutions, provide support.
![Page 18: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/18.jpg)
18
Mission and commitments to OSS and community
![Page 19: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/19.jpg)
HDF Group Website
19
![Page 20: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/20.jpg)
Software
20
![Page 21: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/21.jpg)
HDF Support portal
21
![Page 22: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/22.jpg)
HDF Forum
22
![Page 23: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/23.jpg)
Blogs
23
![Page 24: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/24.jpg)
Webinars
24
![Page 25: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/25.jpg)
Technology Advisory Board
25
![Page 26: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/26.jpg)
Get engaged
HDFql in Nov/Dec 2019Present a Webinar about your project
About your project or some aspect of HDF5 Write a blog
[email protected] Contact
![Page 27: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/27.jpg)
27
Future directions for the HDF Group and HDF R&D
![Page 28: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/28.jpg)
1998-2007 - Maturation
• Matured the HDF5 data model and customized I/O including parallel I/O
• Community standards NeXUS, HDF-EOS, etc. emerged
28
![Page 29: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/29.jpg)
2008-2019 - Expanding to non-HPC communities and Exascale computing
VPIC VORPAL
0
5000
10000
15000
Hopper Intrepid Hopper Intrepid
I/O
Ba
nd
wid
th (
MB
/s)
Default
Tuned
0
100
200
300
400
500
600
Num
ber o
f uniqu
e us
ers
Top 15 libraries used on Edison (2014)
• Widespread use• Improved I/O performance• Growth of software ecosystem• New features to support particle accelerators, exascalearchitectures, observational data, etc.
29
![Page 30: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/30.jpg)
Future directions
30
![Page 31: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/31.jpg)
Continue…
31
Support legacy library versions
Adapt to OS changes, etc.
Improve library, tools, docs
Increase community involvement
![Page 32: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/32.jpg)
Leverage new memory, computing and storage architectures
• Deep memory hierarchy
• Alternate storage and I/O
• Non-traditional HDF5 uses
32
![Page 33: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/33.jpg)
Facilitate convergence of Big Data and HPC
• Combine data and compute services
• Large scale query
![Page 34: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/34.jpg)
Ease of use, productivity
• New APIs for C++ community
• Working with tool developers
![Page 35: URXS 3DVW 3UHVHQW )XWXUH - The HDF Group...2xwolqh :kr zh duh dqg zkhuh zh frph iurp 0lvvlrq dqg frpplwphqwv wr frppxqlw\)xwxuh gluhfwlrqv iru +') *urxs +') 5 '](https://reader035.fdocuments.in/reader035/viewer/2022081620/61068b491c70461ed173d8d8/html5/thumbnails/35.jpg)
THANK YOU!Questions & Comments?
Proprietary and Confidential. Copyright 2016, The HDF Group.
35