SAN DIEGO SUPERCOMPUTER CENTER
HDF5/SRB IntegrationAugust 28, 2006
Mike [email protected]
SRB, SDSC
Peter [email protected]
HDF, NCSA
Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration
SAN DIEGO SUPERCOMPUTER CENTER
Current Status
• Improved transferring raw data (SDSC)• Tested performance
test #1 (270MB) from 18 to 15 (seconds)test #2 (550MB) from 231 to 35 (seconds)test #3 (820MB) from 734 to 113 (seconds)
• Project plan for SRB-FLASH
SAN DIEGO SUPERCOMPUTER CENTER
Things to move forward
• Improve GUI component in HDFView to open remote files (NCSA)
• Support primitive writing capabilities for HDF5 files in SRB (NCSA and SDSC)
• Write documentation and extensive testing (NCSA and SDSC)
• Release the production system and publish documentation (NCSA and SDSC)
SAN DIEGO SUPERCOMPUTER CENTER
SAP/SAC program: FLASH
Slice (gas temperature) and isosurface (gas density) in particle/mesh simulation
• FLASH is a adaptive-mesh simulation code for astrophysical hydrodynamics
• FLASH slice extractor retrieves a specific 2D slice from a 3D AMR (adaptive mesh refinement)
SAN DIEGO SUPERCOMPUTER CENTER
SRB-FLASH: architecture
SAN DIEGO SUPERCOMPUTER CENTER
SRB-FLASH: benefits
• No need for large storage on local machine. Terabytes of data files sit only on the sever machine
• Instant access to the data and metadata. Since only the selected data/metadata are transferred to the client instead of the whole multi-terabyte file, time on the network is tremendously reduced
• Data sharing among scientists. Since data sits on the server machine, it can be shared with scientists without multiple local copies of the same file. Also, everyone will have access to the updated data after a new simulation run
SAN DIEGO SUPERCOMPUTER CENTER
SRB-FLASH: project plan
• This project will be divided into three phases: phase I, phase II and phase III
• Phase I and part of phase II are required • Phase III and part of phase II are optional,
depending on the duration and amount of the funding
• The time estimates are based on an half (50%) FTE
SAN DIEGO SUPERCOMPUTER CENTER
SRB-FLASH: milestonePhase I: a command line client utility and server function to support selection of FLASH AMR mesh data.Tasks Start Finish
Collaboration between NCSA and SDSC
Duration of the project
Collecting requirements 10/2/06 10/13/06
Exploying the current tool 10/16/06 10/27/06
Implementing server function 10/30/06 12/22/06
Implementing client utility 12/25/06 2/16/07
Testing with FLASH AMR mesh data 2/19/07 3/16/07
Deploring to evaluation server 3/19/07 4/13/07
Documentation and release 4/16/07 5/11/07
SAN DIEGO SUPERCOMPUTER CENTER
SRB-FLASH: milestonePhase II: enhanced utilities querying FLASH specific metadata and statistics, and performing data interpolation and simple mathematic manipulation.
Tasks Start Finish
Collaboration between NCSA and SDSC
Duration of the project
Query and return simple statistics 5/14/07 6/22/07
Extract and return particle information 6/25/07 8/17/07
Analyze and interpolate line segment 8/20/07 10/12/07
Add support for png and ASCII text 10/15/07 11/23/07
SAN DIEGO SUPERCOMPUTER CENTER
SRB-FLASH: milestone
Phase III: development of general framework that can be adapted to other applications. We need to scope out the milestone for this phase based on the experience of phase I and II.
Top Related