SAN DIEGO SUPERCOMPUTER CENTER

HDF5/SRB IntegrationAugust 28, 2006

Mike Wanmwan@sdsc.edu

SRB, SDSC

Peter Caoxcao@ncsa.edu

HDF, NCSA

Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

SAN DIEGO SUPERCOMPUTER CENTER

Current Status

• Improved transferring raw data (SDSC)• Tested performance

test #1 (270MB) from 18 to 15 (seconds)test #2 (550MB) from 231 to 35 (seconds)test #3 (820MB) from 734 to 113 (seconds)

• Project plan for SRB-FLASH

SAN DIEGO SUPERCOMPUTER CENTER

Things to move forward

• Improve GUI component in HDFView to open remote files (NCSA)

• Support primitive writing capabilities for HDF5 files in SRB (NCSA and SDSC)

• Write documentation and extensive testing (NCSA and SDSC)

• Release the production system and publish documentation (NCSA and SDSC)

SAN DIEGO SUPERCOMPUTER CENTER

SAP/SAC program: FLASH

Slice (gas temperature) and isosurface (gas density) in particle/mesh simulation

• FLASH is a adaptive-mesh simulation code for astrophysical hydrodynamics

• FLASH slice extractor retrieves a specific 2D slice from a 3D AMR (adaptive mesh refinement)

SAN DIEGO SUPERCOMPUTER CENTER

SRB-FLASH: architecture

SAN DIEGO SUPERCOMPUTER CENTER

SRB-FLASH: benefits

• No need for large storage on local machine. Terabytes of data files sit only on the sever machine

• Instant access to the data and metadata. Since only the selected data/metadata are transferred to the client instead of the whole multi-terabyte file, time on the network is tremendously reduced

• Data sharing among scientists. Since data sits on the server machine, it can be shared with scientists without multiple local copies of the same file. Also, everyone will have access to the updated data after a new simulation run

SAN DIEGO SUPERCOMPUTER CENTER

SRB-FLASH: project plan

• This project will be divided into three phases: phase I, phase II and phase III

• Phase I and part of phase II are required • Phase III and part of phase II are optional,

depending on the duration and amount of the funding

• The time estimates are based on an half (50%) FTE

SAN DIEGO SUPERCOMPUTER CENTER

SRB-FLASH: milestonePhase I: a command line client utility and server function to support selection of FLASH AMR mesh data.Tasks Start Finish

Collaboration between NCSA and SDSC

Duration of the project

Collecting requirements 10/2/06 10/13/06

Exploying the current tool 10/16/06 10/27/06

Implementing server function 10/30/06 12/22/06

Implementing client utility 12/25/06 2/16/07

Testing with FLASH AMR mesh data 2/19/07 3/16/07

Deploring to evaluation server 3/19/07 4/13/07

Documentation and release 4/16/07 5/11/07

SAN DIEGO SUPERCOMPUTER CENTER

SRB-FLASH: milestonePhase II: enhanced utilities querying FLASH specific metadata and statistics, and performing data interpolation and simple mathematic manipulation.

Tasks Start Finish

Collaboration between NCSA and SDSC

Duration of the project

Query and return simple statistics 5/14/07 6/22/07

Extract and return particle information 6/25/07 8/17/07

Analyze and interpolate line segment 8/20/07 10/12/07

Add support for png and ASCII text 10/15/07 11/23/07

SAN DIEGO SUPERCOMPUTER CENTER

SRB-FLASH: milestone

Phase III: development of general framework that can be adapted to other applications. We need to scope out the milestone for this phase based on the experience of phase I and II.