HDF5 Tools
description
Transcript of HDF5 Tools
HDF5 Tools
Peter Cao
The HDF Group
October 15-18, 2008 1HDF and HDF-EOS Workshop XII
Outline
2
• A short demo of HDFView
• HDF5 command line tools
October 15-18, 2008 HDF and HDF-EOS Workshop XII
What is HDFView?
3
For more information and download, visit
www.hdfgroup.org/hdf-java-html
HDFView
HDF Application Browser/Editor Written in Java
October 15-18, 2008 HDF and HDF-EOS Workshop XII
HDFView Demo
• Browse objects in an HDF5 file• Show data content in table, image• Make a subset• View attributes and metadata• Create a new HDF5 file and edit data content
October 15-18, 2008 4HDF and HDF-EOS Workshop XII
HDFView 2.5
• Release planned in December, 2008• Build with HDF4 2r4 and HDF5 1.8.2• Fix bugs and add new features• Plug memory leaks• Support iRODS (Windows and linux only)• Use jdk 1.6.0_06 or above (for performance)
October 15-18, 2008 5HDF and HDF-EOS Workshop XII
Drop mac PowerPC
HDF5 Command Line Tools
October 15-18, 2008 6
h5dump, h5diff, h5ls, h5check, h5stat
h5repack, h5repart, h5import, h5jam, h5unjam, h5copy, h5mkgrp
h4toh5, h5toh4, gif2h5, h52gif
Readers
Writers
Converters
HDF and HDF-EOS Workshop XII
Where to Start?
October 15-18, 2008 7
Tools located in bin/ directory
1. Use “-h” option for help
2. Consult online help http://www.hdfgroup.org/hdf5tools.html
3. Email help desk [email protected]
HDF and HDF-EOS Workshop XII
h5dump
October 15-18, 2008 8
Dumps contents of HDF5 file to stdout
• ASCII text• XML • Binary
HDF and HDF-EOS Workshop XII
h5dump: Object Headers
October 15-18, 2008 9
HDF5 "SDS.h5" { GROUP "/" { GROUP "Floats" { DATASET "FloatArray" { DATATYPE H5T_IEEE_F32LE DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) } } } DATASET "IntArray" { DATATYPE H5T_STD_I32LE DATASPACE SIMPLE { ( 5, 6 ) / ( 5, 6 ) } } }}
> h5dump -H SDS.h5
HDF and HDF-EOS Workshop XII
h5dump: Dataset
October 15-18, 2008 10
HDF5 "SDS.h5" {
DATASET "/Floats/FloatArray" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 4, 3 ) / ( 4, 3 ) }
DATA {
(0,0): 0.01, 0.02, 0.03,
(1,0): 0.1, 0.2, 0.3,
(2,0): 1, 2, 3,
(3,0): 10, 20, 30
}
}
}
> h5dump –d /Floats/FloatArray SDS.h5
HDF and HDF-EOS Workshop XII
h5dump: Binary Output
October 15-18, 2008 11
-b F, --binary=[MEMORY|FILE|LE|BE]
0000000 00000000 00000001 00000002 00000003 00000004 000000050000030 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f0000060 00000014 00000015 00000016 00000017 00000018 000000190000110 0000001e 0000001f 00000020 00000021 00000022 000000230000140 00000028 00000029 0000002a 0000002b 0000002c 0000002d
> h5dump -d /IntArray -o out_le.bin -b LE SDS.h5
HDF and HDF-EOS Workshop XII
h5diff
October 15-18, 2008 12
Showdifferencesbetween…
two objects or
two files
HDF and HDF-EOS Workshop XII
h5diff
October 15-18, 2008 13
Dataset: </IntArray>5 differences found
> h5diff SDS.h5 SDS2.h5
HDF and HDF-EOS Workshop XII
h5diff: Report Differences
October 15-18, 2008 14
Dataset: </IntArray> and </IntArray>position IntArray IntArray difference ------------------------------------------------------
[ 0 0 ] 0 10 10
[ 1 0 ] 10 100 90
[ 2 0 ] 20 200 180
[ 3 0 ] 30 300 270
[ 4 0 ] 40 400 360
5 differences found
> h5diff SDS.h5 SDS2.h5 -r /IntArray
HDF and HDF-EOS Workshop XII
h5diff: Unsolved Issues
How to compare two files with different Structures Storage layouts Properties
October 15-18, 2008 15HDF and HDF-EOS Workshop XII
h5repack
October 15-18, 2008 16
Copies a file to a new file with • Compression filter• Data layout• Other user options
HDF and HDF-EOS Workshop XII
h5repack: Compression Filter
October 15-18, 2008 17
-f {GZIP|SZIP|SHUF|FLET|NBIT|SOFF|NONE}
For example
> h5repack -i SDS.h5 -o out.h5 -f GZIP=6
HDF and HDF-EOS Workshop XII
h5repack: Data Layout
October 15-18, 2008 18
-l {CHUNK| COMPA| CONTI}
For example> h5repack -i SDS.h5 -o out.h5 -l CHUNK=1000x1000
HDF and HDF-EOS Workshop XII
h5repack: File Format Version
October 15-18, 2008 19
h5repack options File size in bytes
Original file 3,167,264
--latest 2,267,346
--latest –compact=9 2,162,254
--latest –compact=9 –ssize=50:dtype 299,882
The test file was created with HDF5 1.6.5. The root group of the file contains 100 groups. Each group at the root contains 9 sub groups. Each sub group contains one compound dataset. The compound dataset has 40 members of type 32 bit integer
HDF and HDF-EOS Workshop XII
h5import
October 15-18, 2008 20
PATH work/First-set INPUT-CLASS TEXTFP RANK 3 DIMENSION-SIZES 5 2 4 OUTPUT-CLASS FP OUTPUT-SIZE 64 OUTPUT-ARCHITECTURE IEEE OUTPUT-BYTE-ORDER LE CHUNKED-DIMENSION-SIZES 2 2 2 MAXIMUM-DIMENSIONS 8 8 -1
GROUP "/" { GROUP "work" { DATASET "First-set" { DATATYPE H5T_IEEE_F64LE DATASPACE SIMPLE { ( 5, 2, 4 ) / ( 8, 8, H5S_UNLIMITED ) } DATA { (0,0,0): 1.01, 1.02, 1.03, 1.04, (0,1,0): 1.11, 1.12, 1.13, 1.14, (1,0,0): 1.21, 1.22, 1.23, 1.24, (1,1,0): 1.31, 1.32, 1.33, 1.34, (2,0,0): 1.41, 1.42, 1.43, 1.44, … } } }}}
Imports binary/ASCII data into an HDF5 file
h5import infile -c config_file [infile -c config_file2 ...] -outfile outfile
For example
> h5import float5x4x2.txt -c First_set.conf -o First_set.h5
HDF and HDF-EOS Workshop XII
h5import: Requested Features
October 15-18, 2008 21
• Make it interchangeable with h5dump Excel spreadsheet
HDF and HDF-EOS Workshop XII
New for 1.6.6+ and 1.8.0+
October 15-18, 2008 22
h5stath5mkgrp
h5copy
h5check
HDF and HDF-EOS Workshop XII
h5copy
• Copies an object within a file or across files
October 15-18, 2008 23
/
FloatArray
FloatsIntArray
/
FloatArray
HDF and HDF-EOS Workshop XII
h5copy –f shallow
October 15-18, 2008 24
SRC
i1
floatsintegers
64-biti2 f32
f2f1
DST2
floats
64-bitf32
f2f1
DST1
floats
64-bitf32-f shallow
HDF and HDF-EOS Workshop XII
h5mkgrp
• Creates a group or group structure
October 15-18, 2008 25
/
info data
test1 test2
> h5mkgrp test.h5 /info /data /data/test1 /data/test2
HDF and HDF-EOS Workshop XII
h5stat
Prints statistics about HDF5 file Header size Raw data size Attributes, datatypes, etc.
Helps Troubleshoot overhead in HDF5 files Choose specific object’s properties and storage strategies
26October 15-18, 2008 HDF and HDF-EOS Workshop XII
h5stat: Example
Filename: hdf5_test.h5
File information
# of unique groups: 4
# of unique datasets: 13
…
Object header size: (total/unused)
Groups: 288/32
Datasets: 6824/864
…
Dataset storage information:
Total raw data size: 1769009
…
October 15-18, 2008 27HDF and HDF-EOS Workshop XII
>h5stat hdf5_test.h5
h5check
• Verifies whether an HDF5 file is compliant with a specific HDF5 File Format.
• By default, the file is checked against 1.8.x. • Use -f16 option to check against 1.6.x.
October 15-18, 2008 28HDF and HDF-EOS Workshop XII
h5check
October 15-18, 2008 29
VALIDATING example.h5 according to library version 1.8.0
FOUND super block signature
VALIDATING the super block at physical address 0...
Validating version 2 superblock...
VALIDATING the object header at logical address 48...
VALIDATING version 2 object header ...
FOUND Version 2 object header signature
VALIDATING version 2 btree at logical address 1864...
VALIDATING version 2 btree header at logical address 1864…
:
No non-compliance errors found
> h5check --verbose=2 example.h5
HDF and HDF-EOS Workshop XII
> h5check --verbose=2 –f16 example.h5
VALIDATING example.h5 according to library version 1.6.6
VALIDATING the super block at physical address 0...
Validating version 0/1 superblock...
***Error***
Errors found when checking superblock. Validation stopped.
Disk Format Level 0-File Metadata-->0A-File Signature and Super Block
Superblock v.0/1:Unable to read root symbol table entry
***End of Error messages***
Non-compliance errors found
h5check
October 15-18, 2008 30HDF and HDF-EOS Workshop XII
October 15-18, 2008 31
This presentation is based upon work supported in part by a Cooperative Agreement with NASA under NASA NNX06AC83A. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Aeronautics and Space Administration
Thank You!
HDF and HDF-EOS Workshop XII
Questions/comments?
October 15-18, 2008 32HDF and HDF-EOS Workshop XII