HDF5 1.12.0-alpha0 snapshot release


#1

Good afternoon!

We are happy to announce our first alpha release for HDF5 1.12.0. You may find the release tar ball at

https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5

For more information about the release see

https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5/hdf5-1.12.0-alpha0-RELEASE.txt

In this release, we introduced several new features and along with performance improvement for hyperslab selection. A short description of the features is below.

Virtual Object Layer (VOL)

In this major HDF5 release we introduce HDF5 Virtual Object Layer (VOL). VOL is an abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model. The VOL layer intercepts all HDF5 API calls that potentially modify data on disk and forwards those calls to a plugin “object driver”. The data on disk can be a different format than the HDF5 format. For more information about VOL we refer the reader to the following documents (under review):

     VOL HDF5 APIs
     https://portal.hdfgroup.org/display/HDF5/Virtual+Object++Layer

For available VOL connectors please see https://bitbucket.hdfgroup.org/projects/HDF5VOL

We are asking VOL connector developers in particular to test this release and let us know about any problems ASAP. Please let us know if you want to contribute your code to the repository above to be more visible to HDF5 community.

Enhancements to HDF5 References

HDF5 references were extended to support attributes, and object and dataset selections that reside in another HDF5 file. For more information including a list of new APIs, see

     https://portal.hdfgroup.org/display/HDF5/Update+to+References

This feature requires HDF5 File Format extension. HDF5 Libraries prior to 1.12.0 will not be able to read them.

New S3 and HDFS Virtual File Drivers (VFDs)

This release has two new VFDs. The S3 VFD allows accessing HDF5 files on AWS S3 buckets. HDFS VFD allows accessing HDF5 file stored on Apache HDFS

See https://portal.hdfgroup.org/display/HDF5/Virtual+File+Drivers±+S3+and+HDFS for information on enabling those drivers and using those APIs.

Hyperslab selection performance improvements

In 1.12.0 we optimized hyperslab selection code to achieve better performance. In general, performance improved by an order of magnitude. In the case of reading a regular selection from 20GB dataset into a one dimensional array performance was improved by a factor of 6000. If you are interested in the benchmark we ran, please login to JIRA and see https://jira.hdfgroup.org/browse/HDFFV-10930.

If you have time to test this alpha release, we would greatly appreciate it. We test HDF5 on a variety of platforms and with multiple compilers, but there is always a system that we couldn’t test on. Your feedback is critical.

Your prompt reports of any issues found will be invaluable and very much appreciated. Unless delayed to address serious newly discovered issues, we expect the final 1.12.0 release in about two months. Thank you and happy testing!

Elena Pourmal
Larry Knox