Second HDF5 1.12.0 alpha release is available for testing

Hello everyone,

Good evening!
We are happy to announce our second alpha release for HDF5 1.12.0. You may find the release tar ball at
https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5_1.12/.

File format and all APIs are stable except H5Oget_info* and H5Ovisit* API calls, the H5Ovisit callback the H5O_info_t type and some VOL connector APIs. Release candidate will be created as soon as new versions of those APIs are available, which will be the default for HDF5 1.12.0.

Major changes from HDF5 1.12.0-alpha0:

  • Added support for new references to h5dump and h5ls tools.
  • Improved performance when creating a large number of small datasets.
  • New public functions H5Sselect_adjust, H5Sselect_project_intersection, and H5VLget_file_type,which may be useful when developing VOL connectors.
  • Updated DDL documentation with the new types of references, VDS and opaque types:
  • https://portal.hdfgroup.org/display/HDF5/DDL+in+BNF+for+HDF5-1.12+and+above

For more information about the release see
https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5_1.12/hdf5-1.12.0-alpha1-RELEASE.txt

In this release, we introduced several new features and along with performance improvement for hyperslab selection. A short description of the features is below.

Virtual Object Layer (VOL)

In this major HDF5 release we introduce HDF5 Virtual Object Layer (VOL). VOL is an abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model. The VOL layer intercepts all HDF5 API calls that potentially modify data on disk and forwards those calls to a plugin “object driver”. The data on disk can be a different format than the HDF5 format. For more information about VOL we refer the reader to the following documents (under review):
VOL HDF5 APIs

For available VOL connectors please see
https://bitbucket.hdfgroup.org/projects/HDF5VOL
We are asking VOL connector developers in particular to test this release and let us know about any problems ASAP. Please let us know if you want to contribute your code to the repository above to be more visible to HDF5 community.

Enhancements to HDF5 References

HDF5 references were extended to support attributes, and object and dataset selections that reside in another HDF5 file. For more information including a list of new APIs, see
https://portal.hdfgroup.org/display/HDF5/Update+to+References

This feature requires HDF5 File Format extension. HDF5 Libraries prior to 1.12.0 will not be able to read them.

New S3 and HDFS Virtual File Drivers (VFDs)

This release has two new VFDs. The S3 VFD allows accessing HDF5 files on AWS S3 buckets. HDFS VFD allows accessing HDF5 file stored on Apache HDFS. See
https://portal.hdfgroup.org/display/HDF5/Virtual+File+Drivers+-+S3+and+HDFS
for information on enabling those drivers and using those APIs.

Hyperslab selection performance improvements

In 1.12.0 we optimized hyperslab selection code to achieve better performance. In general, performance improved by an order of magnitude. In the case of reading a regular selection from 20GB dataset into a one dimensional array performance was improved by a factor of 6000. If you are interested in the benchmark we ran, please login to JIRA and see
https://jira.hdfgroup.org/browse/HDFFV-10930.

If you have time to test this alpha release, we would greatly appreciate it. We test HDF5 on a variety of platforms and with multiple compilers, but there is always a system that we couldn’t test on. Your feedback is critical.

Your prompt reports of any issues found will be invaluable and very much appreciated. Unless delayed to address serious newly discovered issues, we expect the final 1.12.0 release in about two months. Thank you and happy testing!

Elena Pourmal
Larry Knox

Hi,

Is improved SWMR support still planned for 1.12?

Hello,

SWMR VFD that we demonstrated at our Webinar is not part of 1.12.0 release. The feature is still under development.

We do plan to provide the code for the friendly users. We will announce on FORUM when the source is available.

Thank you!
Elena