Hello everyone,
Good evening!
We are happy to announce a beta release for HDF5 1.12.0. You may find the release tar ball at
https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5_1.12/
File format and all APIs should now be stable. A release candidate will be created soon after providing an opportunity for testing VOL connectors to find any unforeseen issues and address them as appropriate.
Major changes from HDF5 1.12.0-alpha1:
Public exposure of haddr_t type was refactored in favor of “object tokens”.
To better accommodate HDF5 VOL connectors where “object addresses in a file” may not be meaningful, a new H5O_token_t “object token” type has been introduced. The new “object tokens” represent unique and permanent identifiers for referencing HDF5 objects within a container and are designed to replace object addresses.
A new H5Oopen_by_token API call was introduced to open an object by a token, similar to how object addresses were previously used with H5Oopen_by_addr. New versions of certain H5L, and H5O functions were also added, along with compatibility macros to use previous or new versions of the functions according to configure options or compile flags to specify default api versions, and a “token” class with its own utility functions was added to a VOL class structure for interacting with object tokens using a VOL connector. See https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5_1.12/hdf5-1.12.0-beta-RELEASE.txt for more information about the release.
Major changes from HDF5 1.12.0-alpha0 to HDF5 1.12.0-alpha1 (previous release 12/12/2019):
Added support for new references to h5dump and h5ls tools.
Improved performance when creating a large number of small datasets.
New public functions H5Sselect_adjust, H5Sselect_project_intersection, and H5VLget_file_type,which may be useful when developing VOL connectors.
Updated DDL documentation with the new types of references, VDS and opaque types:
In this release, we introduced several new features and along with performance improvement for hyperslab selection. A short description of the features is below.
Virtual Object Layer (VOL)
In this major HDF5 release we introduce HDF5 Virtual Object Layer (VOL). VOL is an abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model. The VOL layer intercepts all HDF5 API calls that potentially modify data on disk and forwards those calls to a plugin “object driver”. The data on disk can be a different format than the HDF5 format. For more information about VOL we refer the reader to the following documents (under review):
VOL HDF5 APIs
For available VOL connectors please see
https://bitbucket.hdfgroup.org/projects/HDF5VOL
We are asking VOL connector developers in particular to test this release and let us know about any problems ASAP. Please let us know if you want to contribute your code to the repository above to be more visible to HDF5 community.
Enhancements to HDF5 References
HDF5 references were extended to support attributes, and object and dataset selections that reside in another HDF5 file. For more information including a list of new APIs, see
This feature requires HDF5 File Format extension. HDF5 Libraries prior to 1.12.0 will not be able to read them.
New S3 and HDFS Virtual File Drivers (VFDs)
This release has two new VFDs. The S3 VFD allows accessing HDF5 files on AWS S3 buckets. HDFS VFD allows accessing HDF5 file stored on Apache HDFS. See
for information on enabling those drivers and using those APIs.
Hyperslab selection performance improvements
In 1.12.0 we optimized hyperslab selection code to achieve better performance. In general, performance improved by an order of magnitude. In the case of reading a regular selection from 20GB dataset into a one dimensional array performance was improved by a factor of 6000. If you are interested in the benchmark we ran, please login to JIRA and see
https://jira.hdfgroup.org/browse/HDFFV-10930.
If you have time to test this beta release, we would greatly appreciate it. We test HDF5 on a variety of platforms and with multiple compilers, but there is always a system that we couldn’t test on. Your feedback is critical.
Your prompt reports of any issues found will be invaluable and very much appreciated. Unless delayed to address serious newly discovered issues, we expect the final 1.12.0 release in several weeks. Thank you and happy testing!
Larry Knox