Hello everyone,
Good evening!
We are happy to announce our second alpha release for HDF5 1.12.0. You may find the release tar ball at
https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5_1.12/.
File format and all APIs are stable except H5Oget_info* and H5Ovisit* API calls, the H5Ovisit callback the H5O_info_t type and some VOL connector APIs. Release candidate will be created as soon as new versions of those APIs are available, which will be the default for HDF5 1.12.0.
Major changes from HDF5 1.12.0-alpha0:
- Added support for new references to h5dump and h5ls tools.
- Improved performance when creating a large number of small datasets.
- New public functions H5Sselect_adjust, H5Sselect_project_intersection, and H5VLget_file_type,which may be useful when developing VOL connectors.
- Updated DDL documentation with the new types of references, VDS and opaque types:
- https://portal.hdfgroup.org/display/HDF5/DDL+in+BNF+for+HDF5-1.12+and+above
For more information about the release see
https://gamma.hdfgroup.org/ftp/pub/outgoing/hdf5_1.12/hdf5-1.12.0-alpha1-RELEASE.txt
In this release, we introduced several new features and along with performance improvement for hyperslab selection. A short description of the features is below.
Virtual Object Layer (VOL)
In this major HDF5 release we introduce HDF5 Virtual Object Layer (VOL). VOL is an abstraction layer within the HDF5 library that enables different methods for accessing data and objects that conform to the HDF5 data model. The VOL layer intercepts all HDF5 API calls that potentially modify data on disk and forwards those calls to a plugin “object driver”. The data on disk can be a different format than the HDF5 format. For more information about VOL we refer the reader to the following documents (under review):
VOL HDF5 APIs
For available VOL connectors please see
https://bitbucket.hdfgroup.org/projects/HDF5VOL
We are asking VOL connector developers in particular to test this release and let us know about any problems ASAP. Please let us know if you want to contribute your code to the repository above to be more visible to HDF5 community.
Enhancements to HDF5 References
HDF5 references were extended to support attributes, and object and dataset selections that reside in another HDF5 file. For more information including a list of new APIs, see
https://portal.hdfgroup.org/display/HDF5/Update+to+References
This feature requires HDF5 File Format extension. HDF5 Libraries prior to 1.12.0 will not be able to read them.
New S3 and HDFS Virtual File Drivers (VFDs)
This release has two new VFDs. The S3 VFD allows accessing HDF5 files on AWS S3 buckets. HDFS VFD allows accessing HDF5 file stored on Apache HDFS. See
https://portal.hdfgroup.org/display/HDF5/Virtual+File+Drivers+-+S3+and+HDFS
for information on enabling those drivers and using those APIs.
Hyperslab selection performance improvements
In 1.12.0 we optimized hyperslab selection code to achieve better performance. In general, performance improved by an order of magnitude. In the case of reading a regular selection from 20GB dataset into a one dimensional array performance was improved by a factor of 6000. If you are interested in the benchmark we ran, please login to JIRA and see
https://jira.hdfgroup.org/browse/HDFFV-10930.
If you have time to test this alpha release, we would greatly appreciate it. We test HDF5 on a variety of platforms and with multiple compilers, but there is always a system that we couldn’t test on. Your feedback is critical.
Your prompt reports of any issues found will be invaluable and very much appreciated. Unless delayed to address serious newly discovered issues, we expect the final 1.12.0 release in about two months. Thank you and happy testing!
Elena Pourmal
Larry Knox