Hi All,
The file image operations features discussed in the RFC I posted last July
have been implemented, and a snap shot is available for down load.
For those who may not remember, the driving use case behind these
extensions is to use of the HDF5 library as a packetizing/depacketizing
tool. Simply put, in some cases it is cheaper to construct an image of
an HDF5 file in a buffer, and then transmit the image to another machine
than it is to transfer the same data by writing a file to a commonly
accessible file system.
If this is something you are interested in doing, please give the
current version of the RFC a read, and if appropriate, try out the features
in the snapshot.
The RFC is available in PDF format via ftp at:
Note that there have been many changes in the RFC since the previous
post.
The snapshot is available via ftp at:
ftp://hdfgroup.uiuc.edu/pub/outgoing/mainzer/file_image.21863/hdf5-1.9.102-of20120104.tar.gz
User level documentation is still in progress, but the RFC should tell
you everything you need to know to use the new features.
The RFC is 40 pages long, so the following conceptual summary may
help you skim through and find the points of interest.
The core file driver can already be used to construct buffers containing
HDF5 files, and we already have facilities (albeit nasty ones) that allow the
user access to these buffers. However, on the receiving end, there is no
way to open a buffer containing an HDF5 file without first writing the buffer
to the file system.
In essence, the file image operations extensions to HDF5 remedy this
situation.
To do this, we implemented two new sets of sets of property list calls,
along with supporting code, and modifications to the core file driver.
The first set of property list calls inserts and extracts copies of buffers
containing HDF5 file images into and out of file access property lists. Note
that since the property list structure is designed to support call by value,
this results in two buffer copies -- one when the buffer is copied into a
FAPL, and a second when the buffer is copied from the FAPL into the core
file driver.
The second set of property list calls allows the application to specify
callbacks that are used by the property list code and the core file driver
when allocating, copying, reallocating and freeing buffers containing file
images. From the HDF5 library's perspective, these calls must be
functionally identical to the standard malloc, memcpy, realloc, and free C
library calls -- although as you will see in the RFC, they can be designed in
such a way as to avoid the buffer copies mentioned above.
In addition, we implemented a high level library calls to facilitate
the more common operations on file images, and a low level call to make
it easier to obtain file images.
Again, if this sort of functionality is of potential interest to you,
please take a look and let us know what you think.
Best regards,
John Mainzer