Using CORE vfd with existing buffer?

Hi,

Does anyone know if it's possible to open a "file" with the CORE
driver from e.g. an existing malloc'ed buffer containing the image of
an HDF5 file?

Thanks,
Andrew Collette

I am pretty certain it CANNOT do that.

On read, the core vfd will open the named file, allocate a buffer large
enough for it (modulo allocation inc) and read its entire contents into
that buffer.

However, it shouldn't be too hard to enhance H5FDcore.c to do what you
need. I don't know if that driver is written with or without knowledge
of HDF5 library 'private' stuff.

If it does NOT use HDF5 private stuff, then copy H5FDcore.c to your own
source tree, replace the names of all the functions to something
different like H5FD_mycore_<whatever> and then add an argument to the
set_fapl method for void *buf. In the open method, examine that property
and if its non-null, skip the alloc and read and just set the file
pointer's buf member to whatever buffer address was passed in the
set_fapl. Then, in your application call H5Pset_fapl_mycore(alloc,
back_store, buf); where 'buf' is the 'image' you've allocated. Compile
H5FDmycore.c into your application.

If the H5FDcore.c uses HDF5 private stuff, then its more work to do this
because it can't compile easily outside HDF5's source code tree.

Mark

···

On Thu, 2010-03-25 at 11:49, Andrew Collette wrote:

Hi,

Does anyone know if it's possible to open a "file" with the CORE
driver from e.g. an existing malloc'ed buffer containing the image of
an HDF5 file?

Thanks,
Andrew Collette

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

Hi Andrew

you have to play a bit with the driver. I use it to perform HDF5 IO in
memory with or without file backup. Actually I copy objects from a disk file
to a memory file (usually I use compression as well), attach the buffer to
an XML message with some metadata, pass it through the network and read the
buffer on the other side and write the data into another disk file. Actually
if there was a function like H5F_set_vfd_handle along with the
H5F_get_vfd_handle it would work. Unfortunately, as usual, I cannot share
code....

HTH

-- dimitris

···

2010/3/25 Mark Miller <miller86@llnl.gov>

I am pretty certain it CANNOT do that.

On read, the core vfd will open the named file, allocate a buffer large
enough for it (modulo allocation inc) and read its entire contents into
that buffer.

However, it shouldn't be too hard to enhance H5FDcore.c to do what you
need. I don't know if that driver is written with or without knowledge
of HDF5 library 'private' stuff.

If it does NOT use HDF5 private stuff, then copy H5FDcore.c to your own
source tree, replace the names of all the functions to something
different like H5FD_mycore_<whatever> and then add an argument to the
set_fapl method for void *buf. In the open method, examine that property
and if its non-null, skip the alloc and read and just set the file
pointer's buf member to whatever buffer address was passed in the
set_fapl. Then, in your application call H5Pset_fapl_mycore(alloc,
back_store, buf); where 'buf' is the 'image' you've allocated. Compile
H5FDmycore.c into your application.

If the H5FDcore.c uses HDF5 private stuff, then its more work to do this
because it can't compile easily outside HDF5's source code tree.

Mark

On Thu, 2010-03-25 at 11:49, Andrew Collette wrote:
> Hi,
>
> Does anyone know if it's possible to open a "file" with the CORE
> driver from e.g. an existing malloc'ed buffer containing the image of
> an HDF5 file?
>
> Thanks,
> Andrew Collette
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Thanks to both of you for your suggestions... the context is that a
user of my package (h5py) is retrieving byte streams over a network or
database connection which represent HDF5 files. It would be nice if
there were a way to open and modify the files in memory, instead of
writing them to the file system, modifying, and then re-reading into
memory. Since it looks like HDF5 can't do this natively I suppose the
next best thing is to use tmpfs to store the temporary copy of the
file. Making a new driver is beyond the scope of the project. :slight_smile:

Andrew

···

On Thu, Mar 25, 2010 at 1:15 PM, Dimitris Servis <servisster@gmail.com> wrote:

Hi Andrew

you have to play a bit with the driver. I use it to perform HDF5 IO in
memory with or without file backup. Actually I copy objects from a disk file
to a memory file (usually I use compression as well), attach the buffer to
an XML message with some metadata, pass it through the network and read the
buffer on the other side and write the data into another disk file. Actually
if there was a function like H5F_set_vfd_handle along with the
H5F_get_vfd_handle it would work. Unfortunately, as usual, I cannot share
code....

HTH

-- dimitris