For efficiency reasons we need to be able to create an HDF5 file in memory
and then send it over the network without ever writing it to the hard disk.
In-memory creation is available through using the CORE driver but is there a
way to perform CORE file read and write e.g. to a network socket?
This would be the HDF5 Streaming VFD that was created in the German
TIKSL project by Thomas Radke. It was used within HDF5 1.6.2, a recent
version that would be compatible with the 1.8 is currently available
as part of F5 (but is independent, basically it's just a .h and .c file):
I haven't used it for a while, but it should still work.
Werner
···
On Tue, 07 Dec 2010 16:10:21 +0100, AV <Alex.Voronel@cibc.ca> wrote:
For efficiency reasons we need to be able to create an HDF5 file in memory
and then send it over the network without ever writing it to the hard disk.
In-memory creation is available through using the CORE driver but is there a
way to perform CORE file read and write e.g. to a network socket?
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
I thought the Army Research Lab did something like this with their NDGM
(Network Distributed Global Memory) and a custom HDF5 virtual file
driver. You might try contacting someone at ARL to see if NDGM is still
around.
At least one tricky part is the 'server' processes you need to have to
manage the 'other end' of the sockets.
Mark
···
On Tue, 2010-12-07 at 07:10 -0800, AV wrote:
For efficiency reasons we need to be able to create an HDF5 file in memory
and then send it over the network without ever writing it to the hard disk.
In-memory creation is available through using the CORE driver but is there a
way to perform CORE file read and write e.g. to a network socket?
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================ miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Thanks Werner and Mark, for the links. It appears that our requirements are
not unique. We will try to see if we can reuse the drivers developed for
those other projects.
