Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data.
there is a virtual file driver that operates on a memory image of an HDF5 file. It should be no problem to have this one also operate on shared memory.
Werner
···
On 10.12.2016 05:50, Schneider, David A. wrote:
Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data.
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
I am no expert by any means in your field, but I found that in HFT
(high-frequency trading and similar fields) similar pattern often come up.
Boost circular/ring buffer can handle the storage which one process writes
and many reads from; upon tail eviction/flushing the pocket one can channel
it to the write process --in case of fault tolerance needed you can have
many ring buffers on different hosts.
The above scheme can be done with various transportation mechanism ie:
zeroMQ + protocol buffers/thrift. This is for production use.
For running experiments where being fail-safe has little or no benefit one
can access the HDF5 datasets locally on each cluster nodes; using MPI +
Grid Engine + some_software_task + HDF5_serial pattern.
If you find the above interesting, feel free to ask.
best,
steve
···
On Sun, Dec 11, 2016 at 7:51 AM, Werner Benger <werner@cct.lsu.edu> wrote:
Hi,
there is a virtual file driver that operates on a memory image of an HDF5
file. It should be no problem to have this one also operate on shared
memory.
Werner
On 10.12.2016 05:50, Schneider, David A. wrote:
Is it possible to implement a in memory circular buffer using hdf5? This
would be for multiple processes, sychronizing in a consumer publisher
pattern, or maybe the server just writes to the circular buffer. Our data
acquisition system is moving to hdf5 for offline files, but it also has to
publish data to shared memory for some online analysis. It would be great
if we could use the same schema and interface to the data for both offline
and shared memory access. We're imaging that we will use SWMR to analysis
the data acquisition data as it is being written to disk, but I'm not sure
how this will work for shared memory. On disk, we just append to the
datasets, in memory, we'd be overwriting old data.
--
____________________________________________________________
_______________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
