# Hdf5 for in memory circular buffer? **URL:** \ **Category:** Hdf-forum archives\ **Created:** [December 10, 2016, 4:50am UTC](https://forum.hdfgroup.org/t/hdf5-for-in-memory-circular-buffer/3931 "2016-12-10T04:50:13Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![David\_Schneider](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/d/7ea924/32.png) [@David\_Schneider](https://forum.hdfgroup.org/u/David_Schneider)\ **Post date:** [December 10, 2016, 4:50am UTC](https://forum.hdfgroup.org/t/hdf5-for-in-memory-circular-buffer/3931/1 "2016-12-10T04:50:13Z")
Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data. best, David Schneider SLAC/LCLS ---
**Author:** ![werner](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/w/f4b2a3/32.png) [@werner](https://forum.hdfgroup.org/u/werner)\ **Post date:** [December 11, 2016, 12:51pm UTC](https://forum.hdfgroup.org/t/hdf5-for-in-memory-circular-buffer/3931/2 "2016-12-11T12:51:08Z")
Hi,   there is a virtual file driver that operates on a memory image of an HDF5 file. It should be no problem to have this one also operate on shared memory.         Werner > **···** > > On 10.12.2016 05:50, Schneider, David A. wrote: > > > Is it possible to implement a in memory circular buffer using hdf5? This would be for multiple processes, sychronizing in a consumer publisher pattern, or maybe the server just writes to the circular buffer. Our data acquisition system is moving to hdf5 for offline files, but it also has to publish data to shared memory for some online analysis. It would be great if we could use the same schema and interface to the data for both offline and shared memory access. We're imaging that we will use SWMR to analysis the data acquisition data as it is being written to disk, but I'm not sure how this will work for shared memory. On disk, we just append to the datasets, in memory, we'd be overwriting old data. > > > > best, > > > > David Schneider > > SLAC/LCLS > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > [http://lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > Twitter: [x.com](https://twitter.com/hdf5) > > -- > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > Dr. Werner Benger Visualization Research > Center for Computation & Technology at Louisiana State University (CCT/LSU) > 2019 Digital Media Center, Baton Rouge, Louisiana 70803 > Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 ---
**Author:** ![steven.varga](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/steven.varga/32/1614_2.png) [@steven.varga](https://forum.hdfgroup.org/u/steven.varga)\ **Post date:** [December 11, 2016, 3:08pm UTC](https://forum.hdfgroup.org/t/hdf5-for-in-memory-circular-buffer/3931/3 "2016-12-11T15:08:35Z")
Hello, I am no expert by any means in your field, but I found that in HFT (high-frequency trading and similar fields) similar pattern often come up. Boost circular/ring buffer can handle the storage which one process writes and many reads from; upon tail eviction/flushing the pocket one can channel it to the write process --in case of fault tolerance needed you can have many ring buffers on different hosts. The above scheme can be done with various transportation mechanism ie: zeroMQ + protocol buffers/thrift. This is for production use. For running experiments where being fail-safe has little or no benefit one can access the HDF5 datasets locally on each cluster nodes; using MPI + Grid Engine + some\_software\_task + HDF5\_serial pattern. If you find the above interesting, feel free to ask. best, steve > **···** > > On Sun, Dec 11, 2016 at 7:51 AM, Werner Benger \ wrote: > > > Hi, > > > > there is a virtual file driver that operates on a memory image of an HDF5 > > file. It should be no problem to have this one also operate on shared > > memory. > > > >        Werner > > > > On 10.12.2016 05:50, Schneider, David A. wrote: > > > > > Is it possible to implement a in memory circular buffer using hdf5? This > > > would be for multiple processes, sychronizing in a consumer publisher > > > pattern, or maybe the server just writes to the circular buffer. Our data > > > acquisition system is moving to hdf5 for offline files, but it also has to > > > publish data to shared memory for some online analysis. It would be great > > > if we could use the same schema and interface to the data for both offline > > > and shared memory access. We're imaging that we will use SWMR to analysis > > > the data acquisition data as it is being written to disk, but I'm not sure > > > how this will work for shared memory. On disk, we just append to the > > > datasets, in memory, we'd be overwriting old data. > > > > > > best, > > > > > > David Schneider > > > SLAC/LCLS > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Hdf-forum is for HDF software users discussion. > > > Hdf-forum@lists.hdfgroup.org > > > [http://lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > > Twitter: [x.com](https://twitter.com/hdf5) > > > > -- > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Dr. Werner Benger Visualization Research > > Center for Computation & Technology at Louisiana State University (CCT/LSU) > > 2019 Digital Media Center, Baton Rouge, Louisiana 70803 > > Tel.: +1 225 578 4809 Fax.: +1 225 578-5362 > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@lists.hdfgroup.org > > [http://lists.hdfgroup.org/mailman/listinfo/hdf-forum\_lists.hdfgroup.org](http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org) > > Twitter: [x.com](https://twitter.com/hdf5)