I am a new developer in an automotive company and I wanted to ask you about the current state of HDF5 regarding:
Real-time logging of data from multiple sensors (haven’t measured but I’d say around 5 mbps of data, our sensors work at 25 Hz to 12 Hz)
Use cases for point-cloud data (as in, LiDAR sensor data)
I have looked at the forum archives but the only relevant result I found was from 2012 and about driving simulators. Responses varied, so I feel like this question is again relevant to ask.
I thought of HDF5 as our data doesn’t really need any relational features of a traditional database, and could benefit from the grouping and metadata features of HDF5.
I faced similar problem few years back then with financial data: live stream bid/ask/trade of high frequency trading events. The final result is in H5CPP an easy to use high performance MIT licensed header only library for modern C++ presented in this webinar.
In the examples directory there are entries for various patterns including a high performance packet table. The IO performance is on par or better with POSIX calls: ~300MByte/sec on my laptop.
You might also be interested in the compiler assisted reflection feature, which allows you to use arbitrary deep POD struct by simple referencing it with one of the h5::read | h5::write | h5::create | h5::append operators. The LLVM based compiler tool will scan the TU translation unit and generate the minimum size HDF5 compound datatype descriptor as shown this example and this comparative one.
The work flow is simple: include the C/C++ files in the TU translation unit, and leave the details for the compilers.
