Write many datasets at the same time using OpenMP

The use case you are describing is currently not supported, in the following sense. For any kind of multithreading you need a thread-safe build of the library. If unsure, you can determine that by calling H5is_library_threadsafe (https://portal.hdfgroup.org/display/HDF5/H5_IS_LIBRARY_THREADSAFE). With a thread-safe build, your application should run fine, but you may not see any throughput gains or even slow down. The main reason is that the current implementation of the HDF5 library does not use any multithreading internally. At any given time, only one of your threads will be able “to be in the library” and do things such as create new objects, do I/O, or close handles.

How big are you armadillo objects (DATA)? There’s conceptually nothing wrong with your example. However, you might be faster by pipelining/blocking the creation of your datasets, assuming throughput is the metric.

G.

1 Like