Concurrent read of large dataset by multiple threads on Windows


#1

Hello. My use case is an OPENMP application that need to read a large dataset in a h5 file (opened in read-only mode). I want to avoid a critical section where the threads read the dataset. Will SWMR allow this type of concurrent read? I want this to work on Windows 8.1 and Windows 10.


#2

The control flows referred to by ‘W’ and ‘R’ in SWMR are (OS) processes, not threads. (That’s what OpenMP uses, right?)

What are the types/layouts of the dataset(s) you are trying to read? Are they compressed?

G.


#3

Yes OPENMP uses threading. The dataset is 3D double table with chunking. It is compressed with GZIP. Is there any chance of reading this table concurrently?


#4

Yes, you can obtain the chunk addresses and size via H5Dget_chunk_info or H5Dget_chunk_info_by_coord. After that, you don’t need the HDF5 library anymore. You’ll have to decompress the chunks yourself, but that should be straightforward. G.