Hello,
I am working on a project using the OMPC runtime for task-based parallelism, where each task corresponds to an MPI process. These tasks can operate separately from one another. Some tasks are responsible for reading and writing chunks of a matrix stored in an HDF5 file.
To improve performance, I would like to enable compression during writes. However, since compression in HDF5 relies on filters, the write operation must be collective, requiring all processes to synchronize. Unfortunately, due to the design of my program, not all tasks can synchronize for the write operation.
Does anyone have any ideas or suggestions for enabling compressed writes in HDF5 without requiring full synchronization of all MPI processes? Any guidance or alternative approaches would be greatly appreciated.
Thanks! Any ideas and suggestions welcome.