# Append HDF5 files in parallel **URL:** \ **Category:** HDF5 Library\ **Created:** [October 28, 2021, 10:47pm UTC](https://forum.hdfgroup.org/t/append-hdf5-files-in-parallel/9098 "2021-10-28T22:47:43Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![wkliao](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/w/77aa72/32.png) [@wkliao](https://forum.hdfgroup.org/u/wkliao)\ **Post date:** [October 28, 2021, 10:47pm UTC](https://forum.hdfgroup.org/t/append-hdf5-files-in-parallel/9098/1 "2021-10-28T22:47:43Z")
I have thousands of HDF5 files that need to be merged into a single file. Merging is simply to append all groups and datasets of one file after another in a new output file. The group names of the input files are all different from one another. In addition, all datasets are chunked and compressed. My question is how do I merge the files in parallel? My implementation consists of the following steps: 1. all MPI processes collectively open all input files, 2. all MPI processes collectively create the single shared output file, 3. all processes read the metadata of groups and datasets of all input files and create the same in the output file, 4. assigns the input files evenly among all processes, 5. each process independently reads datasets from the assigned files and writes them to the output file. My intent is such that each process can read and write disjoined sets of datasets in parallel. However, I encountered the error below indicating the compressed datasets can only be written in collective mode. HDF5-DIAG: Error detected in HDF5 (1.12.0) MPI-process 15: #000: …/…/hdf5-1.12.0/src/H5Dio.c line 314 in H5Dwrite(): can’t write data #005: …/…/hdf5-1.12.0/src/H5Dio.c line 1194 in H5D\_\_ioinfo\_adjust(): Can’t perform independent write with filters in pipeline. Can this file merge operation be done in parallel for compressed datasets? Wei-keng ---
**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\ **Post date:** [October 29, 2021, 3:21pm UTC](https://forum.hdfgroup.org/t/append-hdf5-files-in-parallel/9098/2 "2021-10-29T15:21:20Z")
I think you definitely want to use `H5D[read,write]_chunk()` to bypass the filter pipeline. Unfortunately, they aren’t supported w/ MPI, at the moment. How big are the chunks? To maximize bandwidth utilization, maybe you want to `H5Dread_chunk` as many chunks as you can on different ranks from different files, and buffer them in RAM, while you have a single writer that is being fed the chunks (locally and via the interconnect) and that does the `H5Dwrite_chunk` writing. This is admittedly a little kludgy but might get the job done in a reasonable amount of time. Best, G. ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [October 29, 2021, 6:15pm UTC](https://forum.hdfgroup.org/t/append-hdf5-files-in-parallel/9098/3 "2021-10-29T18:15:24Z")
If you don’t actually require everything to get merged into one file (but just want it to look that way from the application’s point of view), you could just create a master file with a bunch of external links to groups/datasets in the other files. This would easy enough that one process would suffice to create the master file. ---
**Author:** ![wkliao](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/w/77aa72/32.png) [@wkliao](https://forum.hdfgroup.org/u/wkliao)\ **Post date:** [October 29, 2021, 9:01pm UTC](https://forum.hdfgroup.org/t/append-hdf5-files-in-parallel/9098/4 "2021-10-29T21:01:00Z")
Using external links is an Interesting idea. I will give it a try. Thanks. ---
**Author:** ![wkliao](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/w/77aa72/32.png) [@wkliao](https://forum.hdfgroup.org/u/wkliao)\ **Post date:** [October 29, 2021, 9:12pm UTC](https://forum.hdfgroup.org/t/append-hdf5-files-in-parallel/9098/5 "2021-10-29T21:12:22Z")
I used H5Dread\_chunk and H5Dwrite\_chunk as suggested. The error message was gone. This appears to work for independent I/O mode. However, I was encountering a new problem. The program hangs at H5Fclose. After some digging into HDF5 source codes, I found the hanging is inside of H5FD\_\_mpio\_truncate() and the same problem was reported earlier by @gdsjaar in [Hang in H5Fclose](https://forum.hdfgroup.org/t/hang-in-h5fclose/5615) I added a printf statement to check the values of ‘size’ and ‘needed\_eof’. Below is what I got from my program running on 2 MPI processes. [0] …/…/hdf5-1.12.0/src/H5FDmpio.c line 1654: size=5113724 needed\_eof=4683672 [1] …/…/hdf5-1.12.0/src/H5FDmpio.c line 1654: size=5113724 needed\_eof=5113724 My program hangs because of the MPI\_Barrier called inside the if condition: if(size != needed\_eof) {