parallel file writing performance
Hello,
I need to write a large table to a file (~600GB) and I’m using HDF5 and MPI. I should point out that this is new to me, so I’m probably missing something essential.
I’ve got a little test code here:
written from one of the examples found on the HDF5Group site, and what I don’t understand is that if I run it on my machine for an array with a size of 2^24 complexes, it takes less than a minute, and if I run it on cluster, after 1/4h I still don’t have the result!
On the sequential machine, I’m on a mechanical disk, and on cluster, the parallel file system is GPFS.
For the slurm script, I’ve reserved 2 nodes and all the memory for each node.
Does anyone have an idea?
Thanks in advance for help