I am developing an unstructured grid based CFD algorithm for
Petascale applications. I have written a program using MPI, which puts
unstructured mesh information (vertices, faces and cells) into a H5 file.
The H5 file will be read by the CFD code for solving equations. My question
is two fold.
1.) What is the best way of organizing data in the file? I am planning to
have just 3 simple datasets for vertices, faces and cells. Should the
layout be 1-dimensional or chunked? The data is contiguous and there is no
contention among different processors during the write phase. Should I
worry about queuing I/O calls upto a limit and then flush the data. Should
I also worry about alignment issues (respecting stripe boundaries)?
2.) My next question is how do I efficiently read the data in an
unstructured manner from the file? The data to be read is not contiguous
and there is contention among processors? I am, presently, using
H5S_select_elements to do this. Is there a better way to do this? Will the
HDF5 API buffer data upto a limit and then transfer to memory?
1.) What is the best way of organizing data in the file? I am planning to
have just 3 simple datasets for vertices, faces and cells. Should the
layout be 1-dimensional or chunked? The data is contiguous and there is no
contention among different processors during the write phase. Should I
worry about queuing I/O calls upto a limit and then flush the data. Should
I also worry about alignment issues (respecting stripe boundaries)?
My advice is predicated on using HDF5's MPI-IO driver and turning on
collective i/o through an HDF5 property list.
If the data is contiguous and there is no contention, then you can
write however you like just about. The HDF5 library and the MPI-IO
implementation should deal with file alignment.
Are you going to write the data often (like a checkpoint)?
2.) My next question is how do I efficiently read the data in an
unstructured manner from the file? The data to be read is not contiguous
and there is contention among processors? I am, presently, using
H5S_select_elements to do this. Is there a better way to do this? Will the
HDF5 API buffer data upto a limit and then transfer to memory?
Are you going to read this data lots and lots of times? or is your
post-simulation analysis a one-time thing?
Optimize for the more frequent case, of course.
If you are write-often, read-once, then your layout seems fine. Turn
on MPI-IO and collective I/O and it matters a lot less how data is
laid out.
==rob
···
On Sat, May 11, 2013 at 10:31:05AM +0530, Suman Vajjala wrote:
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
