Query regarding Parallel HDF5 dataset creation

Hello All,

       I am trying to write a HDF5 file in parallel in a specific format.
Each rank has variable length data and data of each process has to be
created as a separate dataset e.g. /data_i (i is the rank). I have used the
following code but to no success:

/////Code
    data = (int *) malloc(sizeof(int)*dimsf[0]*dimsf[1]);
    for (i=0; i < dimsf[0]*dimsf[1]; i++) {
        data[i] = mpi_rank;
    }

    plist_id = H5Pcreate(H5P_FILE_ACCESS);
    H5Pset_fapl_mpio(plist_id, comm, info);

    file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id);
    H5Pclose(plist_id);

    plist_id = H5Pcreate(H5P_DATASET_XFER);
    H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT);

    /*
     * Create the dataspace for the dataset.
     */
    filespace = H5Screate_simple(RANK, dimsf, NULL);

    i = mpi_rank;
    sprintf(name,"/node%d",i);
    dset_id = H5Dcreate(file_id, name, H5T_NATIVE_INT,
filespace,H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

    status = H5Dwrite(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
H5P_DEFAULT,
data);
    H5Dclose(dset_id);

  Can you please shed light on how to do it?

Regards
Suman Vajjala

Hi Suman,

Parallel HDF5 doesn't support multiple write feature.
So this would result in only one process (out of many) would perform any writing IO.
(even dset name and data value (indicate process) can be from different process)

You may be able to do it by completely serialize writing per process from create/reopen/close file, but that would defeat the purpose of parallel.

Jonathan

···

On 12/11/2012 12:49 AM, Suman Vajjala wrote:

Hello All,

       I am trying to write a HDF5 file in parallel in a specific format. Each rank has variable length data and data of each process has to be created as a separate dataset e.g. /data_i (i is the rank). I have used the following code but to no success:

/////Code
    data = (int *) malloc(sizeof(int)*dimsf[0]*dimsf[1]);
    for (i=0; i < dimsf[0]*dimsf[1]; i++) {
        data[i] = mpi_rank;
    }

    plist_id = H5Pcreate(H5P_FILE_ACCESS);
    H5Pset_fapl_mpio(plist_id, comm, info);

    file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id);
    H5Pclose(plist_id);

    plist_id = H5Pcreate(H5P_DATASET_XFER);
    H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT);

    /*
     * Create the dataspace for the dataset.
     */
    filespace = H5Screate_simple(RANK, dimsf, NULL);

    i = mpi_rank;
    sprintf(name,"/node%d",i);
    dset_id = H5Dcreate(file_id, name, H5T_NATIVE_INT, filespace,H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

    status = H5Dwrite(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
data);
    H5Dclose(dset_id);

  Can you please shed light on how to do it?

Regards
Suman Vajjala

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Just want to clarify....

H5Dcreate ia a collective call (i.e, all processes have to participate in creation of a sprintf(name,"/node%d",i); dataset). Then each process can write to its own dataset.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Dec 11, 2012, at 10:50 AM, Jonathan Kim wrote:

Hi Suman,

Parallel HDF5 doesn't support multiple write feature.
So this would result in only one process (out of many) would perform any writing IO.
(even dset name and data value (indicate process) can be from different process)

You may be able to do it by completely serialize writing per process from create/reopen/close file, but that would defeat the purpose of parallel.

Jonathan

On 12/11/2012 12:49 AM, Suman Vajjala wrote:

Hello All,

       I am trying to write a HDF5 file in parallel in a specific format. Each rank has variable length data and data of each process has to be created as a separate dataset e.g. /data_i (i is the rank). I have used the following code but to no success:

/////Code
    data = (int *) malloc(sizeof(int)*dimsf[0]*dimsf[1]);
    for (i=0; i < dimsf[0]*dimsf[1]; i++) {
        data[i] = mpi_rank;
    }

    plist_id = H5Pcreate(H5P_FILE_ACCESS);
    H5Pset_fapl_mpio(plist_id, comm, info);

    file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id);
    H5Pclose(plist_id);

    plist_id = H5Pcreate(H5P_DATASET_XFER);
    H5Pset_dxpl_mpio(plist_id, H5FD_MPIO_INDEPENDENT);

    /*
     * Create the dataspace for the dataset.
     */
    filespace = H5Screate_simple(RANK, dimsf, NULL);

    i = mpi_rank;
    sprintf(name,"/node%d",i);
    dset_id = H5Dcreate(file_id, name, H5T_NATIVE_INT, filespace,H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);

    status = H5Dwrite(dset_id, H5T_NATIVE_INT, H5S_ALL, H5S_ALL, H5P_DEFAULT,
data);
    H5Dclose(dset_id);

  Can you please shed light on how to do it?

Regards
Suman Vajjala

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi,

I thank you for the replies. The problem is that for creating a dataset
using H5Dcreate every processor has to know that dataset's size. Is there a
way of of doing it without the processors knowing the data size.

Regards
Suman

···

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org>wrote:

reate ia

Yes, you can do this.

You will need to use chunking and unlimited dimensions when you create each dataset, and issue H5Dset_extent (http://www.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-SetExtent) to set the correct size before writing. Unfortunately this call is also collective.

···

On Dec 11, 2012, at 9:33 PM, Suman Vajjala wrote:

Hi,

I thank you for the replies. The problem is that for creating a dataset using H5Dcreate every processor has to know that dataset's size. Is there a way of of doing it without the processors knowing the data size.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Regards
Suman

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:
reate ia

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi,

I thank you for the replies. The problem is that for creating a dataset using H5Dcreate every processor has to know that dataset's size. Is there a way of of doing it without the processors knowing the data size.

If the dataset size if based on the process rank, then you can just calculate that at each process then call H5Dcreate collectively on all processes for each dataset. If you can't calculate that size at each process, you can use MPI_Allgather to distribute the size for each process to all other processes then you would be able to call the H5Dcreate n times.

This is a limitation currently in the HDF5 standard that we plan to relax in the future. But this work is still in the prototyping stage. Ping me early January and might have a beta version that you might be able to use.

Mohamad

···

On Dec 11, 2012, at 9:33 PM, Suman Vajjala <suman.geek@gmail.com> wrote:

Regards
Suman

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

reate ia

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Of course, a much simpler way to do this is to create the file using MPI_COMM_SELF, create the data sets independently as you did since collective on comm_self is well, independent, and close the file.
Then you can actually open the file again using comm_world, open the dataset for each process (H5Dopen is not collective), then access the dataset for each process independently as you were doing.

Mohamad

···

On Dec 11, 2012, at 10:19 PM, Mohamad Chaarawi <chaarawi@hdfgroup.org> wrote:

On Dec 11, 2012, at 9:33 PM, Suman Vajjala <suman.geek@gmail.com> wrote:

Hi,

I thank you for the replies. The problem is that for creating a dataset using H5Dcreate every processor has to know that dataset's size. Is there a way of of doing it without the processors knowing the data size.

If the dataset size if based on the process rank, then you can just calculate that at each process then call H5Dcreate collectively on all processes for each dataset. If you can't calculate that size at each process, you can use MPI_Allgather to distribute the size for each process to all other processes then you would be able to call the H5Dcreate n times.

This is a limitation currently in the HDF5 standard that we plan to relax in the future. But this work is still in the prototyping stage. Ping me early January and might have a beta version that you might be able to use.

Mohamad

Regards
Suman

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

reate ia

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Of course, a much simpler way to do this is to create the file using MPI_COMM_SELF, create the data sets independently as you did since collective on comm_self is well, independent, and close the file.

Ahh wait sorry i was not thinking right since it will be the same file. this will not work as bad things will happen with this access pattern so disregard this..

Mohamad

···

On Dec 11, 2012, at 10:26 PM, Mohamad Chaarawi <chaarawi@hdfgroup.org> wrote:

Then you can actually open the file again using comm_world, open the dataset for each process (H5Dopen is not collective), then access the dataset for each process independently as you were doing.

Mohamad

On Dec 11, 2012, at 10:19 PM, Mohamad Chaarawi <chaarawi@hdfgroup.org> wrote:

On Dec 11, 2012, at 9:33 PM, Suman Vajjala <suman.geek@gmail.com> wrote:

Hi,

I thank you for the replies. The problem is that for creating a dataset using H5Dcreate every processor has to know that dataset's size. Is there a way of of doing it without the processors knowing the data size.

If the dataset size if based on the process rank, then you can just calculate that at each process then call H5Dcreate collectively on all processes for each dataset. If you can't calculate that size at each process, you can use MPI_Allgather to distribute the size for each process to all other processes then you would be able to call the H5Dcreate n times.

This is a limitation currently in the HDF5 standard that we plan to relax in the future. But this work is still in the prototyping stage. Ping me early January and might have a beta version that you might be able to use.

Mohamad

Regards
Suman

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

reate ia

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Thanks Elena. I'll have to think of a way to avoid this situation.

Suman

···

On Wed, Dec 12, 2012 at 9:30 AM, Elena Pourmal <epourmal@hdfgroup.org>wrote:

On Dec 11, 2012, at 9:33 PM, Suman Vajjala wrote:

Hi,

I thank you for the replies. The problem is that for creating a dataset
using H5Dcreate every processor has to know that dataset's size. Is there a
way of of doing it without the processors knowing the data size.

Yes, you can do this.

You will need to use chunking and unlimited dimensions when you create
each dataset, and issue H5Dset_extent (
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5D.html#Dataset-SetExtent) to set
the correct size before writing. Unfortunately this call is also collective.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Regards
Suman

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org>wrote:

reate ia

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Thanks Mohamad for the info. I will surely look to try out the upcoming
beta version.

Suman

···

On Wed, Dec 12, 2012 at 9:59 AM, Mohamad Chaarawi <chaarawi@hdfgroup.org>wrote:

On Dec 11, 2012, at 10:26 PM, Mohamad Chaarawi <chaarawi@hdfgroup.org> > wrote:

Of course, a much simpler way to do this is to create the file using
MPI_COMM_SELF, create the data sets independently as you did since
collective on comm_self is well, independent, and close the file.

Ahh wait sorry i was not thinking right since it will be the same file.
this will not work as bad things will happen with this access pattern so
disregard this..

Mohamad

Then you can actually open the file again using comm_world, open the
dataset for each process (H5Dopen is not collective), then access the
dataset for each process independently as you were doing.

Mohamad

On Dec 11, 2012, at 10:19 PM, Mohamad Chaarawi <chaarawi@hdfgroup.org> > wrote:

On Dec 11, 2012, at 9:33 PM, Suman Vajjala <suman.geek@gmail.com> wrote:

Hi,

I thank you for the replies. The problem is that for creating a dataset
using H5Dcreate every processor has to know that dataset's size. Is there a
way of of doing it without the processors knowing the data size.

If the dataset size if based on the process rank, then you can just
calculate that at each process then call H5Dcreate collectively on all
processes for each dataset. If you can't calculate that size at each
process, you can use MPI_Allgather to distribute the size for each process
to all other processes then you would be able to call the H5Dcreate n times.

This is a limitation currently in the HDF5 standard that we plan to relax
in the future. But this work is still in the prototyping stage. Ping me
early January and might have a beta version that you might be able to use.

Mohamad

Regards
Suman

On Wed, Dec 12, 2012 at 8:58 AM, Elena Pourmal <epourmal@hdfgroup.org>wrote:

reate ia

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org