Parallel dataset creation: unable to create and link to dataset

Hdf-forum archives
1

Hi,

I am trying to create a library to OpenFOAM (CFD toolbox from www.openfoam.org) to write the solution field to HDF5 archives.

Creating the file and dataspace seems to go well, but the creation of the dataset with H5Dcreate2 fails miserably. I have followed the example/tutorial found on http://www.hdfgroup.org/HDF5/Tutor/pcrtaccd.html

The error message is:

HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) MPI-process 0:
   #000: ../../../src/H5D.c line 171 in H5Dcreate2(): unable to create dataset
     major: Dataset
     minor: Unable to initialize object
   #001: ../../../src/H5Dint.c line 428 in H5D_create_named(): unable to create and link to dataset
     major: Dataset
     minor: Unable to initialize object
   #002: ../../../src/H5L.c line 1639 in H5L_link_object(): unable to create new link to object
     major: Links
     minor: Unable to initialize object
   #003: ../../../src/H5L.c line 1862 in H5L_create_real(): can't insert link
     major: Symbol table
     minor: Unable to insert object
   #004: ../../../src/H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal failed
     major: Symbol table
     minor: Object not found
   #005: ../../../src/H5Gtraverse.c line 776 in H5G_traverse_real(): component not found
     major: Symbol table
     minor: Object not found

and similar for other ranks. The source code is attached. When
meshInit();
meshWrite();
meshClose();
is called, the error occures (in H5DCreate2).

The following is in the h5Write.H header file:
#define H5T_SCALAR H5T_NATIVE_DOUBLE
(to include the possibility to compile with support for single precision).

The system is Ubuntu 12.04 with system OpenMPI and HDF5-OpenMPI libraries installed. Compiler is GCC. I have tried to compile the same code on another system with SLES 11sp1, Intel compilers and SGI-MPI. Same error message occurs on that system.

Do you have any hints to help me? Thanks in advance.

Best regards,
Håkon Strandenes
Norway

h5WriteMesh.C (4.24 KB)

2

and similar for other ranks. The source code is attached. When
meshInit();

meshInit sets the "use parallel I/O" property and opens the already
existing file

meshWrite();

which in turn calls meshWritePoints and meshWriteCells
I guess this is the problematic file.

You've opened the file H5_ACC_TRUNC, so that zaps everything in the
dataset. Then you create a dataset "/%f/points/processor%i".

I think you first need to create the '%f' and 'points' groups before
making processor%i, right? Or will HDF5 make parent groups in this
situation? nearly 10 years on and I still don't use the hierarchy
features of HDF5..

==rob

···

On Mon, Sep 10, 2012 at 05:20:50PM +0200, Håkon Strandenes wrote:

meshClose();
is called, the error occures (in H5DCreate2).

The following is in the h5Write.H header file:
#define H5T_SCALAR H5T_NATIVE_DOUBLE
(to include the possibility to compile with support for single precision).

The system is Ubuntu 12.04 with system OpenMPI and HDF5-OpenMPI
libraries installed. Compiler is GCC. I have tried to compile the
same code on another system with SLES 11sp1, Intel compilers and
SGI-MPI. Same error message occurs on that system.

Do you have any hints to help me? Thanks in advance.

Best regards,
Håkon Strandenes
Norway

/*---------------------------------------------------------------------------*\
  ========= |
  \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
   \\ / O peration |
    \\ / A nd | Copyright (C) 2011 OpenFOAM Foundation
     \\/ M anipulation |
-------------------------------------------------------------------------------
License
    This file is part of OpenFOAM.

    OpenFOAM is free software: you can redistribute it and/or modify it
    under the terms of the GNU General Public License as published by
    the Free Software Foundation, either version 3 of the License, or
    (at your option) any later version.

    OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
    ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
    FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
    for more details.

    You should have received a copy of the GNU General Public License
    along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>.

\*---------------------------------------------------------------------------*/

#include "h5Write.H"
#include "Time.H"
#include "SortableList.H"
#include "OFstream.H"
#include "string.H"
#include "stringOps.H"

#define H5DATA_MESH_FILE_NAME "h5Data/h5DataMesh.h5"

// * * * * * * * * * * * * * * Static Data Members * * * * * * * * * * * * * //

/*
const Foam::label Foam::h5Write::foamToXDMFFaceAddr[4][6] =
{
    { 4, 5, 2, 3, 0, 1 }, // 11 = pro-STAR hex
    { 0, 1, 4, 5, 2, -1 }, // 12 = pro-STAR prism
    { 5, 4, 2, 0, -1, -1 }, // 13 = pro-STAR tetra
    { 0, 4, 3, 5, 2, -1 } // 14 = pro-STAR pyramid
};
*/

// * * * * * * * * * * * * * Private Member Functions * * * * * * * * * * * //

void Foam::h5Write::meshInit()
{
    Info<< "h5Write::meshInit" << endl << endl;
    
    // Set up file access property list with parallel I/O access
    meshPlistID_ = H5Pcreate(H5P_FILE_ACCESS);
    H5Pset_fapl_mpio(meshPlistID_, MPI_COMM_WORLD, MPI_INFO_NULL);

    // Create a new file collectively.
    meshFileID_ = H5Fcreate
      (
          H5DATA_MESH_FILE_NAME,
          H5F_ACC_TRUNC,
          H5P_DEFAULT,
          meshPlistID_
      );
    
    // Close the property list
    H5Pclose(meshPlistID_);
    
}

void Foam::h5Write::meshClose()
{
    Info<< "h5Write::meshClose" << endl << endl;
    
    // Close the file.
    H5Fclose(meshFileID_);
    
}

void Foam::h5Write::meshWrite()
{
    meshWritePoints();
    meshWriteCells();
}

void Foam::h5Write::meshWritePoints()
{
    const pointField& points = mesh_.points();
    
    // Create the dataspace for the dataset
    hsize_t dimsf[] = {points.size(), 3};
    hid_t fileSpace = H5Screate_simple(2, dimsf, NULL);
    
    // Construct dataset name
    char datasetName[80];
    sprintf(datasetName, "/%f/points/processor%i", mesh_.time().value(), Pstream::myProcNo());
    
    //if (debug)
    {
        Pout<< "Dataset name: "<< datasetName << endl;
    }
    
    // Create the dataset with default properties
    hid_t dsetID = H5Dcreate2
        (
            meshFileID_,
            datasetName,
            H5T_SCALAR,
            fileSpace,
            H5P_DEFAULT,
            H5P_DEFAULT,
            H5P_DEFAULT
        );
    
    // Create property list for collective dataset write.
    meshPlistID_ = H5Pcreate(H5P_DATASET_XFER);
                   H5Pset_dxpl_mpio(meshPlistID_, H5FD_MPIO_COLLECTIVE);

    //if (debug)
    {
        Pout<< "Writing " << points.size() << " points" << endl;
    }
    
    // Create a simple array of points (to pass on to H5Dwrite)
    scalar pointList[points.size()][3];
    
    // Fill the array
    forAll(points, ptI)
    {
        pointList[ptI][0] = points[ptI].x();
        pointList[ptI][1] = points[ptI].y();
        pointList[ptI][2] = points[ptI].z();
    }
    
    // Do the actual write
    meshStatus_ = H5Dwrite
        (
            dsetID,
            H5T_SCALAR,
            H5S_ALL,
            H5S_ALL,
            meshPlistID_,
            pointList
        );
    
    // Close/release resources.
    H5Dclose(dsetID);
    H5Sclose(fileSpace);
    H5Pclose(meshPlistID_);
}

void Foam::h5Write::meshWriteCells()
{
    
}

// ************************************************************************* //

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

3

Tank you very much. I needed of course to create the groups before the dataset.

Regards

···

On 11. sep. 2012 21:17, Rob Latham wrote:

On Mon, Sep 10, 2012 at 05:20:50PM +0200, Håkon Strandenes wrote:

and similar for other ranks. The source code is attached. When
meshInit();

meshInit sets the "use parallel I/O" property and opens the already
existing file

meshWrite();

which in turn calls meshWritePoints and meshWriteCells
I guess this is the problematic file.

You've opened the file H5_ACC_TRUNC, so that zaps everything in the
dataset. Then you create a dataset "/%f/points/processor%i".

I think you first need to create the '%f' and 'points' groups before
making processor%i, right? Or will HDF5 make parent groups in this
situation? nearly 10 years on and I still don't use the hierarchy
features of HDF5..

==rob

meshClose();
is called, the error occures (in H5DCreate2).

The following is in the h5Write.H header file:
#define H5T_SCALAR H5T_NATIVE_DOUBLE
(to include the possibility to compile with support for single precision).

The system is Ubuntu 12.04 with system OpenMPI and HDF5-OpenMPI
libraries installed. Compiler is GCC. I have tried to compile the
same code on another system with SLES 11sp1, Intel compilers and
SGI-MPI. Same error message occurs on that system.

Do you have any hints to help me? Thanks in advance.

Best regards,
Håkon Strandenes
Norway

/*---------------------------------------------------------------------------*\
   ========= |
   \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
    \\ / O peration |
     \\ / A nd | Copyright (C) 2011 OpenFOAM Foundation
      \\/ M anipulation |
-------------------------------------------------------------------------------
License
     This file is part of OpenFOAM.

     OpenFOAM is free software: you can redistribute it and/or modify it
     under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
     (at your option) any later version.

     OpenFOAM is distributed in the hope that it will be useful, but WITHOUT
     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
     FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
     for more details.

     You should have received a copy of the GNU General Public License
     along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>.

\*---------------------------------------------------------------------------*/

#include "h5Write.H"
#include "Time.H"
#include "SortableList.H"
#include "OFstream.H"
#include "string.H"
#include "stringOps.H"

#define H5DATA_MESH_FILE_NAME "h5Data/h5DataMesh.h5"

// * * * * * * * * * * * * * * Static Data Members * * * * * * * * * * * * * //

/*
const Foam::label Foam::h5Write::foamToXDMFFaceAddr[4][6] =
{
     { 4, 5, 2, 3, 0, 1 }, // 11 = pro-STAR hex
     { 0, 1, 4, 5, 2, -1 }, // 12 = pro-STAR prism
     { 5, 4, 2, 0, -1, -1 }, // 13 = pro-STAR tetra
     { 0, 4, 3, 5, 2, -1 } // 14 = pro-STAR pyramid
};
*/

// * * * * * * * * * * * * * Private Member Functions * * * * * * * * * * * //

void Foam::h5Write::meshInit()
{
     Info<< "h5Write::meshInit" << endl << endl;

     // Set up file access property list with parallel I/O access
     meshPlistID_ = H5Pcreate(H5P_FILE_ACCESS);
     H5Pset_fapl_mpio(meshPlistID_, MPI_COMM_WORLD, MPI_INFO_NULL);

     // Create a new file collectively.
     meshFileID_ = H5Fcreate
       (
           H5DATA_MESH_FILE_NAME,
           H5F_ACC_TRUNC,
           H5P_DEFAULT,
           meshPlistID_
       );

     // Close the property list
     H5Pclose(meshPlistID_);

}

void Foam::h5Write::meshClose()
{
     Info<< "h5Write::meshClose" << endl << endl;

     // Close the file.
     H5Fclose(meshFileID_);

}

void Foam::h5Write::meshWrite()
{
     meshWritePoints();
     meshWriteCells();
}

void Foam::h5Write::meshWritePoints()
{
     const pointField& points = mesh_.points();

     // Create the dataspace for the dataset
     hsize_t dimsf[] = {points.size(), 3};
     hid_t fileSpace = H5Screate_simple(2, dimsf, NULL);

     // Construct dataset name
     char datasetName[80];
     sprintf(datasetName, "/%f/points/processor%i", mesh_.time().value(), Pstream::myProcNo());

     //if (debug)
     {
         Pout<< "Dataset name: "<< datasetName << endl;
     }

     // Create the dataset with default properties
     hid_t dsetID = H5Dcreate2
         (
             meshFileID_,
             datasetName,
             H5T_SCALAR,
             fileSpace,
             H5P_DEFAULT,
             H5P_DEFAULT
         );

     // Create property list for collective dataset write.
     meshPlistID_ = H5Pcreate(H5P_DATASET_XFER);
                    H5Pset_dxpl_mpio(meshPlistID_, H5FD_MPIO_COLLECTIVE);

     //if (debug)
     {
         Pout<< "Writing " << points.size() << " points" << endl;
     }

     // Create a simple array of points (to pass on to H5Dwrite)
     scalar pointList[points.size()][3];

     // Fill the array
     forAll(points, ptI)
     {
         pointList[ptI][0] = points[ptI].x();
         pointList[ptI][1] = points[ptI].y();
         pointList[ptI][2] = points[ptI].z();
     }

     // Do the actual write
     meshStatus_ = H5Dwrite
         (
             dsetID,
             H5T_SCALAR,
             H5S_ALL,
            meshPlistID_,
            pointList
        );

     // Close/release resources.
     H5Dclose(dsetID);
     H5Sclose(fileSpace);
     H5Pclose(meshPlistID_);
}

void Foam::h5Write::meshWriteCells()
{

}

// ************************************************************************* //

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

Or pass a link creation property list to H5Dcreate2 with
H5Pset_create_intermediate_group

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetCreateIntermedia
teGroup

to create missing intermediate groups as a side-effect.

G.

On Behalf Of Håkon Strandenes

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
Sent: Thursday, September 13, 2012 10:31 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Parallel dataset creation: unable to create and
link to dataset

Tank you very much. I needed of course to create the groups before the
dataset.

Regards

On 11. sep. 2012 21:17, Rob Latham wrote:

On Mon, Sep 10, 2012 at 05:20:50PM +0200, Håkon Strandenes wrote:

and similar for other ranks. The source code is attached. When
meshInit();

meshInit sets the "use parallel I/O" property and opens the already
existing file

meshWrite();

which in turn calls meshWritePoints and meshWriteCells I guess this is
the problematic file.

You've opened the file H5_ACC_TRUNC, so that zaps everything in the
dataset. Then you create a dataset "/%f/points/processor%i".

I think you first need to create the '%f' and 'points' groups before
making processor%i, right? Or will HDF5 make parent groups in this
situation? nearly 10 years on and I still don't use the hierarchy
features of HDF5..

==rob

meshClose();
is called, the error occures (in H5DCreate2).

The following is in the h5Write.H header file:
#define H5T_SCALAR H5T_NATIVE_DOUBLE
(to include the possibility to compile with support for single

precision).

The system is Ubuntu 12.04 with system OpenMPI and HDF5-OpenMPI
libraries installed. Compiler is GCC. I have tried to compile the
same code on another system with SLES 11sp1, Intel compilers and
SGI-MPI. Same error message occurs on that system.

Do you have any hints to help me? Thanks in advance.

Best regards,
Håkon Strandenes
Norway

/*--------------------------------------------------------------------------
-*\

   ========= |
   \\ / F ield | OpenFOAM: The Open Source CFD Toolbox
    \\ / O peration |
     \\ / A nd | Copyright (C) 2011 OpenFOAM Foundation
      \\/ M anipulation |
---------------------------------------------------------------------
----------
License
     This file is part of OpenFOAM.

     OpenFOAM is free software: you can redistribute it and/or modify it
     under the terms of the GNU General Public License as published by
     the Free Software Foundation, either version 3 of the License, or
     (at your option) any later version.

     OpenFOAM is distributed in the hope that it will be useful, but

WITHOUT

     ANY WARRANTY; without even the implied warranty of MERCHANTABILITY

or

     FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public

License

     for more details.

     You should have received a copy of the GNU General Public License
     along with OpenFOAM. If not, see <http://www.gnu.org/licenses/>.

\*-------------------------------------------------------------------
--------*/

#include "h5Write.H"
#include "Time.H"
#include "SortableList.H"
#include "OFstream.H"
#include "string.H"
#include "stringOps.H"

#define H5DATA_MESH_FILE_NAME "h5Data/h5DataMesh.h5"

// * * * * * * * * * * * * * * Static Data Members * * * * * * * * *
* * * * //

/*
const Foam::label Foam::h5Write::foamToXDMFFaceAddr[4][6] = {
     { 4, 5, 2, 3, 0, 1 }, // 11 = pro-STAR hex
     { 0, 1, 4, 5, 2, -1 }, // 12 = pro-STAR prism
     { 5, 4, 2, 0, -1, -1 }, // 13 = pro-STAR tetra
     { 0, 4, 3, 5, 2, -1 } // 14 = pro-STAR pyramid
};
*/

// * * * * * * * * * * * * * Private Member Functions * * * * * * *
* * * * //

void Foam::h5Write::meshInit()
{
     Info<< "h5Write::meshInit" << endl << endl;

     // Set up file access property list with parallel I/O access
     meshPlistID_ = H5Pcreate(H5P_FILE_ACCESS);
     H5Pset_fapl_mpio(meshPlistID_, MPI_COMM_WORLD, MPI_INFO_NULL);

     // Create a new file collectively.
     meshFileID_ = H5Fcreate
       (
           H5DATA_MESH_FILE_NAME,
           H5F_ACC_TRUNC,
           H5P_DEFAULT,
           meshPlistID_
       );

     // Close the property list
     H5Pclose(meshPlistID_);

}

void Foam::h5Write::meshClose()
{
     Info<< "h5Write::meshClose" << endl << endl;

     // Close the file.
     H5Fclose(meshFileID_);

}

void Foam::h5Write::meshWrite()
{
     meshWritePoints();
     meshWriteCells();
}

void Foam::h5Write::meshWritePoints() {
     const pointField& points = mesh_.points();

     // Create the dataspace for the dataset
     hsize_t dimsf[] = {points.size(), 3};
     hid_t fileSpace = H5Screate_simple(2, dimsf, NULL);

     // Construct dataset name
     char datasetName[80];
     sprintf(datasetName, "/%f/points/processor%i",
mesh_.time().value(), Pstream::myProcNo());

     //if (debug)
     {
         Pout<< "Dataset name: "<< datasetName << endl;
     }

     // Create the dataset with default properties
     hid_t dsetID = H5Dcreate2
         (
             meshFileID_,
             datasetName,
             H5T_SCALAR,
             fileSpace,
             H5P_DEFAULT,
             H5P_DEFAULT,
             H5P_DEFAULT
         );

     // Create property list for collective dataset write.
     meshPlistID_ = H5Pcreate(H5P_DATASET_XFER);
                    H5Pset_dxpl_mpio(meshPlistID_,
H5FD_MPIO_COLLECTIVE);

     //if (debug)
     {
         Pout<< "Writing " << points.size() << " points" << endl;
     }

     // Create a simple array of points (to pass on to H5Dwrite)
     scalar pointList[points.size()][3];

     // Fill the array
     forAll(points, ptI)
     {
         pointList[ptI][0] = points[ptI].x();
         pointList[ptI][1] = points[ptI].y();
         pointList[ptI][2] = points[ptI].z();
     }

     // Do the actual write
     meshStatus_ = H5Dwrite
         (
             dsetID,
             H5T_SCALAR,
             H5S_ALL,
             H5S_ALL,
            meshPlistID_,
            pointList
        );

     // Close/release resources.
     H5Dclose(dsetID);
     H5Sclose(fileSpace);
     H5Pclose(meshPlistID_);
}

void Foam::h5Write::meshWriteCells()
{

}

//
*********************************************************************
**** //

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org