Reading Data Event-By-Event in C

HDF5 Library
1

A brief summary about the dataset : 4 x 3 matrix where row indicates object
index and column is co-ordinate index of that object (i.e. x = 0, y = 1 and z =
2). The file stores an event ensemble of this 4-object entity i.e. dataset is of
shape 100 x 4 x 3 for a total of 100 events. For example, first 4 events are
printed using h5dump (+ the dataset information)

$ h5dump -p -d "Position" -s 0 -c 4,4,3 Data.h5

HDF5 "Data.h5" {
DATASET "Position" {
   DATATYPE  H5T_NATIVE_FLOAT
   DATASPACE  SIMPLE { ( 100, 4, 3 ) / ( 100, 4, 3 ) }
   STORAGE_LAYOUT {
      CHUNKED ( 10, 4, 3 )
      SIZE 2625 (1.078:1 COMPRESSION)
   }
   FILTERS {
      COMPRESSION DEFLATE { LEVEL 4 }
   }
   FILLVALUE {
      FILL_TIME H5D_FILL_TIME_IFSET
      VALUE  H5D_FILL_VALUE_DEFAULT
   }
   ALLOCATION_TIME {
      H5D_ALLOC_TIME_INCR
   }
   SUBSET {
      START ( 0, 0, 0 );
      STRIDE ( 1, 1, 1 );
      COUNT ( 4, 4, 3 );
      BLOCK ( 1, 1, 1 );
      DATA {
      (0,0,0): -2.0039, -1.47425, 0.505329,
      (0,1,0): 1.45148, -0.369972, 2.3359,
      (0,2,0): 1.09121, -0.449007, -0.671806,
      (0,3,0): 0.804375, -0.685877, -3.2945,
      (1,0,0): -0.911479, 1.13416, -1.58155,
      (1,1,0): -0.804118, 1.98438, 0.407231,
      (1,2,0): -2.96317, 2.73922, -1.64719,
      (1,3,0): 4.53443, 3.50225, 0.866857,
      (2,0,0): -0.615941, -4.48291, -0.828907,
      (2,1,0): -1.9841, -1.47157, 1.81817,
      (2,2,0): -2.7926, -1.35195, 0.820239,
      (2,3,0): 1.26252, -1.09291, -1.09834,
      (3,0,0): -0.386284, -1.10739, -0.524769,
      (3,1,0): 2.4723, 0.112418, 3.04439,
      (3,2,0): 1.42348, 0.894244, 0.846616,
      (3,3,0): -0.578185, 2.11916, -0.997675
      }
   }
}
}

Now, I am trying to read this data in an event-by-event manner using a for loop

int main(){

   // Variables

   hsize_t NEvent     = 0;
   hsize_t DataDim[3] = {1, 4, 3};
   hsize_t Index[3]   = {0, 0, 0};
   float  Data[4][3];

   // ------------------------------------------------------ Input

   hid_t File       = H5Fopen("Data.h5", H5F_ACC_RDONLY, H5P_DEFAULT); 
   hid_t DataSet    = H5Dopen(File, "Position", H5P_DEFAULT);
   hid_t DataType   = H5Dget_type(DataSet);
   hid_t DataSpace  = H5Dget_space(DataSet);
   hid_t EventSpace = H5Screate_simple(3, DataDim, NULL);
   H5Sget_simple_extent_dims(DataSpace, &NEvent, NULL);

   // ------------------------------------------------------ Load

   for(Index[0] = 0; Index[0] < NEvent; Index[0]++){
      H5Sselect_hyperslab(DataSpace, H5S_SELECT_SET, Index, NULL, DataDim, NULL);
      H5Dread(DataSet, DataType, EventSpace, DataSpace, H5P_DEFAULT, Data);
   }

   H5Dclose(DataSet);
   H5Fclose(File);
  
   return 0;
}

And I get the following error

HDF5-DIAG: Error detected in HDF5 (1.10.8) thread 1:
  #000: ../../../src/H5Dio.c line 173 in H5Dread(): could not get a validated dataspace from file_space_id
    major: Invalid arguments to routine
    minor: Bad value
  #001: ../../../src/H5S.c line 243 in H5S_get_validated_dataspace(): selection + offset not within extent
    major: Dataspace
    minor: Out of range

In case of h5dump, each entry can be printed in the following way

$ h5dump -d "Position" -s i,0,0 -c 1,4,3 Data.h5

where i represents the i + 1 th entry or event. As per my understanding from
the h5dump output, this is what the C code should look like but certainly that is
not the case.

What am I doing wrong actually ? Any help will be appreciated. Thanks.

2

Hi @santanu300498,

Working with HDF5 hyperslabs in C can be challenging. That said, if you are not committed to a specific HDF5 API, you may want to try out HDFql. This is a high-level declarative language that relieves users from HDF5 low-level details such as hyperslabs.

Using HDFql, you can solve your use-case in C as shown below:

// declare variables
float data[4][3];
char script[100];
int i;

// use (i.e. open) file 'Data.h5'
hdfql_execute("USE FILE Data.h5");

// register variable 'Data' for subsequent use (by HDFql)
hdfql_variable_register(&data);

for(i = 0; i < 100; i++)
{
    // prepare script to select (i.e. read) dataset 'Position' using an hyperslab and populate variable 'Data' with it
    sprintf(script, "SELECT FROM Position[%d:::1] INTO MEMORY 0", i);

    // execute script
    hdfql_execute(script);

    // process variable 'Data'
    (...)
}

// unregister variable 'Data'
hdfql_variable_unregister(&data);

// close file
hdfql_execute("CLOSE FILE");

Hope it helps!

3

Hi @santanu300498,

@contact just beat me to it! If you still wish to use just C, the line

H5Sget_simple_extent_dims(DataSpace, &NEvent, NULL);

looks a bit suspect in that your dataspace is 3-dimensional, so the function will expect NEvent to be an array of 3 elements and so is likely overwriting some memory somewhere. While Index[0] is probably correct, I’m going to guess some part of DataDim is being overwritten and so your calls to H5Sselect_hyperslab would be incorrect, with the given offset + count value not falling within the defined dimensions for the dataset.