Read 64-bit integer data to 32-bit integer buffer

HDF5 Library
1

Greetings!

I have a conceptual question involving “datasets (H5Ds)”, “Dataspaces (H5Ss)”,“hyperslab selections”, and datatypes (H5Ts).

Background:
The motivation for asking is that I am working with the CFD General Notation System (CGNS) API in C, which is implemented via HDF5. Some of the H5Ds in that format are routinely permitted to be 32-bit or 64-bit as per CGNS specifications. Because of the need-to-know aspect of the element size, my C program will have something like:

int32_t* Buff32;
int64_t* Buff64;
// assume apriori knowledge of array size
if(eSize == 4){//32-bit 
    Buff32 = (int32_t*)malloc(nElem*eSize);
    H5Dread(..., Buff32);
}else if(eSize == 8){//64-bit
    Buff64 = (int64_t*)malloc(nElem*eSize);
    H5Dread(...., Buff64);
}

I would prefer something like:

void* Buff;
Buff = malloc(nElem*eSize); 
*H5Dread(..., Buff)*

Why? Because the next step in my program is to effectuate data conversions via typecasting.

MyDatatype_t outBuff*;
outBuff = (MyDatatype_t*)malloc(nElem*sizeof(MyDatatype_t));
for(int idx = 0; idx < nElem; idx++){
    outBuff[idx] = (MyDatatype_t)Buff32[idx];
}

I expect the arrays to read in to be substantially large and would like to avoid the overhead of typecasting in C.

The Question

  • Data under file’s H5D is 64-bit, however, the actual numbers are small enough to fit in 32-bit.
  • I want to read data to buffer as 32-bit in spite of the file having a 64-bit H5D.
  • Say there are nElem such 64-bit numbers under the H5D on file. Can I then:
  1. Define a 32-bit H5T.
  2. Define an H5S for the file with 2*nELem points (because 64-bit = 32-bit*2).
  3. Define an H5S for the buffer as H5S_ALL or some other custom H5S.
  4. Make a hyperslab selection on the file H5S with a stride of 2 (i.e., 64-bit worth of memory), but block size of 1 (i.e., 32-bit worth of memory).
  5. OPTIONAL: Set an offset of 1 (32-bits worth of memory) in the hyperslab for the FIle H5S if data is Little-Endian.
  6. Call H5Dread(loc_id,H5T,fileH5S,buffH5S,dxpl_id,Buff) and read directly to a 32-bit buffer?

Picture this:

Data in the H5D:  [64-bit        ][64-bit        ][64-bit        ]....[64-bit        ]
How I see the H5D:[32-bit][32-bit][32-bit][32-bit][32-bit][32-bit]....[32-bit][32-bit]
Hyperslab (BE):   [  ^   ]        [  ^   ]        [  ^   ]        ....[  ^   ]
Hyperslab (LE):           [  ^   ]        [  ^   ]        [  ^   ]....        [  ^   ]
2

H5Dread includes automatic datatype conversion. It will do exactly what you want without any extra code for size conversion or endian swapping, and no second buffer. Just specify the 32-bit H5T_NATIVE_INT in the mem_type_id argument. This memory data type DOES NOT need to be the same as the data type in the file.

Please refer to the note about “Datatype conversion” near the bottom of the H5Dread documentation page. Also see the corresponding section in the user guide. If there is any chance of oversized integers, you might want to set up an exception callback function, as described.

3

Oops. The datatype conversion note is missing from the new documentation website! Please refer the the OLD doc page for H5Dread. This page also helpfully includes the correct link into the user guide.

https://portal.hdfgroup.org/display/HDF5/H5D_READ

1 Like
4

Thank you very much for clarifying!
Awesome, this will make the API I’m working on more compact.

Yeah, the documentation that I was consulting didn’t mention the automatic handling of type conversion and endianess swapping. I was consulting from: https://docs.hdfgroup.org/hdf5/develop/group___h5_d.html