virtual dataset with hyperslab parameter greater than 32 bit

vwilms · April 13, 2017, 7:40pm

Hi,

currently I am trying to create a virtual dataset for our measurement data. At the moment the highest sample rate of the measurement system is 100 Hz and I would like to collect the daily files into one virtual dataset which starts on 1st January, 2000 and end on 1st January, 2030. So this is a total dataset length of 86400 s/day * 100 samples/s * 10958 days = 94677120000 which in turn doesn't fit into a 32 bit number.

I use C# in combination with the HDF.PInvoke Nuget package. The parameters of H5S.select_hyperslab (needed to create a virtual dataset) are actually of type uint64 so at this point there is no problem. But when I create that virtual dataset and analyze it later with h5dump, the start, stride, count and block parameters of the hyperslab are truncated to 32 bit values.

In the C source code (hdf5-1.10.0-patch1) I identified three functions that are related to that problem:

  * H5S_hyper_serial_size (to calculate the size that is needed by an hyperslab when it is serialized to a file)
  * H5S_hyper_serialize (to serialize the hyperslab structure to a file)
  * H5S_hyper_deserialize (to serialize the hyperslab structure from a file)

Each function checks if there are any unlimited dimensions in the hyperslab. If true, then the hyperslab parameters are serialized as 64 bit numbers. If false, a 32 bit macro is used. I can see the same behavior also when I inspect the file with an hex-editor and locate the start/stride/count/block parameters (in the 32 bit version I could only find the truncated start parameter).

So my question is: Is this a bug or is there a way to enforce the use of the 64-bit macro without modifying the source code?

I made a short C# example, which should be easily adaptable:

H5S.select_hyperslab(spaceId_VDS, H5S.seloper_t.SET, new ulong[] { 90000000000 }, new ulong[] { 1 }, new ulong[] { 1 }, new ulong[] { 10 });
H5P.set_virtual(dcpl, spaceId_VDS, @"C:\Test.h5", "test_dataset", spaceId_source);

var datasetId = H5D.create(fileId, "Test", H5T.NATIVE_INT, spaceId_VDS, H5P.DEFAULT, dcpl, H5P.DEFAULT);

           H5S.close(spaceId_VDS);
           H5S.close(spaceId_source);
           H5D.close(datasetId);
           H5F.close(fileId);

If I then analyze that file with h5dump -p --header "C:\Users\Vincent\Desktop\HDF\VDS.h5", I get the following:

HDF5 "C:\Users\Vincent\Desktop\HDF\file.h5" {
...
VIRTUAL {
               SELECTION REGULAR_HYPERSLAB {
                  START (4100654080)
                  STRIDE (1)
                  COUNT (1)
                  BLOCK (10)
               }
            }
            SOURCE {
               FILE "C:\Test.h5"
               DATASET "test_dataset"
               SELECTION ALL
            }

...
}

so the "start" parameter is:

410065408010

···

=

0x00F46B0400

instead it should be:

9000000000010

=

0x14F46B0400

Hopefully someone can guide me into the right direction.

Thank you very much
Vincent

epourmal · April 13, 2017, 10:04pm

Vincent,

This is a known (to us issue. We will be addressing it in our next release.

Elena

···

On Apr 13, 2017, at 2:40 PM, Vincent Wilms <vwilms@outlook.com<mailto:vwilms@outlook.com>> wrote:

Hi,

currently I am trying to create a virtual dataset for our measurement data. At the moment the highest sample rate of the measurement system is 100 Hz and I would like to collect the daily files into one virtual dataset which starts on 1st January, 2000 and end on 1st January, 2030. So this is a total dataset length of 86400 s/day * 100 samples/s * 10958 days =94677120000 which in turn doesn‘t fit into a 32 bit number.