Writing 1D float array to hsds dataset depends on array length?

HSDS
1

Hi,

When I tried to write 1D float array to one dataset, “test” here, I found a very strange behavior. The python code using h5pyd package is listed below, with “f” as the hdf5 file object.

dset=f.create_dataset(“test”,data=np.linspace(0,100,N_ARRAY), dtype=np.float32)

  1. When N_ARRAY <=255, the readback dset is correct as a linear array.

  2. When N_ARRAY>256, the readback dset is just random and it’s different if I rewrite it. One example is attached below.

  3. Even strangely, when N_ARRAY is sufficiently large, like > 8000, the correct readback can be obtained excepted the very first four numbers, which are just random.

Does anyone know what can cause this issue?

Thanks,
Peiyun

One example at N_ARRAY=256,
[0.00e+00 1.40e-43 0.00e+00 0.00e+00 0.00e+00 0.00e+00 2.80e-45 0.00e+00
0.00e+00 1.26e-43 0.00e+00 0.00e+00 0.00e+00 0.00e+00 2.80e-45 0.00e+00
0.00e+00 1.40e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.10e-44 0.00e+00
0.00e+00 1.40e-43 2.80e-45 0.00e+00 0.00e+00 0.00e+00 2.10e-44 0.00e+00
0.00e+00 1.51e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.10e-44 0.00e+00
0.00e+00 1.26e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.10e-44 0.00e+00
0.00e+00 1.40e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.38e-44 0.00e+00
0.00e+00 1.40e-43 2.80e-45 0.00e+00 0.00e+00 0.00e+00 2.38e-44 0.00e+00
0.00e+00 1.51e-43 2.80e-45 0.00e+00 0.00e+00 0.00e+00 2.38e-44 0.00e+00
0.00e+00 1.26e-43 2.80e-45 0.00e+00 0.00e+00 0.00e+00 2.38e-44 0.00e+00
0.00e+00 1.40e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.52e-44 0.00e+00
0.00e+00 1.40e-43 2.80e-45 0.00e+00 0.00e+00 0.00e+00 2.52e-44 0.00e+00
0.00e+00 1.51e-43 4.20e-45 0.00e+00 0.00e+00 0.00e+00 2.52e-44 0.00e+00
0.00e+00 1.26e-43 4.20e-45 0.00e+00 0.00e+00 0.00e+00 2.52e-44 0.00e+00
0.00e+00 1.40e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.66e-44 0.00e+00
0.00e+00 1.40e-43 4.20e-45 0.00e+00 0.00e+00 0.00e+00 2.66e-44 0.00e+00
0.00e+00 1.51e-43 5.61e-45 0.00e+00 0.00e+00 0.00e+00 2.66e-44 0.00e+00
0.00e+00 1.53e-43 7.01e-45 0.00e+00 0.00e+00 0.00e+00 2.66e-44 0.00e+00
0.00e+00 1.26e-43 7.01e-45 0.00e+00 0.00e+00 0.00e+00 2.66e-44 0.00e+00
0.00e+00 1.40e-45 0.00e+00 0.00e+00 0.00e+00 0.00e+00 2.66e-44 0.00e+00
0.00e+00 1.40e-43 1.40e-45 0.00e+00 0.00e+00 0.00e+00 2.80e-44 0.00e+00
0.00e+00 1.40e-43 2.80e-45 0.00e+00 0.00e+00 0.00e+00 2.80e-44 0.00e+00
0.00e+00 1.51e-43 8.41e-45 0.00e+00 0.00e+00 0.00e+00 2.80e-44 0.00e+00
0.00e+00 1.26e-43 9.81e-45 0.00e+00 0.00e+00 0.00e+00 2.80e-44 0.00e+00
0.00e+00 1.40e-43 5.61e-45 0.00e+00 0.00e+00 0.00e+00 3.36e-44 0.00e+00
0.00e+00 1.26e-43 1.12e-44 0.00e+00 0.00e+00 0.00e+00 3.36e-44 0.00e+00
0.00e+00 1.40e-43 7.01e-45 0.00e+00 0.00e+00 0.00e+00 3.50e-44 0.00e+00
0.00e+00 1.26e-43 1.12e-44 0.00e+00 0.00e+00 0.00e+00 3.50e-44 0.00e+00
0.00e+00 1.40e-43 8.41e-45 0.00e+00 0.00e+00 0.00e+00 3.64e-44 0.00e+00
0.00e+00 1.26e-43 1.26e-44 0.00e+00 0.00e+00 0.00e+00 3.64e-44 0.00e+00
0.00e+00 1.40e-43 9.81e-45 0.00e+00 0.00e+00 0.00e+00 3.78e-44 0.00e+00
0.00e+00 1.26e-43 1.40e-44 0.00e+00 0.00e+00 0.00e+00 3.78e-44 0.00e+00]

2

Hi,
I should mention for future reference there’s a forum specifically for HSDS, but the hdf5 forum is fine as well.

I tried to re-create the error in HDFLab, but it looks ok to me. See: https://gist.github.com/jreadey/c7628b6fa4636958e6e59741626f94e0.

Am I missing something? Which version of h5pyd & HSDS are you using? (you can get this with hsinfo)

3

Hi,

Thanks for your response! The hsds version is 0.7.1.

The only difference I noticed is that I am doing this write in remotely with endpoint. I am still have no idea what causes this. But I figured out one work around - by using table, I can store numerical arrays of length several thousands correctly.

4

Glad to see you found a work-around.
The latest HSDS version is 0.7.3. When you get a chance, give that a try and let us know if you are still having the issue or not.