# Getting error "invalid dataset size, likely file corruption" **URL:** \ **Category:** HDF5 Library\ **Created:** [April 25, 2024, 8:47pm UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246 "2024-04-25T20:47:42Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![apollo3zehn-h5](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/apollo3zehn-h5/32/1589_2.png) [@apollo3zehn-h5](https://forum.hdfgroup.org/u/apollo3zehn-h5)\ **Post date:** [April 25, 2024, 8:47pm UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/1 "2024-04-25T20:47:42Z")
Hi, I am developing the independent library PureHDF with write support for HDF5 files. I use h5dump to verify that the written files conform to the spec. With h5dump `1.14.1` I was able to create a file with a dataset with null dataspace and h5dump was able to dump it: ```auto HDF5 "" { GROUP "/" { DATASET "Null" { DATATYPE H5T_STD_I32LE DATASPACE NULL DATA { } } } } ``` I upgraded to version `1.14.4.2` and now I get the error message `invalid dataset size, likely file corruption`. I traced it back to this commit ([Fixes for file format security issues (#4283) · HDFGroup/hdf5@ce53bc0 · GitHub](https://github.com/HDFGroup/hdf5/commit/ce53bc020d37e10a06a59501c45f1ab82539f0c5)) where the following problematic check has been introduced (line 408): > ```c if (H5_addr_le((layout->storage.u.contig.addr + data_size), layout->storage.u.contig.addr)) HGOTO_ERROR(H5E_DATASET, H5E_OVERFLOW, FAIL, "invalid dataset size, likely file corruption"); ``` This check fails when the data size is = 0, which is the case when the data space is a null dataspace. Is the error on my side (maybe I understood something wrong), or is the newly introduced check the problem? Thanks! ---
**Author:** ![jhenderson](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/73ab20/32.png) [@jhenderson](https://forum.hdfgroup.org/u/jhenderson)\ **Post date:** [April 25, 2024, 9:34pm UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/2 "2024-04-25T21:34:28Z")
Hi @apollo3zehn-h5, do you happen to have that file available somewhere? We do test NULL dataspaces, but generally the library shouldn’t have allocated space in the file for a 0-sized contiguous dataset, so the check should normally be skipped. For example, here’s one of the test files we have that has a dataset with a NULL dataspace: ```auto h5dump -pH tnullspace.h5 GROUP "/" { ATTRIBUTE "attr" { DATATYPE H5T_STD_U32LE DATASPACE NULL } DATASET "dset" { DATATYPE H5T_STD_I32BE DATASPACE NULL STORAGE_LAYOUT { CONTIGUOUS SIZE 0 OFFSET HADDR_UNDEF } FILTERS { NONE } FILLVALUE { FILL_TIME H5D_FILL_TIME_IFSET VALUE H5D_FILL_VALUE_DEFAULT } ALLOCATION_TIME { H5D_ALLOC_TIME_LATE } } } } ``` Notice the `OFFSET HADDR_UNDEF` part, which means that file space isn’t allocated and the check is skipped. What does the output of `h5dump -pH` look like for your file? It’s possible there’s a bug either in older versions or the current version of HDF5 with regards to file space allocation for 0-sized datasets, but I also think the use of `H5_addr_le` may have been unintentional and `H5_addr_lt` may have been intended instead. ---
**Author:** ![apollo3zehn-h5](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/apollo3zehn-h5/32/1589_2.png) [@apollo3zehn-h5](https://forum.hdfgroup.org/u/apollo3zehn-h5)\ **Post date:** [April 26, 2024, 7:20am UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/3 "2024-04-26T07:20:13Z")
Thanks for your quick reply! I skip now file space allocation when there is a null dataspace and h5dump is happy now: ```auto HDF5 "/tmp/tmpDgTBco.tmp" { GROUP "/" { DATASET "Null" { DATATYPE H5T_STD_I32LE DATASPACE NULL STORAGE_LAYOUT { CONTIGUOUS SIZE 0 OFFSET HADDR_UNDEF } FILTERS { NONE } FILLVALUE { FILL_TIME H5D_FILL_TIME_NEVER VALUE H5D_FILL_VALUE_UNDEFINED } ALLOCATION_TIME { H5D_ALLOC_TIME_EARLY } } } } ``` If space allocation is generally not allowed in combination with a null dataspace, it would be great to add this information to the spec document ([HDF5: HDF5 File Format Specification Version 3.0](https://docs.hdfgroup.org/hdf5/develop/_f_m_t3.html)) so others will not run into the same issue. Thanks again! My issue is solved now 🙂 ---
**Author:** ![apollo3zehn-h5](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/apollo3zehn-h5/32/1589_2.png) [@apollo3zehn-h5](https://forum.hdfgroup.org/u/apollo3zehn-h5)\ **Post date:** [April 26, 2024, 7:25am UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/4 "2024-04-26T07:25:54Z")
To avoid confusion: I am not creating files using the C-library but instead using my independent implementation. I was simply following the HDF5 spec and ran into this issue. To avoid this for other library authors and to align the spec with the reality I would appreciate if the condition `null dataspace = no allocation allowed` would become part of that spec 🙂 ---
**Author:** ![jhenderson](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/73ab20/32.png) [@jhenderson](https://forum.hdfgroup.org/u/jhenderson)\ **Post date:** [April 26, 2024, 3:47pm UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/5 "2024-04-26T15:47:32Z")
> [@apollo3zehn-h5](#): > > If space allocation is generally not allowed in combination with a null dataspace, it would be great to add this information to the spec document ([HDF5: HDF5 File Format Specification Version 3.0](https://docs.hdfgroup.org/hdf5/develop/_f_m_t3.html)) so others will not run into the same issue. It’s an interesting question for sure. Considering the behavior of previous versions of the library, I’m tempted to say that there shouldn’t be any reason for this to not be allowed and that the library should be fixed. When you previously allocated file space for the dataset, was there a particular fixed address given to the dataset (since you would have been allocating a 0-byte region)? ---
**Author:** ![apollo3zehn-h5](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/apollo3zehn-h5/32/1589_2.png) [@apollo3zehn-h5](https://forum.hdfgroup.org/u/apollo3zehn-h5)\ **Post date:** [April 26, 2024, 4:40pm UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/6 "2024-04-26T16:40:07Z")
Yes, previously there was a specific address. It was produced by the simple `FreeSpaceManager` class: [PureHDF/src/PureHDF/VOL/Native/Core.Writing/FreeSpaceManager.cs at 2c300909a30048cea79cecd91180ca10679aac92 · Apollo3zehn/PureHDF · GitHub](https://github.com/Apollo3zehn/PureHDF/blob/2c300909a30048cea79cecd91180ca10679aac92/src/PureHDF/VOL/Native/Core.Writing/FreeSpaceManager.cs#L12) Previously it did not check for the length and always returned a valid address. ---
**Author:** ![jhenderson](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/73ab20/32.png) [@jhenderson](https://forum.hdfgroup.org/u/jhenderson)\ **Post date:** [April 26, 2024, 4:51pm UTC](https://forum.hdfgroup.org/t/getting-error-invalid-dataset-size-likely-file-corruption/12246/7 "2024-04-26T16:51:06Z")
Thanks for the pointer! It may have been that this was intentionally left unspecified in the file format specification, but we plan on discussing this internally to determine a resolution on it. In the meantime, I’ll fix the newly-added check in the library because I believe it’s an ill-formed check in either case.