Broken file, can not add attribute

dave.allured · January 27, 2017, 12:48am

HDF5 support,

We traced this problem to a known bug, JIRA DB HDFFV-9281, "Attribute
creation index can't be incremented", previously reported by Andrey
Paramonov:

https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015-April/008462.html

I believe the root cause is the management of the per-variable maximum
attribute creation index within HDF5. This is a 16-bit counter with
maximum value 65535. Careless usage patterns within client applications
can easily saturate the counter. In particular, repeated delete/recreate
cycles on the same attribute name will ramp up the counter. That is what
happened in our own use case below.

Here is a related netcdf-C Github issue with excellent analysis by
Constantine Khroulev:

github.com/Unidata/netcdf-c

NetCDF-4 limits the number of times an attribute can be modified

opened 08:16PM - 23 Jan 17 UTC

closed 12:22AM - 16 Nov 19 UTC

ckhroulev

status/under review

# Summary Attributes of a variable or a group in a NetCDF-4 file cannot be mo…dified more than 2^16 times. (This includes special attributes such as `_NCProperties`.) This refers to the **total number** of modifications of **all** attributes of a variable. Here - Creating an attribute counts as a modification. - A "modification" is the number of `nc_put_att_...` calls followed by synchronizing the dataset to disk. For example, three `nc_put_att_int(...)` calls followed by one `nc_sync(...)` is **one** modification. (`nc_enddef` and `nc_close` calls also cause synchronization and have the same effect.) In particular: - a variable may not have more than 2^16 attributes (including ones used by the NetCDF library), - if a variable has 2^16 attributes then their values cannot be changed (i.e. this will cause a "crash"). This issue was first reported by [Heiko Klein][7] in March of 2015. It would be great if this behavior could be changed to match the NetCDF-3 library. If this is not feasible please consider documenting it so that it does not surprise others the way it surprised me. Thanks! # Details NetCDF-4 enables attribute creation order tracking using the HDF5 call `H5Pset_attr_creation_order` (see [the NetCDF manual][1]). With creation order tracking turned "on" the HDF5 library limits the number of times an attribute can be **created**. This HDF5 "feature" is present in both branches (1.8.x and 1.10) and is not documented. See the code for [the place where the limit is enforced][2] and [the definition of the limit][3]. In NetCDF-4 attribute modification is performed in three steps: 1. If an attribute exists, [delete it][4]. 2. [Re-create][5] an attribute. 3. [Write][6] the new value. So, each attribute modification increments the attribute order creation index and the program crashes once it reaches 65535. # Why does this matter? 1. This is a change in behavior compared to NetCDF-3. 2. This behavior is not documented. 3. It is important to be able to update metadata. 4. The number of attribute modifications (the "attribute creation order index") is saved in the file. This means that a user may acquire a file that has metadata that cannot be updated. (This can happen even if this user's workflow is perfectly normal.) It is easy to modify the code below to produce a file like that. # Reproducing the issue This code mimics the use case in which a file is modified using several tools, showing that the attribute creation order index is stored in a file. ``` c #include <stdio.h> /* printf */ #include <stdlib.h> /* exit */ #include <netcdf.h> static const char *filename = "netcdf_bug_report.nc"; static void check(const char *func, int ierr, int i) { if (ierr != NC_NOERR) { printf("NetCDF error: %s failed with %s at iteration %d\n", func, nc_strerror(ierr), i); exit(ierr); } } int main(int argc, char **argv) { int ierr = NC_NOERR, ncid = 0, i = 0; /* Create a NetCDF-4 file. */ ierr = nc_create(filename, NC_NETCDF4, &ncid); check("nc_create", ierr, i); ierr = nc_close(ncid); check("nc_close", ierr, i); /* Modify an attribute 2^16 times. */ for (i = 0; i < 65536; ++i) { ierr = nc_open(filename, NC_WRITE, &ncid); check("nc_open", ierr, i); ierr = nc_put_att_int(ncid, NC_GLOBAL, "attribute", NC_INT, 1, &i); check("nc_put_att_int", ierr, i); ierr = nc_close(ncid); check("nc_close", ierr, i); } return 0; } ``` ## Expected result This code should create a NetCDF-4 file with one global attribute with the value of 65535. ## Observed result ``` ~/netcdf4-bug $ ./bug_report_1 HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0: #000: H5Adeprec.c line 190 in H5Acreate1(): unable to create attribute major: Attribute minor: Unable to initialize object #001: H5Aint.c line 280 in H5A_create(): unable to create attribute in object header major: Attribute minor: Unable to insert object #002: H5Oattribute.c line 310 in H5O_attr_create(): attribute creation index can't be incremented major: Attribute minor: Unable to increment reference count NetCDF error: nc_close failed with NetCDF: Can't open HDF5 attribute at iteration 65534 ``` ## Alternative test code This code avoids closing and re-opening the file. (This is significantly faster.) ``` c #include <stdio.h> /* printf */ #include <stdlib.h> /* exit */ #include <netcdf.h> static const char *filename = "netcdf_bug_report.nc"; static void check(const char *func, int ierr, int i) { if (ierr != NC_NOERR) { printf("NetCDF error: %s failed with %s at iteration %d\n", func, nc_strerror(ierr), i); exit(ierr); } } int main(int argc, char **argv) { int ierr = NC_NOERR, ncid = 0, i = 0; /* Create a NetCDF-4 file. */ ierr = nc_create(filename, NC_NETCDF4, &ncid); check("nc_create", ierr, i); ierr = nc_enddef(ncid); check("nc_enddef", ierr, i); /* Modify an attribute 2^16 times. */ for (i = 0; i < 65536; ++i) { ierr = nc_redef(ncid); check("nc_redef", ierr, i); ierr = nc_put_att_int(ncid, NC_GLOBAL, "attribute", NC_INT, 1, &i); check("nc_put_att_int", ierr, i); ierr = nc_enddef(ncid); check("nc_enddef", ierr, i); } ierr = nc_close(ncid); check("nc_close", ierr, i); return 0; } ``` ### Output ``` HDF5-DIAG: Error detected in HDF5 (1.8.18) thread 0: #000: H5Adeprec.c line 190 in H5Acreate1(): unable to create attribute major: Attribute minor: Unable to initialize object #001: H5Aint.c line 280 in H5A_create(): unable to create attribute in object header major: Attribute minor: Unable to insert object #002: H5Oattribute.c line 310 in H5O_attr_create(): attribute creation index can't be incremented major: Attribute minor: Unable to increment reference count NetCDF error: nc_enddef failed with NetCDF: Can't open HDF5 attribute at iteration 65534 ``` # Environment Information * What platform are you using? (please provide specific distribution/version in summary) * [X] Linux * [ ] Windows * [X] OSX * [ ] Other * [ ] NA * 32 and/or 64 bit? * [ ] 32-bit * [X] 64-bit * What build system are you using? * [X] autotools (`configure`) * [ ] cmake * Can you provide a sample netCDF file or `C` code to recreate the issue? * [X] Yes (please attach to this issue, thank you!) * [ ] No * [ ] Not at this time [1]: http://www.unidata.ucar.edu/software/netcdf/docs/file_format_specifications.html#creation_order [2]: https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse/src/H5Oattribute.c?at=refs%2Fheads%2Fhdf5_1_8#310 [3]: https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse/src/H5Opkg.h?at=refs%2Fheads%2Fhdf5_1_8#35 [4]: https://github.com/Unidata/netcdf-c/blob/v4.4.1.1/libsrc4/nc4hdf.c#L1323 [5]: https://github.com/Unidata/netcdf-c/blob/v4.4.1.1/libsrc4/nc4hdf.c#L1395 [6]: https://github.com/Unidata/netcdf-c/blob/v4.4.1.1/libsrc4/nc4hdf.c#L1400 [7]: http://netcdf-group.1586084.n2.nabble.com/HDF5-error-after-rewriting-variable-attribute-for-65533-times-tt7575745.html

I think you can close the newer JIRA ticket HDFFV-10026 and mark it
duplicate, or merge it with the previous JIRA DB HDFFV-9281.

We look forward to some kind of fix at the NetCDF or HDF5 level. Thank you
for looking into this.

--Dave A.
NOAA/OAR/ESRL/PSD/CIRES

···

On Tue, Nov 15, 2016 at 12:10 PM, Dave Allured - NOAA Affiliate < dave.allured@noaa.gov> wrote:

Elena,

Thank you for reproducing the issue, and for showing me how to display the
object structure. h5debug versions 1.8.16 and 1.10.0 are both showing
ridiculous values for number of attributes, for known good Netcdf-4 files,
even trivial examples. There might be a problem in h5debug, independent
from this broken file problem.

--Dave

On Mon, Nov 14, 2016 at 10:54 PM, Elena Pourmal <epourmal@hdfgroup.org> > wrote:

Dave,

I reproduced the issue with h5py and entered HDFFV-10026
<https://jira.hdfgroup.org/browse/HDFFV-10026> (for your reference) into
our JIRA database.

h5py reports: Unable to create attribute (Attribute creation index can't
be incremented)

h5debug tool reports suspicious number of attributes for that dataset
See below. We will need to investigate.

Thank you for your report!

Elena

[volga:~] epourmal% h5debug precip.V1.0.2016.nc 2482
Reading signature at address 2482 (rel)
Object Header...
Dirty: FALSE
Version: 2
Header size (in bytes): 12
Number of links: 1
Attribute creation order tracked: Yes
Attribute creation order indexed: Yes
Attribute storage phase change values: Default
Timestamps: Disabled
Number of messages (allocated): 7 (8)
Number of chunks (allocated): 1 (2)
Chunk 0...
   Address: 2482
   Size in bytes: 263
   Gap: 0
Message 0...
   Message ID (sequence number): 0x0001 `dataspace' (0)
   Dirty: FALSE
   Message flags: <none>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (14, 52) bytes
   Message Information:
      Rank: 3
      Dim Size: {310, 120, 300}
      Dim Max: {UNLIM, 120, 300}
Message 1...
   Message ID (sequence number): 0x0005 `fill_new' (0)
   Dirty: FALSE
   Message flags: <C>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (72, 10) bytes
   Message Information:
      Space Allocation Time: Incremental
      Fill Time: If Set
      Fill Value Defined: User Defined
      Size: 4
      Data type: <dataset type>
Message 2...
   Message ID (sequence number): 0x000b `filter
pipeline' (0)
   Dirty: FALSE
   Message flags: <C>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (88, 22) bytes
   Message Information:
      Number of filters: 2/2
      Filter at position 0
         Filter identification: 0x0002
         Filter name: NONE
         Flags: 0x0001
         Num CD values: 1
            CD value 0 4
      Filter at position 1
         Filter identification: 0x0001
         Filter name: NONE
         Flags: 0x0001
         Num CD values: 1
            CD value 0 2
Message 3...
   Message ID (sequence number): 0x0008 `layout' (0)
   Dirty: FALSE
   Message flags: <C>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (116, 27) bytes
   Message Information:
      Version: 3
      Type: Chunked
      Number of dimensions: 4
      Size: {1, 120, 300, 4}
      Index Type: v1 B-tree
      B-tree address: 17919
Message 4...
   Message ID (sequence number): 0x0015 `ainfo' (0)
   Dirty: FALSE
   Message flags: <DS>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (149, 28) bytes
   Message Information:
      Number of attributes: 18446744073709551615
      Track creation order of attributes: TRUE
      Index creation order of attributes: TRUE
      Max. creation index value: 65535
      'Dense' attribute storage fractal heap address: 4671
      'Dense' attribute storage name index v2 B-tree address: 4817
      'Dense' attribute storage creation order index v2 B-tree address:
4855
Message 5...
   Message ID (sequence number): 0x0003 `datatype' (0)
   Dirty: FALSE
   Message flags: <C>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (183, 20) bytes
   Message Information:
      Type class: floating-point
      Size: 4 bytes
      Version: 1
      Byte order: little endian
      Precision: 32 bits
      Offset: 0 bits
      Low pad type: zero
      High pad type: zero
      Internal pad type: zero
      Normalization: implied
      Sign bit location: 31
      Exponent location: 23
      Exponent bias: 0x0000007f
      Exponent size: 8
      Mantissa location: 0
      Mantissa size: 23
Message 6...
   Message ID (sequence number): 0x0000 `null' (0)
   Dirty: FALSE
   Message flags: <none>
   Chunk number: 0
   Raw message data (offset, size) in chunk: (209, 62) bytes
   Message Information:
      <No info for this message>

On Nov 14, 2016, at 4:10 PM, Dave Allured - NOAA Affiliate < >> dave.allured@noaa.gov> wrote:

HDF5 support,

My work group encounters rare failures when attempting to update
collections of HDF5 files. The usual symptom is that a previously valid
file becomes partially broken, such that it becomes impossible to add new
attributes to one particular data variable. The file otherwise reads and
writes normally. Attributes can be added to other variables, and data can
still be written to the suspect variable, including extending the unlimited
dimension.

We normally operate on these files with the Netcdf-4 interface to HDF5,
but I think I isolated the failure to pure HDF5 functionality. You can
examine one of these broken files here (9 Mb):

ftp://ftp.cdc.noaa.gov/Public/dallured/hdf5/precip.V1.0.2016.nc

Diagnostics such as h5check, h5debug, h5dump, and h5stat all report this
file as valid. h5edit demonstrates the problem:

> ls -go precip.V1.0.2016.nc <http://precip.v1.0.2016.nc/>
-rw-r--r-- 1 9024497 Nov 9 15:02 precip.V1.0.2016.nc
<http://precip.v1.0.2016.nc/>

> h5edit -c 'CREATE /precip/att12 { DATATYPE H5T_IEEE_F32LE DATASPACE
SIMPLE ( 2 ) DATA { 777.0, 778.0 } } ;' precip.V1.0.2016.nc
<http://precip.v1.0.2016.nc/>
failed to create attribute (null)
CREATE Attribute command failed

... and the data file is unchanged. It does not matter whether the added
attribute is scalar or array.

We are currently using the HDF5 library version 1.8.16 in combination
with Netcdf to read and write these data sets. The version of h5edit above
is 1.3.1. The broken file behaves the same way on both Mac and Linux
platforms, and with several Netcdf-based attribute writers in addition to
h5edit.

We are investigating, but we have not yet found a good way to isolate the
exact program or event that creates this broken file condition. We are
suspicious that NFS connected file servers may contribute to the problem,
but no strong evidence yet to back this up.

Today I have these questions:

1. Can someone identify exactly what is wrong with the above sample file?

2. Can h5check or another utility be updated to diagnose this condition?

3. Has anyone else experienced partially broken files with this symptom?

Thank you for any insights.

--Dave A.
NOAA/OAR/ESRL/PSD/CIRES

Attention! https://support.hdfgroup.org is the NEW home for documentation from The HDF Group. (Details)

Broken file, can not add attribute