Bug in HDF-1.8.14 - Recursive named data types [SEC=UNCLASSIFIED]

Mark_Hodson · March 17, 2015, 6:09am

UNCLASSIFIED
Hi all,

In our application's use of HDF-5 we have a requirement for a slightly richer flavour of data type than HDF natively supports. We chose to make all the data types that we use for datasets in our HDF-5 files named (committed) data types, to which we attach a handful of metadata attributes.

However, this process causes a particular pattern of recursive use of named data types that in our application causes HDF-5 to crash. Consider that to create a dataset w/ a metadata-decorated data type for C++ type "std::vector<double>" we first create a decorated named data type representing the C++ "double" data type. This named HDF-5 data type is then used as the super-type in declaring a variable length HDF-5 array data type, thus representing the std::vector part. Hence, we have one named HDF-5 data type whose super-type is another named HDF-5 data type.

This appears to function correctly so long as you hold the file open; if you open the dataset and use H5Tget_type() to reveal the data type, H5Tcommitted() returns 1 as it should. Further, if you get the data type's super-type, it also reports H5Tcommitted() == 1. However, if you then close the file and re-open it and repeat the process, it seems to corrupt the slack list associated w/ data types. In our application we eventually dereference an invalid file pointer held inside one of the slack list nodes, and it crashes. Fortunately I have been able to reproduce the issue by creating a new test case (patch attached) though in this case the test fails (as H5Tcommitted() reports 0 for the super-type on re-opening of the file, not 1) and then the HDF library reset causes an infinite loop to be reported. The output is:

Testing non-aligned conversions (ALIGNMENT=1)....
Testing H5Tget_class() PASSED
Testing H5Tcopy() PASSED
Testing H5Tdetect_class() PASSED
Testing compound datatypes PASSED
Testing query functions of compound and enumeration types PASSED
Testing transient datatypes PASSED
Testing named datatypes PASSED
Testing functions of encoding and decoding datatypes PASSED
Testing encoding datatypes with the 'use the latest format' flag PASSED
Testing exceptions for int <-> float conversions PASSED
Testing indirectly reopening committed datatypes PASSED
Testing indirectly reopening recursively committed datatypes including file reopening*FAILED*
   at ..\..\hdf5-1.8.14\test\dtypes.c:6635 in test_named_indirect_reopen_file()...
Testing deleting objects that use named datatypes PASSED
Testing deleting objects that use named datatypes PASSED
Testing H5Tset/get_order for compound type PASSED
Testing string type creation using H5Tcreate PASSED
Testing deprected API routines for datatypes PASSED
HDF5: infinite loop closing library
      D,A,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing string conversions PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing random string conversion speed PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing some type functions for string PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound element reordering PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound subset conversions PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound element shrinking & reordering PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing optimized struct converter PASSED
Testing compound element growing PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing compound element insertion PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing packing compound datatypes PASSED
Testing compound datatype with VL string PASSED
Testing array datatype of compound type with VL string PASSED
Testing registering type conversion routine with compound conversions PASSED
Testing adjust size of compound datatypes PASSED
Testing compound datatypes of boundary size with latest format PASSED
Testing unaligned VL strings in compound PASSED
Testing compound subset conversion with extra space in source PASSED
Testing visibility of internally registered type ids PASSED
Testing that H5Tpack removes trailing bytes PASSED
Testing accessing objects with compound datatypes that have no fields PASSED
Testing random enum conversion O(N) PASSED
Testing random enum conversion O(N log N) PASSED
HDF5: infinite loop closing library
      D,G,A,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,
FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing non-native enumeration type conversion PASSED
Testing bitfield conversions PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing some type functions for bitfield PASSED
HDF5: infinite loop closing library
      D,T,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Testing opaque datatypes PASSED
Testing H5Tset/get_order PASSED
Testing string conversion between ASCII and UTF PASSED
***** 1 FAILURE! *****
HDF5: infinite loop closing library
      D,S,T,F,FD,P,FD,P,FD,P,E,E,SL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,F
L,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL,FL
Press any key to continue . . .

The system is Microsoft Visual Studio 2008 on Windows 7 (32-bit build, 64-bit operating system).

Interestingly, if you comment out just the one offending call to H5Tcommitted() on the super-type after re-opening, the test passes and the library resets fine. This is strange as this method only does a H5SL_SEARCH on the slack list - which really shouldn't mutate any state (?). If you choose not to commit the super-type, and instead assert H5Tcommitted() == 0, that also works. Despite all this I suspect the content of the HDF-5 file is being setup incorrectly when one named data type refers to another, but I can't figure out where. It doesn't seem to require metadata attributes also attached to cause the failure.

Any ideas?

Mark

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

dtypes.c.patch (5.49 KB)

epourmal · March 18, 2015, 10:14pm

Mark,

Thank you for the patch that illustrates the problem! It really saves a lot of investigation time!

HDF5 doesn’t support nested committed datatypes. In your example, when committed VL string is used as a member of the compound datatype, it is treated as a regular datatype. You can easily see it by using h5ls on the created dtypes1.h5 file. In the output below, cmp_type and str_type are shared (committed), but “vlstr” member of the cpm_type type is shown as a regular datatype.

[epourmal@jam test]$ h5ls -v dtypes1.h5
Opened "dtypes1.h5" with sec2 driver.
cmp_dset Dataset {3/3}
    Location: 1:1272
    Links: 1
    Storage: 12 logical bytes, 0 allocated bytes
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
cmp_type Type
    Location: 1:1176
    Links: 2
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
str_type Type
    Location: 1:800
    Links: 1
    Type: shared-1:800 variable-length null-terminated ASCII string

When the second call to

if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR

is commented out, the infinite loop goes away.

Hopefully, you can fix your application since the member of the committed compound datatype cannot be a committed datatype (even it was created using a committed one), but we still have an issue in HDF5.

I confirmed that HDF5 1.8.* gives an infinite loop while HDF5 trunk is fine. It is not clear why the first H5committed call for the VL string type is not failing and why your test triggers an infinite loop.

For you information I entered JIRA issue HDFFV-9174.

Thanks again for your report!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 17, 2015, at 1:09 AM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

<dtypes.c.patch>

Mark_Hodson · March 19, 2015, 12:16am

UNCLASSIFIED
Thanks. It's unfortunate that HDF-5 does not support nested committed data types.

I may have discovered why the test is entering an infinite loop. It is actually a problem with the test. When the second call to
if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR
fails, it jumps to the end where it closes "strtype" and "cmptype" - both of which are already closed. If I set these handles to -1 when they are closed mid-way through the test, no infinite loop occurs. I suspect it is not valid to close a handle twice in HDF and so there is a logical error with the test code.

I will send through a new patch later on.

I guess I can accept that HDF-5 does not support nested committed data types, however the behaviour of H5Tcommitted() is clearly inconsistent depending on whether you have closed and reopened the file or not. The super-type currently says it *is* committed in the first part of the test. Would it be reasonable to restrict the bug report then to say that H5Tcommitted() should return 0 both before and after reopening the file? When designing our application it would have been useful if the HDF documentation stated this limitation. We may have designed it differently if we knew.

Can I request a feature enhancement issue also be created in JIRA also for "adding support for nested committed data types"? The use case is for something similar to H5view but where the views available for a data type (including the types of elements of an array and members of a compound) depend on the *name* of the data type (not the name of the field). So, an array of 3x floating-point values might have a committed name "position" and another otherwise identical data type might have committed name "velocity" - the viewer application would treat them differently. Our work-around is to use metadata attributes on the top-level committed data type and then separately and recursively open each committed super-type.

In terms of our application crash, my colleague has been debugging HDF while running in our application and may have found why it is crashing - and it is not this bug (which we thought it was). It is a different problem, and I'll hopefully submit a separate issue later on. We'll try and reproduce it in the HDF test harness in some detail before submitting.

PS. Do you allow public access to your JIRA on-demand instance? Or is it for internal use only?

Mark

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Thursday, 19 March 2015 8:45 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Bug in HDF-1.8.14 - Recursive named data types [SEC=UNCLASSIFIED]

Mark,

Thank you for the patch that illustrates the problem! It really saves a lot of investigation time!

HDF5 doesn't support nested committed datatypes. In your example, when committed VL string is used as a member of the compound datatype, it is treated as a regular datatype. You can easily see it by using h5ls on the created dtypes1.h5 file. In the output below, cmp_type and str_type are shared (committed), but "vlstr" member of the cpm_type type is shown as a regular datatype.

[epourmal@jam test]$ h5ls -v dtypes1.h5
Opened "dtypes1.h5" with sec2 driver.
cmp_dset Dataset {3/3}
    Location: 1:1272
    Links: 1
    Storage: 12 logical bytes, 0 allocated bytes
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
cmp_type Type
    Location: 1:1176
    Links: 2
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
str_type Type
    Location: 1:800
    Links: 1
    Type: shared-1:800 variable-length null-terminated ASCII string

When the second call to

if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR

is commented out, the infinite loop goes away.

Hopefully, you can fix your application since the member of the committed compound datatype cannot be a committed datatype (even it was created using a committed one), but we still have an issue in HDF5.

I confirmed that HDF5 1.8.* gives an infinite loop while HDF5 trunk is fine. It is not clear why the first H5committed call for the VL string type is not failing and why your test triggers an infinite loop.

For you information I entered JIRA issue HDFFV-9174.

Thanks again for your report!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 17, 2015, at 1:09 AM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

<dtypes.c.patch>

Mark_Hodson · March 19, 2015, 5:25am

UNCLASSIFIED
Elena et al,

Attached is a replacement patch for revised issue HDFFV-9174. This test does not double-close any handles, and does not cause an infinite loop. It fails due to the inconsistent behaviour of H5Tcommitted() before and after re-opening the test data file. I'm happy if you want to limit the scope of this issue to this relatively benign defect.

We're still trying to resolve our main issue.

Thanks for the prompt response.
Mark

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

dtypes.c.patch (5.83 KB)

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Thursday, 19 March 2015 8:45 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Bug in HDF-1.8.14 - Recursive named data types [SEC=UNCLASSIFIED]

Mark,

Thank you for the patch that illustrates the problem! It really saves a lot of investigation time!

HDF5 doesn't support nested committed datatypes. In your example, when committed VL string is used as a member of the compound datatype, it is treated as a regular datatype. You can easily see it by using h5ls on the created dtypes1.h5 file. In the output below, cmp_type and str_type are shared (committed), but "vlstr" member of the cpm_type type is shown as a regular datatype.

[epourmal@jam test]$ h5ls -v dtypes1.h5
Opened "dtypes1.h5" with sec2 driver.
cmp_dset Dataset {3/3}
    Location: 1:1272
    Links: 1
    Storage: 12 logical bytes, 0 allocated bytes
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
cmp_type Type
    Location: 1:1176
    Links: 2
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
str_type Type
    Location: 1:800
    Links: 1
    Type: shared-1:800 variable-length null-terminated ASCII string

When the second call to

if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR

is commented out, the infinite loop goes away.

Hopefully, you can fix your application since the member of the committed compound datatype cannot be a committed datatype (even it was created using a committed one), but we still have an issue in HDF5.

I confirmed that HDF5 1.8.* gives an infinite loop while HDF5 trunk is fine. It is not clear why the first H5committed call for the VL string type is not failing and why your test triggers an infinite loop.

For you information I entered JIRA issue HDFFV-9174.

Thanks again for your report!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 17, 2015, at 1:09 AM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

<dtypes.c.patch>

epourmal · March 19, 2015, 3:26pm

Thank you!

I added the new patch to the JIRA issue.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 19, 2015, at 12:25 AM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

UNCLASSIFIED

Elena et al,

Attached is a replacement patch for revised issue HDFFV-9174. This test does not double-close any handles, and does not cause an infinite loop. It fails due to the inconsistent behaviour of H5Tcommitted() before and after re-opening the test data file. I’m happy if you want to limit the scope of this issue to this relatively benign defect.

We’re still trying to resolve our main issue.

Thanks for the prompt response.
Mark

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Thursday, 19 March 2015 8:45 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Bug in HDF-1.8.14 - Recursive named data types [SEC=UNCLASSIFIED]

Mark,

Thank you for the patch that illustrates the problem! It really saves a lot of investigation time!

HDF5 doesn’t support nested committed datatypes. In your example, when committed VL string is used as a member of the compound datatype, it is treated as a regular datatype. You can easily see it by using h5ls on the created dtypes1.h5 file. In the output below, cmp_type and str_type are shared (committed), but “vlstr” member of the cpm_type type is shown as a regular datatype.

[epourmal@jam test]$ h5ls -v dtypes1.h5
Opened "dtypes1.h5" with sec2 driver.
cmp_dset Dataset {3/3}
    Location: 1:1272
    Links: 1
    Storage: 12 logical bytes, 0 allocated bytes
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
cmp_type Type
    Location: 1:1176
    Links: 2
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
str_type Type
    Location: 1:800
    Links: 1
    Type: shared-1:800 variable-length null-terminated ASCII string

When the second call to

if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR

is commented out, the infinite loop goes away.

Hopefully, you can fix your application since the member of the committed compound datatype cannot be a committed datatype (even it was created using a committed one), but we still have an issue in HDF5.

I confirmed that HDF5 1.8.* gives an infinite loop while HDF5 trunk is fine. It is not clear why the first H5committed call for the VL string type is not failing and why your test triggers an infinite loop.

For you information I entered JIRA issue HDFFV-9174.

Thanks again for your report!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org<http://hdfgroup.org/>
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 17, 2015, at 1:09 AM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

<dtypes.c.patch>

<dtypes.c.patch>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

epourmal · March 19, 2015, 3:42pm

Mark,

···

On Mar 18, 2015, at 7:16 PM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

UNCLASSIFIED

Thanks. It’s unfortunate that HDF-5 does not support nested committed data types.

Committed datatypes were initially designed to be shared by datasets and attributes. It looks like it would be useful to extend the feature to the committed datatypes.

I may have discovered why the test is entering an infinite loop. It is actually a problem with the test. When the second call to
if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR
fails, it jumps to the end where it closes “strtype” and “cmptype” – both of which are already closed. If I set these handles to -1 when they are closed mid-way through the test, no infinite loop occurs. I suspect it is not valid to close a handle twice in HDF and so there is a logical error with the test code.

I will send through a new patch later on.

Thank you! I think HDF5 library shouldn’t go into infinite loop if there is an application error. We will take a look.

I guess I can accept that HDF-5 does not support nested committed data types, however the behaviour of H5Tcommitted() is clearly inconsistent depending on whether you have closed and reopened the file or not. The super-type currently says it *is* committed in the first part of the test. Would it be reasonable to restrict the bug report then to say that H5Tcommitted() should return 0 both before and after reopening the file?
This is definitely an issue we need to address.

When designing our application it would have been useful if the HDF documentation stated this limitation. We may have designed it differently if we knew.

We encourage organizations to talk to us before making any design decisions and application development. This is what our Priority Support and Special Projects services are for see http://www.hdfgroup.org/services/. I agree that we should make our documentation more clear.

Can I request a feature enhancement issue also be created in JIRA also for “adding support for nested committed data types”? The use case is for something similar to H5view but where the views available for a data type (including the types of elements of an array and members of a compound) depend on the *name* of the data type (not the name of the field). So, an array of 3x floating-point values might have a committed name “position” and another otherwise identical data type might have committed name “velocity” – the viewer application would treat them differently. Our work-around is to use metadata attributes on the top-level committed data type and then separately and recursively open each committed super-type.
Done. HDFFV-9176.

In terms of our application crash, my colleague has been debugging HDF while running in our application and may have found why it is crashing – and it is not this bug (which we thought it was). It is a different problem, and I’ll hopefully submit a separate issue later on. We’ll try and reproduce it in the HDF test harness in some detail before submitting.

PS. Do you allow public access to your JIRA on-demand instance? Or is it for internal use only?

Our JIRA is for internal use only. The issue numbers are given for reference only.

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Mark

IMPORTANT: This email remains the property of the Department of Defence and is subject to the jurisdiction of section 70 of the Crimes Act 1914. If you have received this email in error, you are requested to contact the sender and delete the email.

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Thursday, 19 March 2015 8:45 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Bug in HDF-1.8.14 - Recursive named data types [SEC=UNCLASSIFIED]

Mark,

Thank you for the patch that illustrates the problem! It really saves a lot of investigation time!

HDF5 doesn’t support nested committed datatypes. In your example, when committed VL string is used as a member of the compound datatype, it is treated as a regular datatype. You can easily see it by using h5ls on the created dtypes1.h5 file. In the output below, cmp_type and str_type are shared (committed), but “vlstr” member of the cpm_type type is shown as a regular datatype.

[epourmal@jam test]$ h5ls -v dtypes1.h5
Opened "dtypes1.h5" with sec2 driver.
cmp_dset Dataset {3/3}
    Location: 1:1272
    Links: 1
    Storage: 12 logical bytes, 0 allocated bytes
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
cmp_type Type
    Location: 1:1176
    Links: 2
    Type: shared-1:1176 struct {
                   "vlstr" +0 variable-length null-terminated ASCII string
               } 4 bytes
str_type Type
    Location: 1:800
    Links: 1
    Type: shared-1:800 variable-length null-terminated ASCII string

When the second call to

if(H5Tcommitted(reopened_strtype) != 1) TEST_ERROR

is commented out, the infinite loop goes away.

Hopefully, you can fix your application since the member of the committed compound datatype cannot be a committed datatype (even it was created using a committed one), but we still have an issue in HDF5.

I confirmed that HDF5 1.8.* gives an infinite loop while HDF5 trunk is fine. It is not clear why the first H5committed call for the VL string type is not failing and why your test triggers an infinite loop.

For you information I entered JIRA issue HDFFV-9174.

Thanks again for your report!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org<http://hdfgroup.org/>
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 17, 2015, at 1:09 AM, Hodson, Mark (Contractor) <Mark.Hodson@dsto.defence.gov.au<mailto:Mark.Hodson@dsto.defence.gov.au>> wrote:

<dtypes.c.patch>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5