getting debugging information

Hdf-forum archives
1

I have a program that is a lot more complicated than I like (sadly, there's not a lot I can do about that), that I've recently converted to using HDF as its data store (from flat files). It also uses WebSphere MQ middleware to publish the results from the processing it does, so there's a bit of a loop. Program 1 generates data and publishes it, program 2 gets that published data and writes it to an HDF file, which program 1 in turn will occasionally read.

There's a lot of data involved, so using an internal store isn't really an option, and the HDF structure gives the advantage of being able to quickly find exactly the data that's needed at the time. Other programs also use the data being published by program 1, so removing the publication wouldn't be an option.

Anyway, hopefully that's sufficient background. The problem I'm seeing at the moment is that program 1 eventually will try to read some data from the HDF file that will fail in:
H5Z_filter_deflate(): inflate() failed
major: Data filters
minor: Unable to initialize object

This continues until memory runs out.

The fact that 1) a read is failing when all other programs have not exhibited this problem since I modified program 2 to be very diligent about flushing its data to disc regularly, and 2) memory is consumed on repeated calls to the function in program 1 that's supposed to read the HDF file until it runs out; makes it look like I have the file open and the HDF library is caching the file's superblock, which is causing it to fail to read the data.

I'm definitely closing the HDF5 object handles used in that function and in all the other functions I've checked so far, so my question is this: how can I get the library to give me a list of open hids? I've looked through the library API documentation and found H5Inmembers, which appears to provide the ability to determine *how many* identifiers of a given type are open, and I found H5Fget_obj_ids which appears to give one the ability to find all open object IDs given an open file ID, but I don't see a way to start from nothing and get open file IDs, object, group, link, etc. IDs.

Any help would be appreciated :slight_smile:

2

Hi John,

···

On Mar 11, 2011, at 9:56 AM, John Knutson wrote:

I have a program that is a lot more complicated than I like (sadly, there's not a lot I can do about that), that I've recently converted to using HDF as its data store (from flat files). It also uses WebSphere MQ middleware to publish the results from the processing it does, so there's a bit of a loop. Program 1 generates data and publishes it, program 2 gets that published data and writes it to an HDF file, which program 1 in turn will occasionally read.

There's a lot of data involved, so using an internal store isn't really an option, and the HDF structure gives the advantage of being able to quickly find exactly the data that's needed at the time. Other programs also use the data being published by program 1, so removing the publication wouldn't be an option.

Anyway, hopefully that's sufficient background. The problem I'm seeing at the moment is that program 1 eventually will try to read some data from the HDF file that will fail in:
H5Z_filter_deflate(): inflate() failed
major: Data filters
minor: Unable to initialize object

This continues until memory runs out.

The fact that 1) a read is failing when all other programs have not exhibited this problem since I modified program 2 to be very diligent about flushing its data to disc regularly, and 2) memory is consumed on repeated calls to the function in program 1 that's supposed to read the HDF file until it runs out; makes it look like I have the file open and the HDF library is caching the file's superblock, which is causing it to fail to read the data.

I'm definitely closing the HDF5 object handles used in that function and in all the other functions I've checked so far, so my question is this: how can I get the library to give me a list of open hids? I've looked through the library API documentation and found H5Inmembers, which appears to provide the ability to determine *how many* identifiers of a given type are open, and I found H5Fget_obj_ids which appears to give one the ability to find all open object IDs given an open file ID, but I don't see a way to start from nothing and get open file IDs, object, group, link, etc. IDs.

Any help would be appreciated :slight_smile:

  If you pass in H5F_OBJ_ALL for the file_id parameter (and H5F_OBJ_ALL for the types parameter) of H5Fget_obj_ids(), it will give you a list of all the open file IDs, along with the list of other open IDs.

  Quincey

3

Quincey Koziol wrote:

Hi John,

  If you pass in H5F_OBJ_ALL for the file_id parameter (and H5F_OBJ_ALL for the types parameter) of H5Fget_obj_ids(), it will give you a list of all the open file IDs, along with the list of other open IDs.
  

That's great news, thanks for the response. I also see it's documented,
I just failed to read the details about the parameters when I looked at
it. :slight_smile:

I've spent the past few days reexamining our code to make sure we take
special care to close any open HDF objects, but this function should
help confirm whether or not I missed anything.