memory leak reading compressed files

I am reading a large number (186) of HDF5 compressed datafiles. Eventually the allocated memory exceeds 20g and the program fails.

I've traced the increasing memory allocation to the H5Z_filter_deflate() function in H5Zdeflate.c, specifically the lines:

        nalloc *= 2;
        if(NULL == (new_outbuf = H5MM_realloc(outbuf, nalloc))) {
            (void)inflateEnd(&z_strm);
            HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, 0, "memory allocation failed for deflate uncompression")

It appears as if this memory is never freed causing the memory leak.

Is this the desired behaviour or could there be a bug that prevents this
memory from being freed?

···

=========================
Joel Gales
Futuretech Corp.

SIMBIOS Code 616.0
Phone: (301) 286-1403
FAX: (301) 286-0268

Bin Globally Map Locally

Hi Joel,

Which version of HDF5 are you using? Would you be able to send me a small program or code snippet and a sample file to reproduce the issue? A similar issue came up a while back (pre-HDF5-1.8.5) with a netCDF-4 user, but we never determined what the issue was.

You can place the files on our local ftp server. Let me know (at help@hdfgroup.org) when they are there.

Here are the instructions:

···

---------------
ftp ftp.hdfgroup.uiuc.edu

Login as "anonymous"

Go to the pub/incoming/help/ directory:
cd pub/incoming/help

Type 'pwd' to see if you are in that location. (You will not be able to see the contents of
the directory.):
pwd

Type 'bin' to put in binary mode:
bin

Place file in directory after you are in binary mode:
put xxx

Get out of ftp:
quit
--------------

Thanks!
-Barbara

=================
Barbara Jones
The HDF Helpdesk

The HDF Group
help@hdfgroup.org

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Joel Gales
Sent: Sunday, February 23, 2014 2:01 PM
To: hdf-forum@lists.hdfgroup.org
Cc: Bryan Franz; sean bailey
Subject: [Hdf-forum] memory leak reading compressed files

I am reading a large number (186) of HDF5 compressed datafiles.
Eventually the allocated memory exceeds 20g and the program fails.

I've traced the increasing memory allocation to the H5Z_filter_deflate() function in H5Zdeflate.c, specifically the lines:

        nalloc *= 2;
        if(NULL == (new_outbuf = H5MM_realloc(outbuf, nalloc))) {
            (void)inflateEnd(&z_strm);
            HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, 0, "memory allocation failed for deflate uncompression")

It appears as if this memory is never freed causing the memory leak.

Is this the desired behaviour or could there be a bug that prevents this memory from being freed?

=========================
Joel Gales
Futuretech Corp.

SIMBIOS Code 616.0
Phone: (301) 286-1403
FAX: (301) 286-0268

Bin Globally Map Locally

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

Barbara,

I've place a datafile A2003061013500.L2_LAC_AT110 on the ftp server as well as a routine compress_memleak_multifiles.c that I hope will demonstrate the problem.

It is actually a netcdf4 file but the HDF5 library is called to do the reads and the example program uses only HDF5 calls.

It simulates multiple files by opening the same one five times. The allocated memory leak is about 15M.

We are using hdf5-1.8.10-patch1.

Joel

···

On Fri, 28 Feb 2014, Barbara Jones wrote:

Hi Joel,

Which version of HDF5 are you using? Would you be able to send me a small program or code snippet and a sample file to reproduce the issue? A similar issue came up a while back (pre-HDF5-1.8.5) with a netCDF-4 user, but we never determined what the issue was.

You can place the files on our local ftp server. Let me know (at help@hdfgroup.org) when they are there.

Here are the instructions:
---------------
ftp ftp.hdfgroup.uiuc.edu

Login as "anonymous"

Go to the pub/incoming/help/ directory:
cd pub/incoming/help

Type 'pwd' to see if you are in that location. (You will not be able to see the contents of
the directory.):
pwd

Type 'bin' to put in binary mode:
bin

Place file in directory after you are in binary mode:
put xxx

Get out of ftp:
quit
--------------

Thanks!
-Barbara

=================
Barbara Jones
The HDF Helpdesk

The HDF Group
help@hdfgroup.org

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Joel Gales
Sent: Sunday, February 23, 2014 2:01 PM
To: hdf-forum@lists.hdfgroup.org
Cc: Bryan Franz; sean bailey
Subject: [Hdf-forum] memory leak reading compressed files

I am reading a large number (186) of HDF5 compressed datafiles.
Eventually the allocated memory exceeds 20g and the program fails.

I've traced the increasing memory allocation to the H5Z_filter_deflate() function in H5Zdeflate.c, specifically the lines:

       nalloc *= 2;
       if(NULL == (new_outbuf = H5MM_realloc(outbuf, nalloc))) {
           (void)inflateEnd(&z_strm);
           HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, 0, "memory allocation failed for deflate uncompression")

It appears as if this memory is never freed causing the memory leak.

Is this the desired behaviour or could there be a bug that prevents this memory from being freed?

=========================
Joel Gales
Futuretech Corp.

SIMBIOS Code 616.0
Phone: (301) 286-1403
FAX: (301) 286-0268

Bin Globally Map Locally

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

=========================
Joel Gales
Futuretech Corp.

SIMBIOS Code 616.0
Phone: (301) 286-1403
FAX: (301) 286-0268

Bin Globally Map Locally

Thanks, Joel! I have your files and will take a look.
-Barbara

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Joel Gales
Sent: Friday, February 28, 2014 2:27 PM
To: HDF Users Discussion List
Cc: Bryan Franz; sean bailey
Subject: Re: [Hdf-forum] memory leak reading compressed files

Barbara,

I've place a datafile A2003061013500.L2_LAC_AT110 on the ftp server as well as a routine compress_memleak_multifiles.c that I hope will demonstrate the problem.

It is actually a netcdf4 file but the HDF5 library is called to do the reads and the example program uses only HDF5 calls.

It simulates multiple files by opening the same one five times. The allocated memory leak is about 15M.

We are using hdf5-1.8.10-patch1.

Joel

On Fri, 28 Feb 2014, Barbara Jones wrote:

Hi Joel,

Which version of HDF5 are you using? Would you be able to send me a small program or code snippet and a sample file to reproduce the issue? A similar issue came up a while back (pre-HDF5-1.8.5) with a netCDF-4 user, but we never determined what the issue was.

You can place the files on our local ftp server. Let me know (at help@hdfgroup.org) when they are there.

Here are the instructions:
---------------
ftp ftp.hdfgroup.uiuc.edu

Login as "anonymous"

Go to the pub/incoming/help/ directory:
cd pub/incoming/help

Type 'pwd' to see if you are in that location. (You will not be able
to see the contents of the directory.):
pwd

Type 'bin' to put in binary mode:
bin

Place file in directory after you are in binary mode:
put xxx

Get out of ftp:
quit
--------------

Thanks!
-Barbara

=================
Barbara Jones
The HDF Helpdesk

The HDF Group
help@hdfgroup.org

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On
Behalf Of Joel Gales
Sent: Sunday, February 23, 2014 2:01 PM
To: hdf-forum@lists.hdfgroup.org
Cc: Bryan Franz; sean bailey
Subject: [Hdf-forum] memory leak reading compressed files

I am reading a large number (186) of HDF5 compressed datafiles.
Eventually the allocated memory exceeds 20g and the program fails.

I've traced the increasing memory allocation to the H5Z_filter_deflate() function in H5Zdeflate.c, specifically the lines:

       nalloc *= 2;
       if(NULL == (new_outbuf = H5MM_realloc(outbuf, nalloc))) {
           (void)inflateEnd(&z_strm);
           HGOTO_ERROR(H5E_RESOURCE, H5E_NOSPACE, 0, "memory
allocation failed for deflate uncompression")

It appears as if this memory is never freed causing the memory leak.

Is this the desired behaviour or could there be a bug that prevents this memory from being freed?

=========================
Joel Gales
Futuretech Corp.

SIMBIOS Code 616.0
Phone: (301) 286-1403
FAX: (301) 286-0268

Bin Globally Map Locally

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgro
up.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgro
up.org

=========================
Joel Gales
Futuretech Corp.

SIMBIOS Code 616.0
Phone: (301) 286-1403
FAX: (301) 286-0268

Bin Globally Map Locally

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org