Thread-safe hdf5.dll leaks memory

paramon · November 28, 2014, 4:39pm

Hello everyone!

I configure HDF5 build with command:

cmake -G "Visual Studio 12" -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DHDF5_ENABLE_Z_LIB_SUPPORT=ON -DHDF5_ALLOW_EXTERNAL_SUPPORT=SVN -DZLIB_SVN_URL=http://svn.hdfgroup.uiuc.edu/zlib/trunk -DHDF5_ENABLE_THREADSAFE=ON hdf5-1.8.14

Then I compile and run the following minimal (Delphi) program, which emulates multiple hdf5.dll dynamic load-use-unload pattern:

program test;

uses
windows;

type
   herr_t = Integer;
var
   Dll: HMODULE;
   H5open: function: herr_t; cdecl;
   H5close: function: herr_t; cdecl;
begin
   while True do
   begin
     Dll := LoadLibrary('hdf5.dll');
     try
       @H5open := GetProcAddress(Dll, 'H5open');
       @H5close := GetProcAddress(Dll, 'H5close');
       H5open();
       H5close();
     finally
       FreeLibrary(Dll);
     end;
   end;
end.

When I run the above program for some time, I see memory usage constantly growing (~1MB per minute).

The problem is reproducible with HDF5 1.8.13, but not reproducible when I configure with -DHDF5_ENABLE_THREADSAFE=OFF or use stock HDF5 binaries.

Albeit my minimal program is pathological, it illustrates real-life usage pattern (imagine long-running server application). Also, the presence of even minimal memory leaks in HDF5 complicates memory leak profiling for the whole application.

Is it a known problem?
Is there additional info I should provide for you to help localize and eliminate the problem?

Best wishes,
Andrey Paramonov

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

derobins · November 29, 2014, 1:16am

Hi Andrey,

There have been issues in the past with thread-local storage on Windows, but I thought I had fixed those when I moved TLS setup and teardown to a DllMain function in the library.

I'll create a JIRA issue for this and take another look during this release cycle. One thing I did not do was test repeated load/unload from a managed environment so I'm not sure if there are issues there. I don't have Delphi, but I can try doing the same thing in C# or some other environment.

Cheers,

Dana

···

________________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Андрей Парамонов <paramon@acdlabs.ru>
Sent: Friday, November 28, 2014 10:39 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Thread-safe hdf5.dll leaks memory

Hello everyone!

I configure HDF5 build with command:

cmake -G "Visual Studio 12" -DBUILD_SHARED_LIBS=ON
-DCMAKE_BUILD_TYPE=Release -DHDF5_ENABLE_Z_LIB_SUPPORT=ON
-DHDF5_ALLOW_EXTERNAL_SUPPORT=SVN
-DZLIB_SVN_URL=http://svn.hdfgroup.uiuc.edu/zlib/trunk
-DHDF5_ENABLE_THREADSAFE=ON hdf5-1.8.14

Then I compile and run the following minimal (Delphi) program, which
emulates multiple hdf5.dll dynamic load-use-unload pattern:

program test;

uses
windows;

type
   herr_t = Integer;
var
   Dll: HMODULE;
   H5open: function: herr_t; cdecl;
   H5close: function: herr_t; cdecl;
begin
   while True do
   begin
     Dll := LoadLibrary('hdf5.dll');
     try
       @H5open := GetProcAddress(Dll, 'H5open');
       @H5close := GetProcAddress(Dll, 'H5close');
       H5open();
       H5close();
     finally
       FreeLibrary(Dll);
     end;
   end;
end.

When I run the above program for some time, I see memory usage
constantly growing (~1MB per minute).

The problem is reproducible with HDF5 1.8.13, but not reproducible when
I configure with -DHDF5_ENABLE_THREADSAFE=OFF or use stock HDF5 binaries.

Albeit my minimal program is pathological, it illustrates real-life
usage pattern (imagine long-running server application). Also, the
presence of even minimal memory leaks in HDF5 complicates memory leak
profiling for the whole application.

Is it a known problem?
Is there additional info I should provide for you to help localize and
eliminate the problem?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

derobins · November 29, 2014, 2:36am

For future reference, the JIRA ticket is HDFFV-9033.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
Dana Robinson
Sent: Friday, November 28, 2014 8:17 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Thread-safe hdf5.dll leaks memory

Hi Andrey,

There have been issues in the past with thread-local storage on Windows,
but I thought I had fixed those when I moved TLS setup and teardown to a
DllMain function in the library.

I'll create a JIRA issue for this and take another look during this
release cycle. One thing I did not do was test repeated load/unload from a
managed environment so I'm not sure if there are issues there. I don't
have Delphi, but I can try doing the same thing in C# or some other
environment.

Cheers,

Dana

________________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Андрей
Парамонов <paramon@acdlabs.ru>
Sent: Friday, November 28, 2014 10:39 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Thread-safe hdf5.dll leaks memory

Hello everyone!

I configure HDF5 build with command:

cmake -G "Visual Studio 12" -DBUILD_SHARED_LIBS=ON -
DCMAKE_BUILD_TYPE=Release -DHDF5_ENABLE_Z_LIB_SUPPORT=ON -
DHDF5_ALLOW_EXTERNAL_SUPPORT=SVN -
DZLIB_SVN_URL=http://svn.hdfgroup.uiuc.edu/zlib/trunk
-DHDF5_ENABLE_THREADSAFE=ON hdf5-1.8.14

Then I compile and run the following minimal (Delphi) program, which
emulates multiple hdf5.dll dynamic load-use-unload pattern:

program test;

uses
   windows;

type
   herr_t = Integer;
var
   Dll: HMODULE;
   H5open: function: herr_t; cdecl;
   H5close: function: herr_t; cdecl;
begin
   while True do
   begin
     Dll := LoadLibrary('hdf5.dll');
     try
       @H5open := GetProcAddress(Dll, 'H5open');
       @H5close := GetProcAddress(Dll, 'H5close');
       H5open();
       H5close();
     finally
       FreeLibrary(Dll);
     end;
   end;
end.

When I run the above program for some time, I see memory usage constantly
growing (~1MB per minute).

The problem is reproducible with HDF5 1.8.13, but not reproducible when I
configure with -DHDF5_ENABLE_THREADSAFE=OFF or use stock HDF5 binaries.

Albeit my minimal program is pathological, it illustrates real-life usage
pattern (imagine long-running server application). Also, the presence of
even minimal memory leaks in HDF5 complicates memory leak profiling for
the whole application.

Is it a known problem?
Is there additional info I should provide for you to help localize and
eliminate the problem?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-
forum_lists.hdfgroup.org
Twitter: x.com

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-
forum_lists.hdfgroup.org
Twitter: x.com

paramon · December 1, 2014, 9:19am

29.11.2014 4:16, Dana Robinson ��:

I'll create a JIRA issue for this and take another look during this release cycle. One thing I did not do was test repeated load/unload from a managed environment so I'm not sure if there are issues there. I don't have Delphi, but I can try doing the same thing in C# or some other environment.

Hello Dana!

Please note that Delphi produces unmanaged code (like C and unlike C#).
You can you FreePascal (http://www.freepascal.org/\) to compile my minimal example (fpc test.dpr -Mdelphi).
For your convenience I've converted my code to plain C (in attachment). It also reproduces the problem.

Best wishes,
and Thank you for your support,
Andrey Paramonov

test.c (427 Bytes)

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

derobins · December 1, 2014, 9:28am

Excellent! Thank you very much!

I'll add that to the bug report.

Dana

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
??? ???
Sent: Monday, December 01, 2014 4:20 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] Thread-safe hdf5.dll leaks memory

29.11.2014 4:16, Dana Robinson пишет:
> I'll create a JIRA issue for this and take another look during this
release cycle. One thing I did not do was test repeated load/unload from a
managed environment so I'm not sure if there are issues there. I don't
have Delphi, but I can try doing the same thing in C# or some other
environment.

Hello Dana!

Please note that Delphi produces unmanaged code (like C and unlike C#).
You can you FreePascal (http://www.freepascal.org/\) to compile my minimal
example (fpc test.dpr -Mdelphi).
For your convenience I've converted my code to plain C (in attachment).
It also reproduces the problem.

Best wishes,
and Thank you for your support,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

Attention! https://support.hdfgroup.org is the NEW home for documentation from The HDF Group. (Details)

Thread-safe hdf5.dll leaks memory