Thread-safe hdf5.dll leaks memory

Hello everyone!

I configure HDF5 build with command:

cmake -G "Visual Studio 12" -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release -DHDF5_ENABLE_Z_LIB_SUPPORT=ON -DHDF5_ALLOW_EXTERNAL_SUPPORT=SVN -DZLIB_SVN_URL=http://svn.hdfgroup.uiuc.edu/zlib/trunk -DHDF5_ENABLE_THREADSAFE=ON hdf5-1.8.14

Then I compile and run the following minimal (Delphi) program, which emulates multiple hdf5.dll dynamic load-use-unload pattern:

program test;

uses
   windows;

type
   herr_t = Integer;
var
   Dll: HMODULE;
   H5open: function: herr_t; cdecl;
   H5close: function: herr_t; cdecl;
begin
   while True do
   begin
     Dll := LoadLibrary('hdf5.dll');
     try
       @H5open := GetProcAddress(Dll, 'H5open');
       @H5close := GetProcAddress(Dll, 'H5close');
       H5open();
       H5close();
     finally
       FreeLibrary(Dll);
     end;
   end;
end.

When I run the above program for some time, I see memory usage constantly growing (~1MB per minute).

The problem is reproducible with HDF5 1.8.13, but not reproducible when I configure with -DHDF5_ENABLE_THREADSAFE=OFF or use stock HDF5 binaries.

Albeit my minimal program is pathological, it illustrates real-life usage pattern (imagine long-running server application). Also, the presence of even minimal memory leaks in HDF5 complicates memory leak profiling for the whole application.

Is it a known problem?
Is there additional info I should provide for you to help localize and eliminate the problem?

Best wishes,
Andrey Paramonov

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Hi Andrey,

There have been issues in the past with thread-local storage on Windows, but I thought I had fixed those when I moved TLS setup and teardown to a DllMain function in the library.

I'll create a JIRA issue for this and take another look during this release cycle. One thing I did not do was test repeated load/unload from a managed environment so I'm not sure if there are issues there. I don't have Delphi, but I can try doing the same thing in C# or some other environment.

Cheers,

Dana

···

________________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Андрей Парамонов <paramon@acdlabs.ru>
Sent: Friday, November 28, 2014 10:39 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Thread-safe hdf5.dll leaks memory

Hello everyone!

I configure HDF5 build with command:

cmake -G "Visual Studio 12" -DBUILD_SHARED_LIBS=ON
-DCMAKE_BUILD_TYPE=Release -DHDF5_ENABLE_Z_LIB_SUPPORT=ON
-DHDF5_ALLOW_EXTERNAL_SUPPORT=SVN
-DZLIB_SVN_URL=http://svn.hdfgroup.uiuc.edu/zlib/trunk
-DHDF5_ENABLE_THREADSAFE=ON hdf5-1.8.14

Then I compile and run the following minimal (Delphi) program, which
emulates multiple hdf5.dll dynamic load-use-unload pattern:

program test;

uses
   windows;

type
   herr_t = Integer;
var
   Dll: HMODULE;
   H5open: function: herr_t; cdecl;
   H5close: function: herr_t; cdecl;
begin
   while True do
   begin
     Dll := LoadLibrary('hdf5.dll');
     try
       @H5open := GetProcAddress(Dll, 'H5open');
       @H5close := GetProcAddress(Dll, 'H5close');
       H5open();
       H5close();
     finally
       FreeLibrary(Dll);
     end;
   end;
end.

When I run the above program for some time, I see memory usage
constantly growing (~1MB per minute).

The problem is reproducible with HDF5 1.8.13, but not reproducible when
I configure with -DHDF5_ENABLE_THREADSAFE=OFF or use stock HDF5 binaries.

Albeit my minimal program is pathological, it illustrates real-life
usage pattern (imagine long-running server application). Also, the
presence of even minimal memory leaks in HDF5 complicates memory leak
profiling for the whole application.

Is it a known problem?
Is there additional info I should provide for you to help localize and
eliminate the problem?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

For future reference, the JIRA ticket is HDFFV-9033.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
Dana Robinson
Sent: Friday, November 28, 2014 8:17 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Thread-safe hdf5.dll leaks memory

Hi Andrey,

There have been issues in the past with thread-local storage on Windows,
but I thought I had fixed those when I moved TLS setup and teardown to a
DllMain function in the library.

I'll create a JIRA issue for this and take another look during this
release cycle. One thing I did not do was test repeated load/unload from a
managed environment so I'm not sure if there are issues there. I don't
have Delphi, but I can try doing the same thing in C# or some other
environment.

Cheers,

Dana

________________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Андрей
Парамонов <paramon@acdlabs.ru>
Sent: Friday, November 28, 2014 10:39 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Thread-safe hdf5.dll leaks memory

Hello everyone!

I configure HDF5 build with command:

cmake -G "Visual Studio 12" -DBUILD_SHARED_LIBS=ON -
DCMAKE_BUILD_TYPE=Release -DHDF5_ENABLE_Z_LIB_SUPPORT=ON -
DHDF5_ALLOW_EXTERNAL_SUPPORT=SVN -
DZLIB_SVN_URL=http://svn.hdfgroup.uiuc.edu/zlib/trunk
-DHDF5_ENABLE_THREADSAFE=ON hdf5-1.8.14

Then I compile and run the following minimal (Delphi) program, which
emulates multiple hdf5.dll dynamic load-use-unload pattern:

program test;

uses
   windows;

type
   herr_t = Integer;
var
   Dll: HMODULE;
   H5open: function: herr_t; cdecl;
   H5close: function: herr_t; cdecl;
begin
   while True do
   begin
     Dll := LoadLibrary('hdf5.dll');
     try
       @H5open := GetProcAddress(Dll, 'H5open');
       @H5close := GetProcAddress(Dll, 'H5close');
       H5open();
       H5close();
     finally
       FreeLibrary(Dll);
     end;
   end;
end.

When I run the above program for some time, I see memory usage constantly
growing (~1MB per minute).

The problem is reproducible with HDF5 1.8.13, but not reproducible when I
configure with -DHDF5_ENABLE_THREADSAFE=OFF or use stock HDF5 binaries.

Albeit my minimal program is pathological, it illustrates real-life usage
pattern (imagine long-running server application). Also, the presence of
even minimal memory leaks in HDF5 complicates memory leak profiling for
the whole application.

Is it a known problem?
Is there additional info I should provide for you to help localize and
eliminate the problem?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-
forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-
forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

29.11.2014 4:16, Dana Robinson �����:

I'll create a JIRA issue for this and take another look during this release cycle. One thing I did not do was test repeated load/unload from a managed environment so I'm not sure if there are issues there. I don't have Delphi, but I can try doing the same thing in C# or some other environment.

Hello Dana!

Please note that Delphi produces unmanaged code (like C and unlike C#).
You can you FreePascal (http://www.freepascal.org/\) to compile my minimal example (fpc test.dpr -Mdelphi).
For your convenience I've converted my code to plain C (in attachment). It also reproduces the problem.

Best wishes,
and Thank you for your support,
Andrey Paramonov

test.c (427 Bytes)

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

Excellent! Thank you very much!

I'll add that to the bug report.

Dana

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
??? ???
Sent: Monday, December 01, 2014 4:20 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] Thread-safe hdf5.dll leaks memory

29.11.2014 4:16, Dana Robinson пишет:
> I'll create a JIRA issue for this and take another look during this
release cycle. One thing I did not do was test repeated load/unload from a
managed environment so I'm not sure if there are issues there. I don't
have Delphi, but I can try doing the same thing in C# or some other
environment.

Hello Dana!

Please note that Delphi produces unmanaged code (like C and unlike C#).
You can you FreePascal (http://www.freepascal.org/) to compile my minimal
example (fpc test.dpr -Mdelphi).
For your convenience I've converted my code to plain C (in attachment).
It also reproduces the problem.

Best wishes,
and Thank you for your support,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.