H5T module 'leaks' conversion paths


#1

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier


#2

I am developing on Windows, so valgrind isn't available to me (and thus I don't know the exact semantics of -show-reachable), but the memory profilers I've tried also don't detect this 'leak'. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of "Why does heap memory grow by 50MB each time I open and then close a [file]?" (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion 'paths' to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I've finally traced a memory 'leak' in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can't find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it's not the end of the world.

Thanks,
Matthew Xavier


#3

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks.
See this article for an example:
http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/
I believe there are ways to enact the dump only on changes that happened
between specific points in your code which would let you find something
that is getting cleaned up on exit but still shouldn't be happening. Sorry
I can't remember the specific calls for that but it's in the same part of
the API.

David

···

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com> wrote:

I am developing on Windows, so valgrind isn’t available to me (and thus I
don’t know the exact semantics of –show-reachable), but the memory
profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down
cleanly, it frees the conversion path cache. I found it because I was
chasing a bug report consisting of “Why does heap memory grow by 50MB each
time I open and then close a [file]?” (A file in this case is actually a
more complicated structure and may cause hundreds of HDF5 files to be
accessed.)

I think that the presence of a vlen field is important. As I recall, the
type comparison for compound types does not check what files the types were
defined in, but if a vlen field is present, then the files in which the
vlen field types were defined can make two otherwise identical compound
types compare as different.

I have not used committed datatypes. I will take a look at that function
and see if it changes the behavior.

Matthew

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Miller, Mark C.
*Sent:* Tuesday, March 29, 2016 12:02 PM
*To:* HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Subject:* Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code
and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory
behavior you are seeing?

Mark

*From: *Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of
"Xavier, Matthew" <Matthew.Xavier@mts.com>
*Reply-To: *HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Date: *Monday, March 28, 2016 2:15 PM
*To: *"hdf-forum@lists.hdfgroup.org" <hdf-forum@lists.hdfgroup.org>
*Subject: *[Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When
I read these datasets, HDF5 creates new conversion ‘paths’ to convert
between the file types and memory types involved. HDF5 caches these paths
(see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded
growth of the conversion path cache. HDF5 treats vlen types as different if
they come from different files, so I get a new set of conversion paths for
every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the
cached paths when I close a file. There is a function provided for removing
paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not
work for compound types because of the way that the pers parameter is
handled. If I pass H5T_PERS_HARD, no compound type conversions are removed
because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls
back on the compound->compound soft conversion and generates a new path.
Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister()
removes the default compound->compound soft conversion and I can't read any
more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function
determines file identity depends on pointers that are left dangling when a
file is closed, which both complicated my minimum reproduction of the
problem and also undermines the file identity check. (When the same
allocation is re-used from the free list for a different file, types can
compare as the same even when they are from different files, which is
contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one
argument, which is a path at which it can write a temporary file. To run it
does require a custom build of HDF5 so that the test program can read the
size of the path table. (Or alternately, you can comment out the relevant
parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am
already patching my own HDF5 builds to get Unicode file name support on
Windows, so if I have to make code changes it’s not the end of the world.

Thanks,

Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


#4

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.

David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#5

BTW, I also ran under valgrind/massif tool to examine peak memory usage of your code over time.

I added an H5dont_atexit() call to your code to ensure HDF5 doesn't clean stuff up upon exit so that valgrind gets a good picture of whats allocated at the end.

Here is some ascii-art (thats what massif produces) of the memory profile over time. . .

···

--------------------------------------------------------------------------------
Command: ./Hdf5TypePathLeak gorfo.h5
Massif arguments: (none)
ms_print arguments: massif.out.137704
--------------------------------------------------------------------------------

    MB
2.832^#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#:::@:::@::::::@::::::@::::::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
   0 +----------------------------------------------------------------------->Gi
     0 5.966

Number of snapshots: 88
Detailed snapshots: [1 (peak), 5, 9, 17, 24, 32, 46, 53, 63, 73, 83]

The fact that its flat as a pancake after initial allocations means that the HDF5 library isn't holding on to anything either.

Mark

From: "Miller, Mark C." <miller86@llnl.gov<mailto:miller86@llnl.gov>>
Date: Tuesday, March 29, 2016 11:09 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.

David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#6

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I'm not allocating that memory to store something; I'm allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It's the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn't allocate a new conversion path.

So if you 'fix' my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()...

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn't available to me (and thus I don't know the exact semantics of -show-reachable), but the memory profilers I've tried also don't detect this 'leak'. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of "Why does heap memory grow by 50MB each time I open and then close a [file]?" (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion 'paths' to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I've finally traced a memory 'leak' in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can't find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it's not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#7

Well, ok, but…the *only* leak valgrind reported *was* the H5allocate_memory call.

I removed the H5allocate_memory() call entirely *and* kept the H5dont_atexit() call.

When I do that, I do indeed get a huge *bloat* in HDF5. I have attached a log from valgrind

What I mean by *bloat* is that the HDF5 library has NOT really *leaked* it. It still knows where all the memory is and is showing in the 'still reachable' category of stuff. The library is just not releasing it when it probably should.

The *only* way I can get HDF5 library to free that memory is to call H5close() which is rather drastic!!

Mark

h5_leak.out.gz (21.5 KB)

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path.

So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#8

Hi Matthew,

Try this. . .

extern herr_t H5T_init(void);
extern int H5T_term_interface(void);

int main(int argc, char *argv[])
{
.
.
.
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);
        H5T_term_interface();
        H5T_init();
.
.
.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path.

So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#9

Mark,

That idea seems promising...it alleviates the problem in the test program, at least. I'll see if I can find a way to make it work in my real application.

Matthew

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Wednesday, March 30, 2016 9:58 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Try this. . .

extern herr_t H5T_init(void);
extern int H5T_term_interface(void);

int main(int argc, char *argv[])
{
.
.
.
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);
        H5T_term_interface();
        H5T_init();
.
.
.

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I'm not allocating that memory to store something; I'm allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It's the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn't allocate a new conversion path.

So if you 'fix' my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()...

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn't available to me (and thus I don't know the exact semantics of -show-reachable), but the memory profilers I've tried also don't detect this 'leak'. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of "Why does heap memory grow by 50MB each time I open and then close a [file]?" (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion 'paths' to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I've finally traced a memory 'leak' in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can't find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it's not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#10

Hi Matthew,

I was just wondering. . .

Whatever became of this. Did my hack work?

Did you confirm there was/is a problem in the HDF5 library? If so, has it been fixed?

Mark

···

From: "Miller, Mark C." <miller86@llnl.gov<mailto:miller86@llnl.gov>>
Date: Wednesday, March 30, 2016 at 7:58 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Try this. . .

extern herr_t H5T_init(void);
extern int H5T_term_interface(void);

int main(int argc, char *argv[])
{
.
.
.
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);
        H5T_term_interface();
        H5T_init();
.
.
.

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path.

So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5


#11

This appears to still be a problem. I have used valgrind but am using the client request VALGRIND_DO_LEAK_CHECK just prior to return from main now rather than waiting for exit. And, I am seeing something like this…

==44479==
==44479== 46,624 bytes in 124 blocks are still reachable in loss record 4,235 of 4,235
==44479== at 0x4C29F73: malloc (vg_replace_malloc.c:307)
==44479== by 0x5901452: H5MM_malloc (H5MM.c:292)
==44479== by 0x5806C79: H5FL_malloc (H5FL.c:243)
==44479== by 0x58073F8: H5FL_reg_malloc (H5FL.c:441)
==44479== by 0x5ADD3C8: H5T__initiate_copy (H5T.c:3343)
==44479== by 0x5ADEABE: H5T_copy (H5T.c:3642)
==44479== by 0x5AFC0CA: H5T_conv_struct_init (H5Tconv.c:1975)
==44479== by 0x5AFDCC7: H5T__conv_struct_opt (H5Tconv.c:2392)
==44479== by 0x5AE43FF: H5T__path_find_real (H5T.c:5010)
==44479== by 0x5AE33EB: H5T_path_find (H5T.c:4814)
==44479== by 0x56138FD: H5A__write (H5Aint.c:735)
==44479== by 0x5C38579: H5VL__native_attr_write (H5VLnative_attr.c:205)
==44479==
==44479== LEAK SUMMARY:
==44479== definitely lost: 0 bytes in 0 blocks
==44479== indirectly lost: 0 bytes in 0 blocks
==44479== possibly lost: 0 bytes in 0 blocks
==44479== still reachable: 1,417,857 bytes in 8,242 blocks
==44479== suppressed: 0 bytes in 0 blocks
=

I am seeing same or similar on 1.8. 1.10 and 1.12. Does anyone know of a way to force HDF5 to clean up some of the memory on a periodic basis? Am I maybe missing some important call to free up this memory. I am not calling H5close(). If I do call H5close(), I do get rid of all these “leaks” but is calling that on a regular basis a good idea? And, I am not even sure I know for sure when I can safely call H5close() or not without causing problems elsewhere. Is there any issue with occasionally calling H5close() (at suitably chosen points)?


#12

FYI…, a call to H5garbage_collect() just prior to checking for leaks has an insigificant improvement on the situation.


#13

FYI…, calls to H5set_free_list_limits() using 1024 bytes and 10*1024 bytes for local and global values also has no siginficant effect on behavior here.


#14

It’s funny to see this topic turn up again. It has been a few years since I was maintainer of the relevant code.

About a year ago the current maintainer asked for my help to re-evaluate the H5T_term_interface()/H5T_init() workaround as those (internal) symbols had changed. My reproduction attempt at that time gave:

PS C:\Source\hdf5_1_8-TypePathLeak> .\Hdf5TypePathLeak.exe check.h5
HDF5 version: 1.8.17
          iterations        H5T_g.npaths  heapExceptSpoilers
                  68                 173              564111
                1045                1233             3937827
                2149                2437             7769999
                3323                3715            11820297
                4442                4929            15700563
                5603                6197            19719167
                6737                7411            23566649
                7868                8631            27498613
                8998                9865            31409483
               10121               11101            35326679
               11236               12297            39117131
               12340               13467            42825105
               13455               14713            46774011
               14569               15921            50602499

PS C:\Source\hdf5_1_10_6-TypePathLeak> .\Hdf5TypePathLeak.exe check.h5
HDF5 version: 1.10.6
          iterations        H5T_g.npaths  heapExceptSpoilers
                 335                 421             1364422
                 760                 859             2758488
                1172                1287             4124876
                1597                1727             5521044
                2029                2179             6971776
                2516                2685             8577414
                2999                3173            10125982
                3481                3665            11687202
                3951                4163            13300272
                4429                4647            14836124
                4908                5135            16384676
                5388                5669            18079198
                5843                6137            19564282
                6285                6623            21106420

PS C:\Source\hdf5_1_12_0-TypePathLeak> .\Hdf5TypePathLeak.exe check.h5
HDF5 version: 1.12.0
          iterations        H5T_g.npaths  heapExceptSpoilers
                   3                  97              344184
                 470                 315             1063374
                 905                 325             1096228
                1336                 327             1102802
                1774                 425             1424720
                2251                 437             1464148
                2739                 437             1464148
                3231                 437             1464148
                3723                 437             1464148
                4217                 437             1464148
                4710                 437             1464148
                5200                 437             1464148
                5694                 437             1464148
                6186                 437             1464148
                6661                 437             1464148
                7151                 437             1464148
                7643                 437             1464148
                8134                 437             1464148
                8625                 437             1464148
                9076                 437             1464148
                9510                 437             1464148
                9985                 437             1464148
               10480                 437             1464148
               10974                 437             1464148
               11461                 437             1464148
               11955                 437             1464148
               12448                 437             1464148
               12937                 437             1464148
               13422                 437             1464148

From which I concluded that while the path table still does not get entirely cleared out, there is at least some cap on its growth in 1.12 (and presumably later version). I believe it was sufficient for the new maintainer to remove the workaround.


#15

Ok, really good to know this. Thanks for sharing. That 1464148 number is very close to what I am seeing valgrind report on 1.12 which is 1417857,

==102751== 46,624 bytes in 124 blocks are still reachable in loss record 4,235 of 4,235
==102751== at 0x4C29F73: malloc (vg_replace_malloc.c:307)
==102751== by 0x5901452: H5MM_malloc (H5MM.c:292)
==102751== by 0x5806C79: H5FL_malloc (H5FL.c:243)
==102751== by 0x58073F8: H5FL_reg_malloc (H5FL.c:441)
==102751== by 0x5ADD3C8: H5T__initiate_copy (H5T.c:3343)
==102751== by 0x5ADEABE: H5T_copy (H5T.c:3642)
==102751== by 0x5AFC0CA: H5T_conv_struct_init (H5Tconv.c:1975)
==102751== by 0x5AFDCC7: H5T__conv_struct_opt (H5Tconv.c:2392)
==102751== by 0x5AE43FF: H5T__path_find_real (H5T.c:5010)
==102751== by 0x5AE33EB: H5T_path_find (H5T.c:4814)
==102751== by 0x56138FD: H5A__write (H5Aint.c:735)
==102751== by 0x5C38579: H5VL__native_attr_write (H5VLnative_attr.c:205)
==102751==
==102751== LEAK SUMMARY:
==102751== definitely lost: 0 bytes in 0 blocks
==102751== indirectly lost: 0 bytes in 0 blocks
==102751== possibly lost: 0 bytes in 0 blocks
==102751== still reachable: 1,417,857 bytes in 8,242 blocks
==102751== suppressed: 0 bytes in 0 blocks
==102751==
==102751==
==102751== HEAP SUMMARY:
==102751== in use at exit: 0 bytes in 0 blocks
==102751== total heap usage: 18,492 allocs, 18,492 frees, 5,515,220 bytes allocated


#16

I entered a JIRA issue HDFFV-11249. THG developers will take a look.

Thank you for reporting and interesting analysis.

Elena


#17

I also have the same case of problems. It is important to talk to app development service providers to find out how experienced the developers are. You should also consider the fact that some app development companies offer free trials and may have more features than others. There is also the option of hiring in-house app developers. Look into all these options before deciding on any particular developer. My favorite https://mlsdev.com/services/mobile-app-development


#18

Have the same problem described by the author. Therefore, I continue to follow this conversation. I would be very grateful if someone could describe the ways to solve it here. *patiently waiting