H5T module 'leaks' conversion paths

miller86 · March 29, 2016, 5:01pm

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

matthew.xavier · March 29, 2016, 5:39pm

I am developing on Windows, so valgrind isn't available to me (and thus I don't know the exact semantics of -show-reachable), but the memory profilers I've tried also don't detect this 'leak'. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of "Why does heap memory grow by 50MB each time I open and then close a [file]?" (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion 'paths' to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I've finally traced a memory 'leak' in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can't find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it's not the end of the world.

Thanks,
Matthew Xavier

David1 · March 29, 2016, 5:52pm

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks.
See this article for an example:
http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/
I believe there are ways to enact the dump only on changes that happened
between specific points in your code which would let you find something
that is getting cleaned up on exit but still shouldn't be happening. Sorry
I can't remember the specific calls for that but it's in the same part of
the API.

David

···

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com> wrote:

I am developing on Windows, so valgrind isn’t available to me (and thus I
don’t know the exact semantics of –show-reachable), but the memory
profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down
cleanly, it frees the conversion path cache. I found it because I was
chasing a bug report consisting of “Why does heap memory grow by 50MB each
time I open and then close a [file]?” (A file in this case is actually a
more complicated structure and may cause hundreds of HDF5 files to be
accessed.)

I think that the presence of a vlen field is important. As I recall, the
type comparison for compound types does not check what files the types were
defined in, but if a vlen field is present, then the files in which the
vlen field types were defined can make two otherwise identical compound
types compare as different.

I have not used committed datatypes. I will take a look at that function
and see if it changes the behavior.

Matthew

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Miller, Mark C.
*Sent:* Tuesday, March 29, 2016 12:02 PM
*To:* HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Subject:* Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code
and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory
behavior you are seeing?

Mark

*From: *Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of
"Xavier, Matthew" <Matthew.Xavier@mts.com>
*Reply-To: *HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Date: *Monday, March 28, 2016 2:15 PM
*To: *"hdf-forum@lists.hdfgroup.org" <hdf-forum@lists.hdfgroup.org>
*Subject: *[Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When
I read these datasets, HDF5 creates new conversion ‘paths’ to convert
between the file types and memory types involved. HDF5 caches these paths
(see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded
growth of the conversion path cache. HDF5 treats vlen types as different if
they come from different files, so I get a new set of conversion paths for
every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the
cached paths when I close a file. There is a function provided for removing
paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not
work for compound types because of the way that the pers parameter is
handled. If I pass H5T_PERS_HARD, no compound type conversions are removed
because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls
back on the compound->compound soft conversion and generates a new path.
Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister()
removes the default compound->compound soft conversion and I can't read any
more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function
determines file identity depends on pointers that are left dangling when a
file is closed, which both complicated my minimum reproduction of the
problem and also undermines the file identity check. (When the same
allocation is re-used from the free list for a different file, types can
compare as the same even when they are from different files, which is
contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one
argument, which is a path at which it can write a temporary file. To run it
does require a custom build of HDF5 so that the test program can read the
size of the path table. (Or alternately, you can comment out the relevant
parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am
already patching my own HDF5 builds to get Unicode file name support on
Windows, so if I have to make code changes it’s not the end of the world.

Thanks,

Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

miller86 · March 29, 2016, 6:09pm

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.

David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

miller86 · March 29, 2016, 6:17pm

BTW, I also ran under valgrind/massif tool to examine peak memory usage of your code over time.

I added an H5dont_atexit() call to your code to ensure HDF5 doesn't clean stuff up upon exit so that valgrind gets a good picture of whats allocated at the end.

Here is some ascii-art (thats what massif produces) of the memory profile over time. . .

···

--------------------------------------------------------------------------------
Command: ./Hdf5TypePathLeak gorfo.h5
Massif arguments: (none)
ms_print arguments: massif.out.137704
--------------------------------------------------------------------------------

    MB
2.832^#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#
     >#:::@:::@::::::@::::::@::::::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
     >#:::@:::@::::::@::::::@::: ::@::::::::::::@:::::@::::::@::::::@::::::@::
   0 +----------------------------------------------------------------------->Gi
     0 5.966

Number of snapshots: 88
Detailed snapshots: [1 (peak), 5, 9, 17, 24, 32, 46, 53, 63, 73, 83]

The fact that its flat as a pancake after initial allocations means that the HDF5 library isn't holding on to anything either.

Mark

From: "Miller, Mark C." <miller86@llnl.gov<mailto:miller86@llnl.gov>>
Date: Tuesday, March 29, 2016 11:09 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.

David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

matthew.xavier · March 29, 2016, 6:38pm

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I'm not allocating that memory to store something; I'm allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It's the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn't allocate a new conversion path.

So if you 'fix' my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()...

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn't available to me (and thus I don't know the exact semantics of -show-reachable), but the memory profilers I've tried also don't detect this 'leak'. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of "Why does heap memory grow by 50MB each time I open and then close a [file]?" (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion 'paths' to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I've finally traced a memory 'leak' in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can't find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it's not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

miller86 · March 29, 2016, 9:51pm

Well, ok, but…the *only* leak valgrind reported *was* the H5allocate_memory call.

I removed the H5allocate_memory() call entirely *and* kept the H5dont_atexit() call.

When I do that, I do indeed get a huge *bloat* in HDF5. I have attached a log from valgrind

What I mean by *bloat* is that the HDF5 library has NOT really *leaked* it. It still knows where all the memory is and is showing in the 'still reachable' category of stuff. The library is just not releasing it when it probably should.

The *only* way I can get HDF5 library to free that memory is to call H5close() which is rather drastic!!

Mark

h5_leak.out.gz (21.5 KB)

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path.

So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: Tracking memory leaks in Visual Studio | David Amador I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

miller86 · March 30, 2016, 2:58pm

Hi Matthew,

Try this. . .

extern herr_t H5T_init(void);
extern int H5T_term_interface(void);

int main(int argc, char *argv[])
{
.
.
.
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);
        H5T_term_interface();
        H5T_init();
.
.
.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path.

So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

matthew.xavier · March 30, 2016, 3:58pm

Mark,

That idea seems promising...it alleviates the problem in the test program, at least. I'll see if I can find a way to make it work in my real application.

Matthew

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Wednesday, March 30, 2016 9:58 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Try this. . .

extern herr_t H5T_init(void);
extern int H5T_term_interface(void);

int main(int argc, char *argv[])
{
.
.
.
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);
        H5T_term_interface();
        H5T_init();
.
.
.

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I'm not allocating that memory to store something; I'm allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It's the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn't allocate a new conversion path.

So if you 'fix' my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()...

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn't available to me (and thus I don't know the exact semantics of -show-reachable), but the memory profilers I've tried also don't detect this 'leak'. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of "Why does heap memory grow by 50MB each time I open and then close a [file]?" (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion 'paths' to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I've finally traced a memory 'leak' in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can't find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it's not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

miller86 · May 12, 2016, 9:38pm

Hi Matthew,

I was just wondering. . .

Whatever became of this. Did my hack work?

Did you confirm there was/is a problem in the HDF5 library? If so, has it been fixed?

Mark

···

From: "Miller, Mark C." <miller86@llnl.gov<mailto:miller86@llnl.gov>>
Date: Wednesday, March 30, 2016 at 7:58 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Try this. . .

extern herr_t H5T_init(void);
extern int H5T_term_interface(void);

int main(int argc, char *argv[])
{
.
.
.
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);
        H5T_term_interface();
        H5T_init();
.
.
.

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 11:38 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Yes, the H5allocate_memory() call is kind of sloppy. I could get the same effect by putting all of those allocations in a list and freeing them before the process exits.

As you may note, I’m not allocating that memory to store something; I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request. It’s the same reason that I set the free list limits to zero. If the H5F_t structure when the file is re-opened lands on the same address, then the vlen type equality check passes because the dangling pointer in the cached conversion path happens to match. In that case, HDF5 doesn’t allocate a new conversion path.

So if you ‘fix’ my call to H5allocate_memory(), you almost certainly will not see any errors.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 1:10 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Hi Matthew,

Ok, I ran your code under valgrind and cannot confirm your leak report.

I was doing this against 1.8.16 HDF5 library

I *do* show a leak in your client code due to the H5allocate_memory() call with no matching H5free_memory() call. When I add a matching H5free_memory() call, the leak valgrind *does* report does away. . .

Valgrind run *WITH* H5free_memory()…

gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}272: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135138== Memcheck, a memory error detector
==135138== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135138== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135138== Command: ./Hdf5TypePathLeak gorfo.h5
==135138==
got here
^C==135138==
==135138== HEAP SUMMARY:
==135138== in use at exit: 72 bytes in 1 blocks
==135138== total heap usage: 107,649 allocs, 107,648 frees, 617,724,658 bytes allocated
==135138==
==135138== 72 bytes in 1 blocks are definitely lost in loss record 1 of 1
==135138== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135138== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135138== by 0x4E69930: H5allocate_memory (H5.c:900)
==135138== by 0x401310: main (Hdf5TypePathLeak.c:213)
==135138==
==135138== LEAK SUMMARY:
==135138== definitely lost: 72 bytes in 1 blocks
==135138== indirectly lost: 0 bytes in 0 blocks
==135138== possibly lost: 0 bytes in 0 blocks
==135138== still reachable: 0 bytes in 0 blocks
==135138== suppressed: 0 bytes in 0 blocks
==135138==
==135138== For counts of detected and suppressed errors, rerun with: -v
==135138== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}273: !vi
vi Hdf5TypePathLeak.c

valgrind run *WITHOUT* H5free_memory().

surface86{miller86}274: !g
gcc -Bstatic -g -std=c99 Hdf5TypePathLeak.c -o Hdf5TypePathLeak -I/g/g11/miller86/tmp/hdf5-1.8.16/my_install/include -L/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib -lhdf5 -lm -Bdynamic -lz
surface86{miller86}275: env LD_LIBRARY_PATH=/g/g11/miller86/tmp/hdf5-1.8.16/my_install/lib valgrind --leak-check=full --show-reachable=yes ./Hdf5TypePathLeak gorfo.h5
==135256== Memcheck, a memory error detector
==135256== Copyright (C) 2002-2013, and GNU GPL'd, by Julian Seward et al.
==135256== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==135256== Command: ./Hdf5TypePathLeak gorfo.h5
==135256==
got here
^C==135256==
==135256== HEAP SUMMARY:
==135256== in use at exit: 36,000 bytes in 500 blocks
==135256== total heap usage: 225,358 allocs, 224,858 frees, 1,338,902,242 bytes allocated
==135256==
==135256== 36,000 bytes in 500 blocks are definitely lost in loss record 1 of 1
==135256== at 0x4C2820A: malloc (vg_replace_malloc.c:296)
==135256== by 0x5046907: H5MM_malloc (H5MM.c:66)
==135256== by 0x4E69930: H5allocate_memory (H5.c:900)
==135256== by 0x4012AD: main (Hdf5TypePathLeak.c:215)
==135256==
==135256== LEAK SUMMARY:
==135256== definitely lost: 36,000 bytes in 500 blocks
==135256== indirectly lost: 0 bytes in 0 blocks
==135256== possibly lost: 0 bytes in 0 blocks
==135256== still reachable: 0 bytes in 0 blocks
==135256== suppressed: 0 bytes in 0 blocks
==135256==
==135256== For counts of detected and suppressed errors, rerun with: -v
==135256== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 5 from 5)
surface86{miller86}276:

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, March 29, 2016 10:52 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

Matthew,

If you're using MSVC it has some useful tools for pinpointing memory leaks. See this article for an example: http://www.david-amador.com/2010/10/tracking-memory-leaks-in-visual-studio/ I believe there are ways to enact the dump only on changes that happened between specific points in your code which would let you find something that is getting cleaned up on exit but still shouldn't be happening. Sorry I can't remember the specific calls for that but it's in the same part of the API.
David

On Tue, Mar 29, 2016 at 10:39 AM, Xavier, Matthew <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>> wrote:
I am developing on Windows, so valgrind isn’t available to me (and thus I don’t know the exact semantics of –show-reachable), but the memory profilers I’ve tried also don’t detect this ‘leak’. If HDF5 shuts down cleanly, it frees the conversion path cache. I found it because I was chasing a bug report consisting of “Why does heap memory grow by 50MB each time I open and then close a [file]?” (A file in this case is actually a more complicated structure and may cause hundreds of HDF5 files to be accessed.)

I think that the presence of a vlen field is important. As I recall, the type comparison for compound types does not check what files the types were defined in, but if a vlen field is present, then the files in which the vlen field types were defined can make two otherwise identical compound types compare as different.

I have not used committed datatypes. I will take a look at that function and see if it changes the behavior.

Matthew

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Miller, Mark C.
Sent: Tuesday, March 29, 2016 12:02 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] H5T module 'leaks' conversion paths

I use compound types a lot *but*not* with vlen fields.

I often run valgrind --leak-check=yes --show-reachable=yes on my HDF5 code and do not observe the leak you describe.

But, I also 'commit' the types to the files they are defined in.

I wonder if commiting the types would make a difference in the memory behavior you are seeing?

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Xavier, Matthew" <Matthew.Xavier@mts.com<mailto:Matthew.Xavier@mts.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, March 28, 2016 2:15 PM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] H5T module 'leaks' conversion paths

Some of the datasets I have contain compound types with vlen fields. When I read these datasets, HDF5 creates new conversion ‘paths’ to convert between the file types and memory types involved. HDF5 caches these paths (see struct H5T_g defined in H5T.c).

I’ve finally traced a memory ‘leak’ in my application to the unbounded growth of the conversion path cache. HDF5 treats vlen types as different if they come from different files, so I get a new set of conversion paths for every file I open, even if the types are actually identical.

That would be fine, except that I can’t find a way to get rid of the cached paths when I close a file. There is a function provided for removing paths, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work for compound types because of the way that the pers parameter is handled. If I pass H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to false by H5T_path_find() when it falls back on the compound->compound soft conversion and generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister() removes the default compound->compound soft conversion and I can't read any more datasets because the library can't create conversion paths for them.

Incidentally, I also discovered that the way the type comparison function determines file identity depends on pointers that are left dangling when a file is closed, which both complicated my minimum reproduction of the problem and also undermines the file identity check. (When the same allocation is re-used from the free list for a different file, types can compare as the same even when they are from different files, which is contrary to the intent of the code.)

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

Has anyone else encountered and/or found a solution for this problem? I am already patching my own HDF5 builds to get Unicode file name support on Windows, so if I have to make code changes it’s not the end of the world.

Thanks,
Matthew Xavier

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://secure-web.cisco.com/1J6zNvZQnyPUOQpf6FEcfBYe4wbtvO7U15v0KgX2eW3zNTPmxSwhCWIEmcdCuA4-BJGy8XzcQ3YLNEp8_hhppyfHS4JilXl5TxHvWfnyAdwkFR3MkhEjbRxtv-jFfwK18M9kXUXAgVdEnTaNN8BXC4XXS05GUHTOtL8uCF9v_SyBs8xOOnZAj0qOxEly5O8DHbESzEKHDHxGauJcO02Nf_JPnQhS_8t2rd8EgHt7Az9SD0XrM4Q-8Vif7WQdjYgHmIqBX5D2SFe6UFAXodrM-axV1spucYc0UlOHQ_CvoTdibT8Te50r1kTz-8i_rcvRzssJz6EeRcnmKz68iz66JzvsdLEOep90Fgxu3lcYIi50/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org<http://secure-web.cisco.com/1ndLYQWFYgp2-f9J3bVFT-kZpCTQ22GIndSxF67r7pL2a9jtfht1OlwKeuYHBpsUxF6mPYTqCT6T3dJ3oVlFDuBLmxvH65fXNcqkyzwhoNUDTN4myqm0DnX_6BrAkqf5mfW6wFMwwiJKRMf7N1JFo0hWbCvRSLGQ1ZsicJBXfzZWeeyN8t8WUT5NT89CH6ekB4vBiOXlK3KyaejIBEOv34pSqxh9ukxtSFSjruKvHdPHCYkwqKKhCZTITpIVnbFFANiSxumo2qyQJB2A4x6zXyUv2Kkqn193hPRFXnmAt5U9UKLVn7oToynZkVCrCWk_J39FfERuhJClBPlyXbr2DuxSpfc8VRIrZRSL-PWupN4w/http%3A%2F%2Flists.hdfgroup.org%2Fmailman%2Flistinfo%2Fhdf-forum_lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5

miller86 · March 17, 2021, 6:36pm

This appears to still be a problem. I have used valgrind but am using the client request VALGRIND_DO_LEAK_CHECK just prior to return from main now rather than waiting for exit. And, I am seeing something like this…

==44479==
==44479== 46,624 bytes in 124 blocks are still reachable in loss record 4,235 of 4,235
==44479== at 0x4C29F73: malloc (vg_replace_malloc.c:307)
==44479== by 0x5901452: H5MM_malloc (H5MM.c:292)
==44479== by 0x5806C79: H5FL_malloc (H5FL.c:243)
==44479== by 0x58073F8: H5FL_reg_malloc (H5FL.c:441)
==44479== by 0x5ADD3C8: H5T__initiate_copy (H5T.c:3343)
==44479== by 0x5ADEABE: H5T_copy (H5T.c:3642)
==44479== by 0x5AFC0CA: H5T_conv_struct_init (H5Tconv.c:1975)
==44479== by 0x5AFDCC7: H5T__conv_struct_opt (H5Tconv.c:2392)
==44479== by 0x5AE43FF: H5T__path_find_real (H5T.c:5010)
==44479== by 0x5AE33EB: H5T_path_find (H5T.c:4814)
==44479== by 0x56138FD: H5A__write (H5Aint.c:735)
==44479== by 0x5C38579: H5VL__native_attr_write (H5VLnative_attr.c:205)
==44479==
==44479== LEAK SUMMARY:
==44479== definitely lost: 0 bytes in 0 blocks
==44479== indirectly lost: 0 bytes in 0 blocks
==44479== possibly lost: 0 bytes in 0 blocks
==44479== still reachable: 1,417,857 bytes in 8,242 blocks
==44479== suppressed: 0 bytes in 0 blocks
=

I am seeing same or similar on 1.8. 1.10 and 1.12. Does anyone know of a way to force HDF5 to clean up some of the memory on a periodic basis? Am I maybe missing some important call to free up this memory. I am not calling H5close(). If I do call H5close(), I do get rid of all these “leaks” but is calling that on a regular basis a good idea? And, I am not even sure I know for sure when I can safely call H5close() or not without causing problems elsewhere. Is there any issue with occasionally calling H5close() (at suitably chosen points)?

miller86 · March 17, 2021, 6:53pm

FYI…, a call to H5garbage_collect() just prior to checking for leaks has an insigificant improvement on the situation.

miller86 · March 17, 2021, 7:13pm

FYI…, calls to H5set_free_list_limits() using 1024 bytes and 10*1024 bytes for local and global values also has no siginficant effect on behavior here.

matthew.xavier · March 17, 2021, 9:33pm

It’s funny to see this topic turn up again. It has been a few years since I was maintainer of the relevant code.

About a year ago the current maintainer asked for my help to re-evaluate the H5T_term_interface()/H5T_init() workaround as those (internal) symbols had changed. My reproduction attempt at that time gave:

PS C:\Source\hdf5_1_8-TypePathLeak> .\Hdf5TypePathLeak.exe check.h5
HDF5 version: 1.8.17
          iterations        H5T_g.npaths  heapExceptSpoilers
                  68                 173              564111
                1045                1233             3937827
                2149                2437             7769999
                3323                3715            11820297
                4442                4929            15700563
                5603                6197            19719167
                6737                7411            23566649
                7868                8631            27498613
                8998                9865            31409483
               10121               11101            35326679
               11236               12297            39117131
               12340               13467            42825105
               13455               14713            46774011
               14569               15921            50602499

PS C:\Source\hdf5_1_10_6-TypePathLeak> .\Hdf5TypePathLeak.exe check.h5
HDF5 version: 1.10.6
          iterations        H5T_g.npaths  heapExceptSpoilers
                 335                 421             1364422
                 760                 859             2758488
                1172                1287             4124876
                1597                1727             5521044
                2029                2179             6971776
                2516                2685             8577414
                2999                3173            10125982
                3481                3665            11687202
                3951                4163            13300272
                4429                4647            14836124
                4908                5135            16384676
                5388                5669            18079198
                5843                6137            19564282
                6285                6623            21106420

PS C:\Source\hdf5_1_12_0-TypePathLeak> .\Hdf5TypePathLeak.exe check.h5
HDF5 version: 1.12.0
          iterations        H5T_g.npaths  heapExceptSpoilers
                   3                  97              344184
                 470                 315             1063374
                 905                 325             1096228
                1336                 327             1102802
                1774                 425             1424720
                2251                 437             1464148
                2739                 437             1464148
                3231                 437             1464148
                3723                 437             1464148
                4217                 437             1464148
                4710                 437             1464148
                5200                 437             1464148
                5694                 437             1464148
                6186                 437             1464148
                6661                 437             1464148
                7151                 437             1464148
                7643                 437             1464148
                8134                 437             1464148
                8625                 437             1464148
                9076                 437             1464148
                9510                 437             1464148
                9985                 437             1464148
               10480                 437             1464148
               10974                 437             1464148
               11461                 437             1464148
               11955                 437             1464148
               12448                 437             1464148
               12937                 437             1464148
               13422                 437             1464148

From which I concluded that while the path table still does not get entirely cleared out, there is at least some cap on its growth in 1.12 (and presumably later version). I believe it was sufficient for the new maintainer to remove the workaround.

miller86 · March 17, 2021, 10:53pm

Ok, really good to know this. Thanks for sharing. That 1464148 number is very close to what I am seeing valgrind report on 1.12 which is 1417857,

==102751== 46,624 bytes in 124 blocks are still reachable in loss record 4,235 of 4,235
==102751== at 0x4C29F73: malloc (vg_replace_malloc.c:307)
==102751== by 0x5901452: H5MM_malloc (H5MM.c:292)
==102751== by 0x5806C79: H5FL_malloc (H5FL.c:243)
==102751== by 0x58073F8: H5FL_reg_malloc (H5FL.c:441)
==102751== by 0x5ADD3C8: H5T__initiate_copy (H5T.c:3343)
==102751== by 0x5ADEABE: H5T_copy (H5T.c:3642)
==102751== by 0x5AFC0CA: H5T_conv_struct_init (H5Tconv.c:1975)
==102751== by 0x5AFDCC7: H5T__conv_struct_opt (H5Tconv.c:2392)
==102751== by 0x5AE43FF: H5T__path_find_real (H5T.c:5010)
==102751== by 0x5AE33EB: H5T_path_find (H5T.c:4814)
==102751== by 0x56138FD: H5A__write (H5Aint.c:735)
==102751== by 0x5C38579: H5VL__native_attr_write (H5VLnative_attr.c:205)
==102751==
==102751== LEAK SUMMARY:
==102751== definitely lost: 0 bytes in 0 blocks
==102751== indirectly lost: 0 bytes in 0 blocks
==102751== possibly lost: 0 bytes in 0 blocks
==102751== still reachable: 1,417,857 bytes in 8,242 blocks
==102751== suppressed: 0 bytes in 0 blocks
==102751==
==102751==
==102751== HEAP SUMMARY:
==102751== in use at exit: 0 bytes in 0 blocks
==102751== total heap usage: 18,492 allocs, 18,492 frees, 5,515,220 bytes allocated

epourmal · May 18, 2021, 5:59pm

I entered a JIRA issue HDFFV-11249. THG developers will take a look.

Thank you for reporting and interesting analysis.

Elena

j0rdan23turn3r · August 3, 2021, 5:48pm

Have the same problem described by the author. Therefore, I continue to follow this conversation. I would be very grateful if someone could describe the ways to solve it here. *patiently waiting

keremsur · April 7, 2022, 3:27pm

I’m allocating it to prevent the allocator from giving the same block back to HDF5 on a subsequent request.

vchoi · March 24, 2023, 3:18am

May I have the program (as indicated below) that reproduces the problem?

I have attached a small program that reproduces the problem. It takes one argument, which is a path at which it can write a temporary file. To run it does require a custom build of HDF5 so that the test program can read the size of the path table. (Or alternately, you can comment out the relevant parts of the test program and inspect H5T_g.npaths with a debugger.)

matthew.xavier · March 24, 2023, 3:33pm

I searched my archives and located the original program I sent to the mailing list in 2016. When I try to upload it with a reply, I am told:

… so I am including it inline below, even though I think it is a little long to quote directly.

Please note that I this version was written using HDF5 1.8.x. While my earlier replies to this thread exhibit an improved version of the test program ca. 2020, I do not seem to have kept a copy of that one.

// Hdf5TypePathLeak
// 2016/3/28
//
// This test program demonstrates unbounded growth in the HDF5 type conversion path table when
// reading a file that contains a vlen datatype.
//
// Note that for this program to display the number of paths in the path table, you must add a
// symbol to H5T.c after the declaration of H5T_g to export the address of the allocated paths
// counter:
// H5_DLLVAR int* H5Tnpaths = &H5T_g.npaths;
// In my copy of the source I put it on line #523, but I guess that may vary by version.
//
// The underlying problems are three-fold:
// 1. Type conversion paths for compound and vlen datatypes are cached in a global table, but they
//    are never removed.
// 2. Type conversion attempts to treat vlen datatypes as unique if they are defined in different
//    files.
// 3. The file association of a vlen datatype is saved as a pointer in the type information, which
//    is allowed to dangle when the file is closed.
//
// My application, which is user-interactive, may open hundreds or thousands of hdf5 files during
// the lifetime of the process. Each of these files may contain one or more datasets contianing
// compound types with vlen fields. Because of (1), each time I open a new file and read its
// contents, new entries are added to the type conversion table. (The table contents are tracked by
// the H5T_g static struct defined in H5T.c.)
//
// Because of (2), the conversions for the types in a file go stale as soon as that file is closed.
// This is checked in H5T_cmp() in the following block (H5T.c:4173):
//
// /* Don't allow VL types in different files to compare as equal */
// if (dt1->shared->u.vlen.f < dt2->shared->u.vlen.f)
//    HGOTO_DONE(-1);
// if (dt1->shared->u.vlen.f > dt2->shared->u.vlen.f)
//    HGOTO_DONE(1);
//
// I tried to find a way to manually clear out entries from this table. There is a function
// provided for that purpose, H5Tunregister(pers, name, src_id, dst_id, func), but it does not work
// for compound types because of the way that the pers parameter is handled. If I pass
// H5T_PERS_HARD, no compound type conversions are removed because H5T_path_t.is_hard is set to
// false by H5T_path_find() when it falls back on the compound->compound soft conversion and
// generates a new path. Alternately, if I pass H5T_PERS_DONTCARE or H5T_PERS_SOFT, H5Tunregister()
// removes the default compound->compound soft conversion and I can't read any more datasets
// because the library can't create conversion paths for them.
//
// (1) and (2) together are enough to create unbounded memory growth, but diagnosing the problem is
// made harder by (3). If you consider dt1->shared->u.vlen.f in the code snippet above, the field
// f is a raw H5F_t*. This pointer is assigned in H5T__vlen_set_loc(), which is called when the
// dataset is opened. When the file is closed, the pointer dangles. The dangling pointer doesn't
// cause a crash (at least not in this case), because H5T_cmp() uses the pointer value as an
// identity but does not dereference it.
//
// However, the dangling pointer does make it difficult to diagnose the problem, because you would
// expect a trivial reproduction to look like this:
// A. Open a file with a suitable dataset.
// B. Open the dataset.
// C. Read the dataset (causing the conversion path to be allocated).
// D. Close the datset.
// E. Close the file.
// F. Look at the number of cached conversion paths.
// G. Repeat A-F and watch the number of conversion paths grow.
//
// The trivial reproduction doesn't work because when the file is closed, the H5F_t* is put on a
// free list and re-used for the next opened file. Because the memory allocation was re-used, the
// pointer check in H5T_cmp() passes.
//
// Therefore, to prove the path leak, I need to add two more things:
// * Call H5set_free_list_limits(0, 0, 0, 0, 0, 0) to disable the free lists.
// * Allocate sizeof(H5F_t) after closing the file on each iteration. This prevents the system
//   allocator from giving back the same pointer for subsequent requests, simulating the use of
//   the library in a larger program in which the allocator will be servicing other requests.
//   (On Windows, at least, malloc() is backed by the default process heap.)

#include <stdio.h>
#include <inttypes.h>
#include <signal.h>
#include <time.h>

#define H5_BUILT_AS_DYNAMIC_LIB 1
#include <hdf5.h>

// This must be added to the library to export the size of the path table; see above comments.
H5_DLLVAR int* H5Tnpaths;

// I just need this defined to get its size.
struct H5F_t {
    char *open_name;
    char *actual_name;
    char *extpath;
    void *shared;
    unsigned nopen_objs;
    void *obj_count;
    hid_t file_id;
    hbool_t closing;
    struct H5F_t *parent;
    unsigned nmounts;
};

void WriteFile(char *filename, char *datasetname, char *fieldname, hvl_t *data)
{
    hid_t file = H5Fcreate(filename, H5F_ACC_EXCL, H5P_DEFAULT, H5P_DEFAULT);

    hid_t vlenFileType = H5Tvlen_create(H5T_STD_I64LE);
    hid_t vlenMemoryType = H5Tvlen_create(H5T_NATIVE_INT64);

    hid_t compoundFileType = H5Tcreate(H5T_COMPOUND, sizeof(hvl_t));
    H5Tinsert(compoundFileType, fieldname, 0, vlenFileType);

    hid_t compoundMemoryType = H5Tcreate(H5T_COMPOUND, sizeof(hvl_t));
    H5Tinsert(compoundMemoryType, fieldname, 0, vlenMemoryType);

    hsize_t dimensions[1] = { 1 };
    hid_t space = H5Screate_simple(1, dimensions, dimensions);
    hid_t dataset = H5Dcreate(
        file,
        datasetname,
        compoundFileType,
        space,
        H5P_DEFAULT,
        H5P_DEFAULT,
        H5P_DEFAULT);
    H5Sselect_all(space);

    H5Dwrite(dataset,
        compoundMemoryType,
        space,
        space,
        H5P_DEFAULT,
        data);

    H5Dclose(dataset);
    H5Sclose(space);
    H5Tclose(compoundMemoryType);
    H5Tclose(compoundFileType);
    H5Tclose(vlenMemoryType);
    H5Tclose(vlenFileType);
    H5Fclose(file);
}

void CheckRead(hid_t dataset, char *fieldname, hvl_t* written)
{
    hid_t space = H5Dget_space(dataset);
    H5Sselect_all(space);

    hid_t vlenMemoryType = H5Tvlen_create(H5T_NATIVE_INT64);
    hid_t compoundMemoryType = H5Tcreate(H5T_COMPOUND, sizeof(hvl_t));
    H5Tinsert(compoundMemoryType, fieldname, 0, vlenMemoryType);

    hvl_t read;
    H5Dread(dataset, compoundMemoryType, space, space, H5P_DEFAULT, &read);
    if (written->len != read.len)
    {
        printf(
            "Wrote %" PRIu32 " integers; read %" PRIu32 ".\n",
            written->len,
            read.len);
    }
    for (size_t i = 0; i < read.len; ++i)
    {
        int64_t expected = ((int64_t*)written->p)[i];
        int64_t found = ((int64_t*)read.p)[i];
        if (expected != found)
        {
            printf(
                "Wrote %" PRId64 " at index %" PRIu32 "; read %" PRId64 ".\n",
                expected,
                i,
                found);
        }
    }

    H5Dvlen_reclaim(compoundMemoryType, space, H5P_DEFAULT, &read);
    H5Tclose(compoundMemoryType);
    H5Tclose(vlenMemoryType);
    H5Sclose(space);
}

// Let the main loop terminate cleanly when the program is interrupted.
volatile sig_atomic_t terminated;
void SignalHandler(int signo) { terminated = 1; }

int main(int argc, char *argv[])
{
    signal(SIGINT, SignalHandler);
    H5set_free_list_limits(0, 0, 0, 0, 0, 0);

    char *filename = argv[1];
    remove(filename);

    char *datasetname = "compound";
    char *fieldname = "integers";

    int64_t integers[10] = { 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 };

    hvl_t written;
    written.p = integers;
    written.len = 10;
    WriteFile(filename, datasetname, fieldname, &written);

    time_t then, now;
    time(&then);

    while (!terminated)
    {
        hid_t file = H5Fopen(filename, H5F_ACC_RDWR, H5P_DEFAULT);
        hid_t dataset = H5Dopen(file, datasetname, H5P_DEFAULT);
        CheckRead(dataset, fieldname, &written);
        H5Dclose(dataset);
        H5Fclose(file);
        H5allocate_memory(sizeof(struct H5F_t), 0);

        // I want to rate limit this to 1 Hz, otherwise it just spams the console.
        time(&now);
        if (now != then)
        {
            printf("Type conversion path table size: %i\n", *H5Tnpaths);
            then = now;
        }
    }

    return 0;
}