memory usage in H5Dwrite

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

Hi Nichols,

···

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

hdf5_4D.tar (362 KB)

···

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Nichols,

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

  Yes, it doesn't look like memory should be leaked, given your usage. Can you push the leak analysis further down inside HDF5, to see if it's in the MPI library, or actually in HDF5?

  Thanks,
    Quincey

···

On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<hdf5_4D.tar>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I have investigated further, its definitely an MPI issue. Thanks for taking
a look at this.

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

···

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Thursday, August 25, 2011 10:14:38 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

  Yes, it doesn't look like memory should be leaked, given your usage. Can you push the leak analysis further down inside HDF5, to see if it's in the MPI library, or actually in HDF5?

  Thanks,
    Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<hdf5_4D.tar>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I have investigated further, its definitely an MPI issue. Thanks for taking
a look at this.

  Great, happy to help out. BTW, which MPI implementation has the memory leak issue?

  Quincey

···

On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote:

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Thursday, August 25, 2011 10:14:38 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

  Yes, it doesn't look like memory should be leaked, given your usage. Can you push the leak analysis further down inside HDF5, to see if it's in the MPI library, or actually in HDF5?

  Thanks,
    Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<hdf5_4D.tar>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

It is MPICH, but its not a memory leak (technically speaking).

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

···

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Monday, August 29, 2011 10:47:00 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote:

I have investigated further, its definitely an MPI issue. Thanks for taking
a look at this.

  Great, happy to help out. BTW, which MPI implementation has the memory leak issue?

  Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Thursday, August 25, 2011 10:14:38 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

  Yes, it doesn't look like memory should be leaked, given your usage. Can you push the leak analysis further down inside HDF5, to see if it's in the MPI library, or actually in HDF5?

  Thanks,
    Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<hdf5_4D.tar>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

It is MPICH, but its not a memory leak (technically speaking).

  Ah, something in the library that is only freed at MPI_Finalize?

    Quincey

···

On Aug 29, 2011, at 3:02 PM, Nichols A. Romero wrote:

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Monday, August 29, 2011 10:47:00 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote:

I have investigated further, its definitely an MPI issue. Thanks for taking
a look at this.

  Great, happy to help out. BTW, which MPI implementation has the memory leak issue?

  Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Thursday, August 25, 2011 10:14:38 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

  Yes, it doesn't look like memory should be leaked, given your usage. Can you push the leak analysis further down inside HDF5, to see if it's in the MPI library, or actually in HDF5?

  Thanks,
    Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<hdf5_4D.tar>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Not really... there are two things which are causing problems.

1. glibc malloc on BG/P does not release memory if it goes into the arena, the
threshold between mmap and arena is different on BG/P that on Linux

2. 4D array is being sliced into 4D chunks which leads to a large offsets in
the ROMIO driver (this is somehow problematic for memory, don't know the
details here, someone at ANL from the MPICH will look into this).

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

···

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Monday, August 29, 2011 3:11:55 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

On Aug 29, 2011, at 3:02 PM, Nichols A. Romero wrote:

It is MPICH, but its not a memory leak (technically speaking).

  Ah, something in the library that is only freed at MPI_Finalize?

    Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Monday, August 29, 2011 10:47:00 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

On Aug 28, 2011, at 1:30 PM, Nichols A. Romero wrote:

I have investigated further, its definitely an MPI issue. Thanks for taking
a look at this.

  Great, happy to help out. BTW, which MPI implementation has the memory leak issue?

  Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Thursday, August 25, 2011 10:14:38 AM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Aug 18, 2011, at 10:43 AM, Nichols A. Romero wrote:

This is the test case that I sent to IBM for analysis. They assert that there
is no memory leaked according to mallinfo. However, mmap/munmap are not recorded
to mallinfo (AFAIK). If I analyze the sequence of mmap/munmap and translate the
addresses, then I see heap memory (O(100 MB)) which is not freed.

  Yes, it doesn't look like memory should be leaked, given your usage. Can you push the leak analysis further down inside HDF5, to see if it's in the MPI library, or actually in HDF5?

  Thanks,
    Quincey

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

----- Original Message -----
From: "Quincey Koziol" <koziol@hdfgroup.org>
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Sent: Tuesday, August 16, 2011 2:49:05 PM
Subject: Re: [Hdf-forum] memory usage in H5Dwrite

Hi Nichols,

On Jul 28, 2011, at 5:44 PM, Nichols A. Romero wrote:

Hi,

I am trying to figure out if there is a memory leak or a lack of understanding on my part
with respect to the HDF5 internals. My simple test case writes a 4-dimensional uniformly
distributed array using H5Dwrite. I measure the heap memory before and after the write.
I would expect the heap memory utilization to not change substantially.

I have measured the heap memory a couple of different ways, e.g. using mallinfo or some tool.
Naively it looks like the MPI I/O aggregators are not releasing memory but some tools
have reported no leak at all, while others have report a large one (> 200 MB). I finally
resorted to using mpirun -strace to track the series of mmap/munmap in the H5Dwrite.
I was surprised to see a large number of calls to mmap/munmap which ultimately results
in 60 MB of heap memory not being released.

First question: Is this the intended behavior?

Second question: If this behavior is intentional, is there away to force HDF5 to release
these internal buffers after the H5Dwrite.

  It's a bit hard to tell from your description - could you send a short program that demonstrates the issue?

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<hdf5_4D.tar>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org