H5FL_malloc calls

Hdf-forum archives
1

Hi, this is my first post to hdf-forum so apologies in advance for anything
that is redundant that I've missed in previous posts. I have a process that
is running for several hours opening and reading from many hdf files. The
heap memory is constantly growing, and because all memory is returned upon
exit, I suspect it is from heap fragmentation.

Running my code using valgrind and massif to profile heap usage, it shows a
large amount of memory allocated simply for getNumAttrs. Furthermore, many
calls to read data are also accompanied by malloc's. When I run similar
code that reads a txt version of the input, the memory does not climb as
high.

Are there any settings to get hdf to reuse buffers, or perform fewer
mallocs?

thanks,
-mark

->46.33% (21,936,989B) 0x515FA6B: H5FL_malloc (H5FL.c:203)

->43.55% (20,622,405B) 0x5160049: H5FL_blk_malloc (H5FL.c:889)
> ->38.49% (18,226,752B) 0x50FA556: H5B_load (H5Bcache.c:130)
> > ->38.49% (18,226,752B) 0x510D418: H5C_protect (H5C.c:8175)
> > ->38.49% (18,226,752B) 0x50F4CFF: H5AC_protect (H5AC.c:1587)
> > ->38.37% (18,171,584B) 0x50F6407: H5B_get_info_helper (H5B.c:1989)
> > > ->38.37% (18,171,584B) 0x50F65DF: H5B_get_info_helper

(H5B.c:2008)

> > > ->37.53% (17,771,008B) 0x50F65DF: H5B_get_info_helper

(H5B.c:2008)

> > > > ->37.53% (17,771,008B) 0x50F7397: H5B_get_info (H5B.c:2058)
> > > > ->37.53% (17,771,008B) 0x51157C3: H5D_btree_idx_size

(H5Dbtree.c:1321)

> > > > ->37.53% (17,771,008B) 0x5115E97: H5D_chunk_bh_info

(H5Dchunk.c:4702)

> > > > ->37.53% (17,771,008B) 0x512D465: H5O_dset_bh_info

(H5Doh.c:404)

> > > > ->37.53% (17,771,008B) 0x51B8C63: H5O_get_info

(H5O.c:2827)

> > > > ->37.53% (17,771,008B) 0x5177CB1: H5G_loc_info_cb

(H5Gloc.c:750)

> > > > ->37.53% (17,771,008B) 0x5184541:

H5G_traverse_real (H5Gtraverse.c:886)

> > > > ->37.53% (17,771,008B) 0x5184ABB: H5G_traverse

(H5Gtraverse.c:963)

> > > > ->37.53% (17,771,008B) 0x51777AA:

H5G_loc_info (H5Gloc.c:795)

> > > > ->37.53% (17,771,008B) 0x51BCBC3:

H5Oget_info (H5O.c:612)

> > > > ->37.53% (17,771,008B) 0x4E4F317:

H5::H5Object::getNumAttrs() const (H5Object.cpp:248)

2

Hi Mark,

Hi, this is my first post to hdf-forum so apologies in advance for anything that is redundant that I've missed in previous posts. I have a process that is running for several hours opening and reading from many hdf files. The heap memory is constantly growing, and because all memory is returned upon exit, I suspect it is from heap fragmentation.

Running my code using valgrind and massif to profile heap usage, it shows a large amount of memory allocated simply for getNumAttrs. Furthermore, many calls to read data are also accompanied by malloc's. When I run similar code that reads a txt version of the input, the memory does not climb as high.

Are there any settings to get hdf to reuse buffers, or perform fewer mallocs?

  Which version of HDF5 are you using? If you'd like to use valgrind to analyze the memory use in the HDF5 library, you may want to use the "--enable-using-memchecker" option for configure, to avoid the internal free space memory allocator and call malloc() directly all the time.

  Quincey

···

On Dec 3, 2010, at 4:00 PM, Mark Chaisson wrote:

thanks,
-mark

->46.33% (21,936,989B) 0x515FA6B: H5FL_malloc (H5FL.c:203)
> ->43.55% (20,622,405B) 0x5160049: H5FL_blk_malloc (H5FL.c:889)
> > ->38.49% (18,226,752B) 0x50FA556: H5B_load (H5Bcache.c:130)
> > > ->38.49% (18,226,752B) 0x510D418: H5C_protect (H5C.c:8175)
> > > ->38.49% (18,226,752B) 0x50F4CFF: H5AC_protect (H5AC.c:1587)
> > > ->38.37% (18,171,584B) 0x50F6407: H5B_get_info_helper (H5B.c:1989)
> > > > ->38.37% (18,171,584B) 0x50F65DF: H5B_get_info_helper (H5B.c:2008)
> > > > ->37.53% (17,771,008B) 0x50F65DF: H5B_get_info_helper (H5B.c:2008)
> > > > > ->37.53% (17,771,008B) 0x50F7397: H5B_get_info (H5B.c:2058)
> > > > > ->37.53% (17,771,008B) 0x51157C3: H5D_btree_idx_size (H5Dbtree.c:1321)
> > > > > ->37.53% (17,771,008B) 0x5115E97: H5D_chunk_bh_info (H5Dchunk.c:4702)
> > > > > ->37.53% (17,771,008B) 0x512D465: H5O_dset_bh_info (H5Doh.c:404)
> > > > > ->37.53% (17,771,008B) 0x51B8C63: H5O_get_info (H5O.c:2827)
> > > > > ->37.53% (17,771,008B) 0x5177CB1: H5G_loc_info_cb (H5Gloc.c:750)
> > > > > ->37.53% (17,771,008B) 0x5184541: H5G_traverse_real (H5Gtraverse.c:886)
> > > > > ->37.53% (17,771,008B) 0x5184ABB: H5G_traverse (H5Gtraverse.c:963)
> > > > > ->37.53% (17,771,008B) 0x51777AA: H5G_loc_info (H5Gloc.c:795)
> > > > > ->37.53% (17,771,008B) 0x51BCBC3: H5Oget_info (H5O.c:612)
> > > > > ->37.53% (17,771,008B) 0x4E4F317: H5::H5Object::getNumAttrs() const (H5Object.cpp:248)

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org