# loading chunks or even the whole hdf5 file into memory

**URL:** <https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637>\
**Category:** Hdf-forum archives\
**Created:** [March 3, 2010, 10:47am UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637 "2010-03-03T10:47:18Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 10:47am UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/1 "2010-03-03T10:47:18Z")

</div>

Hello,  
We have a ~30GB HDF5 file with something like 100 million small datasets in  
it. We need to iterate through all of them, and doing so is very slow as each  
one has to be loaded from disk. I also don't know if it is possible to find  
out a proper ordering to go through them, so I suspect that there might also  
be a lot of disk seeks necessary.

Maybe it isn't such a good idea to have so many small objects in the file, but  
I'm stuck with this format now. What options do I have?

I'm working now on a machine with 128GB of RAM, so my file would fit  
comfortably inside. Is it possible to load the file completely into memory to  
avoid all of the above problems?

Thanks,  
Thorben

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 12:18pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/2 "2010-03-03T12:18:02Z")

</div>

On a related note, I've just found this piece of information which might  
accelerate our program as well:

> **···**
>
> ---
> 
> [http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html](http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html)
> 
> H5Pset\_libver\_bounds( hid\_t fapl\_id, H5F\_libver\_t low, H5F\_libver\_t high )
> 
> Compact-or-indexed groups enable much-compressed link storage for groups with  
> very few members and improved efficiency and performance for groups with very  
> large numbers of members. The efficiency and performance impacts are most  
> noticeable at the extremes: all unnecessary overhead is eliminated for groups  
> with zero members; groups with tens of thousands of members may see as much  
> as a 100-fold performance gain.
> 
> H5Pset\_libver\_bounds( hid\_t fapl\_id, H5F\_libver\_t low, H5F\_libver\_t high )  
> H5Pget\_libver\_bounds( hid\_t fapl\_id, H5F\_libver\_t\* low, H5F\_libver\_t\* high )
> 
> Default behavior: If H5Pset\_libver\_bounds is not called with low equal to  
> HDF\_LIBVER\_LATEST, then the HDF5 Library provides the greatest-possible  
> format compatibility. It does this by creating objects with the earliest  
> opssible format that will handle the data being stored and accommodate  
> the action being taken.
> 
> ---
> 
> Though the 30GB file I was talking of was written using HDF5 1.8.4, if I  
> understand correctly, it will not make use of these new features because it  
> tries to maintain downward compatibility to 1.6. Correct?
> 
> Is there a tool available that converts an existing file to a new file version  
> to make use of all these performance improvements? Or should I hack this into  
> h5repack.c myself? I'd like to avoid that...
> 
> Cheers,  
> Thorben
> 
> On Wednesday 03 March 2010 11:47:18 Thorben Kröger wrote:
> 
> > Hello,  
> > We have a ~30GB HDF5 file with something like 100 million small datasets in  
> > it. We need to iterate through all of them, and doing so is very slow as  
> > each one has to be loaded from disk. I also don't know if it is possible  
> > to find out a proper ordering to go through them, so I suspect that there  
> > might also be a lot of disk seeks necessary.
> > 
> > Maybe it isn't such a good idea to have so many small objects in the file,  
> > but I'm stuck with this format now. What options do I have?
> > 
> > I'm working now on a machine with 128GB of RAM, so my file would fit  
> > comfortably inside. Is it possible to load the file completely into memory  
> > to avoid all of the above problems?
> > 
> > Thanks,  
> > Thorben
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)

---

<div class="post-metadata">

**Author:** ![werner](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/w/f4b2a3/32.png) [@werner](https://forum.hdfgroup.org/u/werner)\
**Post date:** [March 3, 2010, 12:43pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/3 "2010-03-03T12:43:38Z")

</div>

Yes, having many small objects is inefficient. H5iterate() just traverses  
all objects as found by name, which is alphabetical ordering, which may or  
may not be what you want. If you want a particular order, you could create  
a table with references/links to the specific datasets, and iterate over  
this table instead.

Are you using Linux as operating system? If so, then you can just copy  
your 30GB file to

&nbsp;&nbsp;/dev/shm

which (usually) is a tmpfs, i.e. a ramdisk under Linux.

A more portable solution is to use the memory virtual file driver,  
where you can read an HDF5 file from an RAM image.

&nbsp;&nbsp;Werner

> **···**
>
> On Wed, 03 Mar 2010 11:47:18 +0100, Thorben Kröger \<thorben.kroeger@iwr.uni-heidelberg.de\> wrote:
> 
> > Hello,  
> > We have a ~30GB HDF5 file with something like 100 million small datasets in  
> > it. We need to iterate through all of them, and doing so is very slow as each  
> > one has to be loaded from disk. I also don't know if it is possible to find  
> > out a proper ordering to go through them, so I suspect that there might also  
> > be a lot of disk seeks necessary.
> > 
> > Maybe it isn't such a good idea to have so many small objects in the file, but  
> > I'm stuck with this format now. What options do I have?
> > 
> > I'm working now on a machine with 128GB of RAM, so my file would fit  
> > comfortably inside. Is it possible to load the file completely into memory to  
> > avoid all of the above problems?
> 
> --  
> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> Dr. Werner Benger Visualization Research  
> Laboratory for Creative Arts and Technology (LCAT)  
> Center for Computation & Technology at Louisiana State University (CCT/LSU)  
> 211 Johnston Hall, Baton Rouge, Louisiana 70803  
> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 1:03pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/4 "2010-03-03T13:03:16Z")

</div>

> \> Hello,  
> \> We have a ~30GB HDF5 file with something like 100 million small datasets  
> \> in it. We need to iterate through all of them, and doing so is very slow  
> \> as each one has to be loaded from disk. I also don't know if it is  
> \> possible to find out a proper ordering to go through them, so I suspect  
> \> that there might also be a lot of disk seeks necessary.  
> \>  
> \> Maybe it isn't such a good idea to have so many small objects in the  
> \> file, but I'm stuck with this format now. What options do I have?  
> \>  
> \> I'm working now on a machine with 128GB of RAM, so my file would fit  
> \> comfortably inside. Is it possible to load the file completely into  
> \> memory to avoid all of the above problems?
> 
> Yes, having many small objects is inefficient. H5iterate() just traverses  
> all objects as found by name, which is alphabetical ordering, which may or  
> may not be what you want. If you want a particular order, you could create  
> a table with references/links to the specific datasets, and iterate over  
> this table instead.
> 
> Are you using Linux as operating system? If so, then you can just copy  
> your 30GB file to
> 
> &nbsp;&nbsp;/dev/shm
> 
> which (usually) is a tmpfs, i.e. a ramdisk under Linux.

I also thought of using a ramdisk, but thought that you needed to have root  
rights for that. As it turns out, I can copy stuff to /dev/shm 🙂  
Thanks, as a temporary solution this is really good 🙂

> **···**
>
> On Wednesday 03 March 2010 13:43:38 Werner Benger wrote:
> 
> > On Wed, 03 Mar 2010 11:47:18 +0100, Thorben Kröger \<thorben.kroeger@iwr.uni- heidelberg.de\> wrote:
> 
> > A more portable solution is to use the memory virtual file driver,  
> > where you can read an HDF5 file from an RAM image.
> > 
> > &nbsp;&nbsp;Werner

---

<div class="post-metadata">

**Author:** ![Francesc\_Alted2](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/bbe5ce/32.png) [@Francesc\_Alted2](https://forum.hdfgroup.org/u/Francesc_Alted2)\
**Post date:** [March 3, 2010, 1:19pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/5 "2010-03-03T13:19:45Z")

</div>

A Wednesday 03 March 2010 13:18:02 Thorben Kröger escrigué:

> On a related note, I've just found this piece of information which might  
> accelerate our program as well:
> 
> ---
> 
> [http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html](http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html)
> 
> H5Pset\_libver\_bounds( hid\_t fapl\_id, H5F\_libver\_t low, H5F\_libver\_t high )
> 
> Compact-or-indexed groups enable much-compressed link storage for groups  
> with very few members and improved efficiency and performance for groups  
> with very large numbers of members. The efficiency and performance impacts  
> are most noticeable at the extremes: all unnecessary overhead is  
> eliminated for groups with zero members; groups with tens of thousands of  
> members may see as much as a 100-fold performance gain.
> 
> H5Pset\_libver\_bounds( hid\_t fapl\_id, H5F\_libver\_t low, H5F\_libver\_t high )  
> H5Pget\_libver\_bounds( hid\_t fapl\_id, H5F\_libver\_t\* low, H5F\_libver\_t\* high  
> )
> 
> Default behavior: If H5Pset\_libver\_bounds is not called with low equal to  
> HDF\_LIBVER\_LATEST, then the HDF5 Library provides the greatest-possible  
> format compatibility. It does this by creating objects with the earliest  
> opssible format that will handle the data being stored and accommodate  
> the action being taken.
> 
> ---
> 
> Though the 30GB file I was talking of was written using HDF5 1.8.4, if I  
> understand correctly, it will not make use of these new features because it  
> tries to maintain downward compatibility to 1.6. Correct?

Yes, you should enable the new link storage by calling H5Pset\_libver\_bounds()  
explicitly. IMO, this would be the best path to improve performance for your  
scenario (except if you don't want to follow the safer path of creating large  
datasets and references to parts of them, as Werner suggested).

> **···**
>
> --  
> Francesc Alted

---

<div class="post-metadata">

**Author:** ![Francesc\_Alted2](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/f/bbe5ce/32.png) [@Francesc\_Alted2](https://forum.hdfgroup.org/u/Francesc_Alted2)\
**Post date:** [March 3, 2010, 1:14pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/6 "2010-03-03T13:14:44Z")

</div>

A Wednesday 03 March 2010 14:03:16 Thorben Kröger escrigué:

> \> Are you using Linux as operating system? If so, then you can just copy  
> \> your 30GB file to  
> \>  
> \> /dev/shm  
> \>  
> \> which (usually) is a tmpfs, i.e. a ramdisk under Linux.
> 
> I also thought of using a ramdisk, but thought that you needed to have root  
> rights for that. As it turns out, I can copy stuff to /dev/shm 🙂  
> Thanks, as a temporary solution this is really good 🙂

I don't think you are going to have a decent speed-up by just copying to  
/dev/shm because your 30 GB file probably was probably completely in your OS  
filesystem cache already, which is memory too. Having said this, in case you  
get some noticeable speed-up please tell us: I like to hear how wrong I am 😉

> **···**
>
> --  
> Francesc Alted

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 1:25pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/7 "2010-03-03T13:25:17Z")

</div>

> \> Though the 30GB file I was talking of was written using HDF5 1.8.4, if I  
> \> understand correctly, it will not make use of these new features because  
> \> it tries to maintain downward compatibility to 1.6. Correct?
> 
> Yes, you should enable the new link storage by calling  
> H5Pset\_libver\_bounds() explicitly. IMO, this would be the best path to  
> improve performance for your scenario

So can I convert my existing file to the 1.8-only file version without having  
to rerun the program that generated it? That would take a week I think ☹

> (except if you don't want to follow  
> the safer path of creating large datasets and references to parts of them,  
> as Werner suggested).

I would, but I do not want to rerun the program that generated the big file,  
and don't want to write a conversion tool atm. either, so I'm stuck with the  
data layout...

Cheers,  
Thorben

---

<div class="post-metadata">

**Author:** ![Cyril\_Giraudon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@Cyril\_Giraudon](https://forum.hdfgroup.org/u/Cyril_Giraudon)\
**Post date:** [March 11, 2010, 11:23am UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/8 "2010-03-11T11:23:30Z")

</div>

Hello,

I'd like to create a null dataspace dataset in java.

For this, the java doc  
[http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S\_NULL](http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S_NULL)  
mentions the HS5\_NULL constant but this  
constant is missing in the source file  
(hdf-java/ncsa/hdf/hdf5lib/HDF5Constants.java).

Is there another way to create a null dataspace in java ?

Best regards (thanks a lot for this awesome HDF5 format / library),

Cyril.

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 1:27pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/9 "2010-03-03T13:27:40Z")

</div>

> I don't think you are going to have a decent speed-up by just copying to  
> /dev/shm because your 30 GB file probably was probably completely in your  
> OS filesystem cache already, which is memory too. Having said this, in  
> case you get some noticeable speed-up please tell us: I like to hear how  
> wrong I am 😉

I'm going to have to disappoint you then:  
I just tried it, and as you said, there really is \_no\_ speed difference at  
all.

So I guess my only option now is to try to get my existing file over to the  
new 1.8-only file format.

Cheers,  
Thorben

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 2:13pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/10 "2010-03-03T14:13:34Z")

</div>

So, I've found the "h5repack" command line tool now that accepts the "--  
latest" options, and am running this now on my file like this:

h5repack --latest oldfile.h5 newfile.h5

Will report back about possible performance improvements...

Cheers,  
Thorben

> **···**
>
> On Wednesday 03 March 2010 14:25:17 Thorben Kröger wrote:
> 
> > \> \> Though the 30GB file I was talking of was written using HDF5 1.8.4, if  
> > \> \> I understand correctly, it will not make use of these new features  
> > \> \> because it tries to maintain downward compatibility to 1.6. Correct?  
> > \>  
> > \> Yes, you should enable the new link storage by calling  
> > \> H5Pset\_libver\_bounds() explicitly. IMO, this would be the best path to  
> > \> improve performance for your scenario
> > 
> > So can I convert my existing file to the 1.8-only file version without  
> > having to rerun the program that generated it? That would take a week I  
> > think ☹

---

<div class="post-metadata">

**Author:** ![Peter\_Cao](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/p/e68b1a/32.png) [@Peter\_Cao](https://forum.hdfgroup.org/u/Peter_Cao)\
**Post date:** [March 11, 2010, 2:01pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/11 "2010-03-11T14:01:53Z")

</div>

The current hdf-java release does not support any HDF5 1.8 features (including constants).  
We are working on supporting HDF5 1.8 in hdf-java. H5S\_NULL will be in our future release.

Thanks  
--pc

Giraudon Cyril wrote:

> **···**
>
> > Hello,
> > 
> > I'd like to create a null dataspace dataset in java.
> > 
> > For this, the java doc  
> > [http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S\_NULL](http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S_NULL)  
> > mentions the HS5\_NULL constant but this  
> > constant is missing in the source file  
> > (hdf-java/ncsa/hdf/hdf5lib/HDF5Constants.java).
> > 
> > Is there another way to create a null dataspace in java ?
> > 
> > Best regards (thanks a lot for this awesome HDF5 format / library),
> > 
> > Cyril.
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)

---

<div class="post-metadata">

**Author:** ![Cyril\_Giraudon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/c/ad7895/32.png) [@Cyril\_Giraudon](https://forum.hdfgroup.org/u/Cyril_Giraudon)\
**Post date:** [March 11, 2010, 4:26pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/12 "2010-03-11T16:26:49Z")

</div>

Thanks Peter for your response.

Le 11/03/2010 15:01, Peter Cao a �crit :

> **···**
>
> > The current hdf-java release does not support any HDF5 1.8 features  
> > (including constants).  
> > We are working on supporting HDF5 1.8 in hdf-java. H5S\_NULL will be in  
> > our future release.
> > 
> > Thanks  
> > --pc
> > 
> > Giraudon Cyril wrote:
> > 
> > > Hello,
> > > 
> > > I'd like to create a null dataspace dataset in java.
> > > 
> > > For this, the java doc  
> > > [http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S\_NULL](http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S_NULL)
> > > 
> > > mentions the HS5\_NULL constant but this  
> > > constant is missing in the source file  
> > > (hdf-java/ncsa/hdf/hdf5lib/HDF5Constants.java).
> > > 
> > > Is there another way to create a null dataspace in java ?
> > > 
> > > Best regards (thanks a lot for this awesome HDF5 format / library),
> > > 
> > > Cyril.
> > > 
> > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > > Hdf-forum is for HDF software users discussion.  
> > > Hdf-forum@hdfgroup.org  
> > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)

---

<div class="post-metadata">

**Author:** ![miller86](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/miller86/32/124_2.png) [@miller86](https://forum.hdfgroup.org/u/miller86)\
**Post date:** [March 3, 2010, 3:47pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/13 "2010-03-03T15:47:20Z")

</div>

Hello Thorben,

So, I am not sure I completely understood the part of this thread  
dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and  
reading the file with the CORE vfd (H5Pset\_fapl\_core) and setting your  
allocation inc to say 32\*(1\<\<32) (thats 32 gig and just a little bigger  
than your file) and then opening the file with that fapl? I'd expect  
that to have helped a lot.

Mark

> **···**
>
> On Wed, 2010-03-03 at 06:13, Thorben Kr�ger wrote:
> 
> > On Wednesday 03 March 2010 14:25:17 Thorben Kr�ger wrote:  
> > \> \> \> Though the 30GB file I was talking of was written using HDF5 1.8.4, if  
> > \> \> \> I understand correctly, it will not make use of these new features  
> > \> \> \> because it tries to maintain downward compatibility to 1.6. Correct?  
> > \> \>  
> > \> \> Yes, you should enable the new link storage by calling  
> > \> \> H5Pset\_libver\_bounds() explicitly. IMO, this would be the best path to  
> > \> \> improve performance for your scenario  
> > \>  
> > \> So can I convert my existing file to the 1.8-only file version without  
> > \> having to rerun the program that generated it? That would take a week I  
> > \> think ☹
> > 
> > So, I've found the "h5repack" command line tool now that accepts the "--  
> > latest" options, and am running this now on my file like this:
> > 
> > h5repack --latest oldfile.h5 newfile.h5
> > 
> > Will report back about possible performance improvements...
> > 
> > Cheers,  
> > Thorben
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > http://\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> 
> --  
> Mark C. Miller, Lawrence Livermore National Laboratory  
> ================!!LLNL BUSINESS ONLY!!================  
> miller86@llnl.gov urgent: miller86@pager.llnl.gov  
> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 4:08pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/14 "2010-03-03T16:08:40Z")

</div>

> Hello Thorben,
> 
> So, I am not sure I completely understood the part of this thread  
> dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and  
> reading the file with the CORE vfd (H5Pset\_fapl\_core)

I'm sorry, obviously I'm not deep into HDF5 yet or maybe I would have known  
that function, but I didn't. I now read up on it  
(for reference: [http://www.hdfgroup.org/HDF5/doc/UG/08\_TheFile.html](http://www.hdfgroup.org/HDF5/doc/UG/08_TheFile.html) )  
and I will try this out now...

> and setting your  
> allocation inc to say 32\*(1\<\<32) (thats 32 gig and just a little bigger  
> than your file)

What's an allocation inc?

> and then opening the file with that fapl? I'd expect  
> that to have helped a lot.

I hope you're right; it will probably take me at least till tomorrow to get a  
result, but will report back

Cheers,  
Thorben

> **···**
>
> On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:

---

<div class="post-metadata">

**Author:** ![miller86](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/miller86/32/124_2.png) [@miller86](https://forum.hdfgroup.org/u/miller86)\
**Post date:** [March 3, 2010, 4:28pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/15 "2010-03-03T16:28:27Z")

</div>

Hello Thorben,

> \> Hello Thorben,  
> \>  
> \> So, I am not sure I completely understood the part of this thread  
> \> dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and  
> \> reading the file with the CORE vfd (H5Pset\_fapl\_core)
> 
> I'm sorry, obviously I'm not deep into HDF5 yet or maybe I would have known  
> that function, but I didn't. I now read up on it  
> (for reference: http://\*www.\*[hdfgroup.org/HDF5/doc/UG/08\_TheFile.html](http://hdfgroup.org/HDF5/doc/UG/08_TheFile.html) )  
> and I will try this out now...

No worries. The interface has A LOT of options.

> \> and setting your  
> \> allocation inc to say 32\*(1\<\<32) (thats 32 gig and just a little bigger  
> \> than your file)
> 
> What's an allocation inc?

Well, I may have been a bit misleading here. Since you really only need  
to READ this file, then the allocation increment may not be that  
relevant. But, if you are creating or appending a file the allocation  
incriment tells the HDF5 library how much MORE memory to grab each time  
the 'file' (in memory) grows beyond the size of the currently allocated  
buffer. If you set this number too small, then you can wind up spending  
a lot of time in re-allocs (and subsequent memcpys) because you exceed  
the current size too often. If you set it too large, you can wind up  
allocating too much more memory than you need.

The ability to READ file from the core vfd is a relatively new feature  
to HDF5. And, when you are READING, I am pretty sure the library is  
smart enough to set the initial buffer allocated to the size of the file  
that gets read. So, it may be that you actually DO NOT WANT to set the  
allocation increment to 32\*(1\<\<32) as I suggested above but instead  
something quite small, say (1\<\<12) (4 kilobytes). Also, if you are ONLY  
reading the file, you probably want to set H5\_ACC\_RDONLY in the modes to  
the open call.

Good luck.

> **···**
>
> On Wed, 2010-03-03 at 08:08, Thorben Kr�ger wrote:
> 
> > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> 
> > \> and then opening the file with that fapl? I'd expect  
> > \> that to have helped a lot.
> > 
> > I hope you're right; it will probably take me at least till tomorrow to get a  
> > result, but will report back
> > 
> > Cheers,  
> > Thorben
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > http://\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> 
> --  
> Mark C. Miller, Lawrence Livermore National Laboratory  
> ================!!LLNL BUSINESS ONLY!!================  
> miller86@llnl.gov urgent: miller86@pager.llnl.gov  
> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 5:07pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/16 "2010-03-03T17:07:52Z")

</div>

instead of opening my file like this:

hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, H5P\_DEFAULT);

I am now doing this:

hid\_t fapl = H5Pcreate(H5P\_FILE\_ACCESS);  
if(fapl \< 0) {throw std::runtime\_error("fapl \< 0");}  
hid\_t ret = H5Pset\_fapl\_core(fapl, 5/\*MB?\*/, 0 /\*no backing store\*/);  
if(ret \< 0) {throw std::runtime\_error("could not create fapl core");}  
hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, fapl);

But using the second way, it cannot open any of my datasets:

HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:  
&nbsp;&nbsp;#000: H5D.c line 321 in H5Dopen2(): not found  
&nbsp;&nbsp;&nbsp;&nbsp;major: Dataset  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#001: H5Gloc.c line 468 in H5G\_loc\_find(): can't find object  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#002: H5Gtraverse.c line 877 in H5G\_traverse(): internal path traversal  
failed  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#003: H5Gtraverse.c line 665 in H5G\_traverse\_real(): can't look up component  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#004: H5Gobj.c line 1047 in H5G\_obj\_lookup(): can't locate object  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#005: H5Gstab.c line 860 in H5G\_stab\_lookup(): not found  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#006: H5B.c line 360 in H5B\_find(): can't lookup key in leaf node  
&nbsp;&nbsp;&nbsp;&nbsp;major: B-Tree node  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
&nbsp;&nbsp;#007: H5Gnode.c line 522 in H5G\_node\_found(): unable to protect symbol table  
node  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache  
&nbsp;&nbsp;#008: H5AC.c line 1831 in H5AC\_protect(): H5C\_protect() failed.  
&nbsp;&nbsp;&nbsp;&nbsp;major: Object cache  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to protect metadata  
&nbsp;&nbsp;#009: H5C.c line 6160 in H5C\_protect(): can't load entry  
&nbsp;&nbsp;&nbsp;&nbsp;major: Object cache  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache  
&nbsp;&nbsp;#010: H5C.c line 10990 in H5C\_load\_entry(): unable to load entry  
&nbsp;&nbsp;&nbsp;&nbsp;major: Object cache  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache  
&nbsp;&nbsp;#011: H5Gcache.c line 176 in H5G\_node\_load(): bad symbol table node  
signature  
&nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
&nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache

I'm at a loss now, what am I doing wrong here?

Cheers,  
Thorben

> **···**
>
> > \> On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:  
> > \> \> Hello Thorben,
> 
> > \> \> However, did you in fact try using HDF5-1.8.4  
> > \> \> and reading the file with the CORE vfd (H5Pset\_fapl\_core)

---

<div class="post-metadata">

**Author:** ![miller86](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/miller86/32/124_2.png) [@miller86](https://forum.hdfgroup.org/u/miller86)\
**Post date:** [March 3, 2010, 5:17pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/17 "2010-03-03T17:17:31Z")

</div>

I think the '5' in the HP5set\_fapl\_core means to allocate only 5 more  
bytes each time it needs to allocate. I think that is too small and  
maybe a problem since its an odd number and not a power of 2.

Although I would not necessarily expect that to have failed in the way  
you are seeing either.

Try '5\<\<20' instead of '5'

Other than that, I don't see anything wrong with what your are doing.  
Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your  
file into core is failing due to too large a request. I guess that is  
possible. To see if that is happening, if your HDF5 installation was  
compiled with tracing/debugging features turned on, then try setting  
HDF5\_DEBUG=all in your enviornment where you are running your  
application.

Mark

> **···**
>
> On Wed, 2010-03-03 at 09:07, Thorben Kr�ger wrote:
> 
> > \> \> On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:  
> > \> \> \> Hello Thorben,
> > 
> > \> \> \> However, did you in fact try using HDF5-1.8.4  
> > \> \> \> and reading the file with the CORE vfd (H5Pset\_fapl\_core)
> > 
> > instead of opening my file like this:
> > 
> > hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, H5P\_DEFAULT);
> > 
> > I am now doing this:
> > 
> > hid\_t fapl = H5Pcreate(H5P\_FILE\_ACCESS);  
> > if(fapl \< 0) {throw std::runtime\_error("fapl \< 0");}  
> > hid\_t ret = H5Pset\_fapl\_core(fapl, 5/\*MB?\*/, 0 /\*no backing store\*/);  
> > if(ret \< 0) {throw std::runtime\_error("could not create fapl core");}  
> > hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, fapl);
> > 
> > But using the second way, it cannot open any of my datasets:
> > 
> > HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:  
> > &nbsp;&nbsp;#000: H5D.c line 321 in H5Dopen2(): not found  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Dataset  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#001: H5Gloc.c line 468 in H5G\_loc\_find(): can't find object  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#002: H5Gtraverse.c line 877 in H5G\_traverse(): internal path traversal  
> > failed  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#003: H5Gtraverse.c line 665 in H5G\_traverse\_real(): can't look up component  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#004: H5Gobj.c line 1047 in H5G\_obj\_lookup(): can't locate object  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#005: H5Gstab.c line 860 in H5G\_stab\_lookup(): not found  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#006: H5B.c line 360 in H5B\_find(): can't lookup key in leaf node  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: B-Tree node  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Object not found  
> > &nbsp;&nbsp;#007: H5Gnode.c line 522 in H5G\_node\_found(): unable to protect symbol table  
> > node  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache  
> > &nbsp;&nbsp;#008: H5AC.c line 1831 in H5AC\_protect(): H5C\_protect() failed.  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Object cache  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to protect metadata  
> > &nbsp;&nbsp;#009: H5C.c line 6160 in H5C\_protect(): can't load entry  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Object cache  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache  
> > &nbsp;&nbsp;#010: H5C.c line 10990 in H5C\_load\_entry(): unable to load entry  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Object cache  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache  
> > &nbsp;&nbsp;#011: H5Gcache.c line 176 in H5G\_node\_load(): bad symbol table node  
> > signature  
> > &nbsp;&nbsp;&nbsp;&nbsp;major: Symbol table  
> > &nbsp;&nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache
> > 
> > I'm at a loss now, what am I doing wrong here?
> > 
> > Cheers,  
> > Thorben
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > http://\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> 
> --  
> Mark C. Miller, Lawrence Livermore National Laboratory  
> ================!!LLNL BUSINESS ONLY!!================  
> miller86@llnl.gov urgent: miller86@pager.llnl.gov  
> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

---

<div class="post-metadata">

**Author:** ![Thorben\_Kroger](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/t/8e7dd6/32.png) [@Thorben\_Kroger](https://forum.hdfgroup.org/u/Thorben_Kroger)\
**Post date:** [March 3, 2010, 5:30pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/18 "2010-03-03T17:30:16Z")

</div>

> I think the '5' in the HP5set\_fapl\_core means to allocate only 5 more  
> bytes each time it needs to allocate. I think that is too small and  
> maybe a problem since its an odd number and not a power of 2.
> 
> Although I would not necessarily expect that to have failed in the way  
> you are seeing either.
> 
> Try '5\<\<20' instead of '5'

I found this in h5test.c, function h5\_fileaccess()

/\* In-core temporary file with 1MB increment \*/  
&nbsp;&nbsp;if (H5Pset\_fapl\_core(fapl, (size\_t)1, TRUE)\<0) return -1;

Anyway, using a number like 5\<\<20 did not make a difference, it still  
aborts...

Cheers,  
Thorben

> **···**
>
> On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:
> 
> > Other than that, I don't see anything wrong with what your are doing.  
> > Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your  
> > file into core is failing due to too large a request. I guess that is  
> > possible. To see if that is happening, if your HDF5 installation was  
> > compiled with tracing/debugging features turned on, then try setting  
> > HDF5\_DEBUG=all in your enviornment where you are running your  
> > application.
> > 
> > Mark
> > 
> > On Wed, 2010-03-03 at 09:07, Thorben Kröger wrote:  
> > \> \> \> On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:  
> > \> \> \> \> Hello Thorben,  
> > \> \> \> \>  
> > \> \> \> \> However, did you in fact try using HDF5-1.8.4  
> > \> \> \> \> and reading the file with the CORE vfd (H5Pset\_fapl\_core)  
> > \>  
> > \> instead of opening my file like this:  
> > \>  
> > \> hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY,  
> > \> H5P\_DEFAULT);  
> > \>  
> > \> I am now doing this:  
> > \>  
> > \> hid\_t fapl = H5Pcreate(H5P\_FILE\_ACCESS);  
> > \> if(fapl \< 0) {throw std::runtime\_error("fapl \< 0");}  
> > \> hid\_t ret = H5Pset\_fapl\_core(fapl, 5/\*MB?\*/, 0 /\*no backing store\*/);  
> > \> if(ret \< 0) {throw std::runtime\_error("could not create fapl core");}  
> > \> hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, fapl);  
> > \>  
> > \> But using the second way, it cannot open any of my datasets:  
> > \>  
> > \> HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:  
> > \> #000: H5D.c line 321 in H5Dopen2(): not found  
> > \>  
> > \> major: Dataset  
> > \> minor: Object not found  
> > \>  
> > \> #001: H5Gloc.c line 468 in H5G\_loc\_find(): can't find object  
> > \>  
> > \> major: Symbol table  
> > \> minor: Object not found  
> > \>  
> > \> #002: H5Gtraverse.c line 877 in H5G\_traverse(): internal path traversal  
> > \>  
> > \> failed  
> > \>  
> > \> major: Symbol table  
> > \> minor: Object not found  
> > \>  
> > \> #003: H5Gtraverse.c line 665 in H5G\_traverse\_real(): can't look up  
> > \> component  
> > \>  
> > \> major: Symbol table  
> > \> minor: Object not found  
> > \>  
> > \> #004: H5Gobj.c line 1047 in H5G\_obj\_lookup(): can't locate object  
> > \>  
> > \> major: Symbol table  
> > \> minor: Object not found  
> > \>  
> > \> #005: H5Gstab.c line 860 in H5G\_stab\_lookup(): not found  
> > \>  
> > \> major: Symbol table  
> > \> minor: Object not found  
> > \>  
> > \> #006: H5B.c line 360 in H5B\_find(): can't lookup key in leaf node  
> > \>  
> > \> major: B-Tree node  
> > \> minor: Object not found  
> > \>  
> > \> #007: H5Gnode.c line 522 in H5G\_node\_found(): unable to protect symbol  
> > \> table  
> > \>  
> > \> node  
> > \>  
> > \> major: Symbol table  
> > \> minor: Unable to load metadata into cache  
> > \>  
> > \> #008: H5AC.c line 1831 in H5AC\_protect(): H5C\_protect() failed.  
> > \>  
> > \> major: Object cache  
> > \> minor: Unable to protect metadata  
> > \>  
> > \> #009: H5C.c line 6160 in H5C\_protect(): can't load entry  
> > \>  
> > \> major: Object cache  
> > \> minor: Unable to load metadata into cache  
> > \>  
> > \> #010: H5C.c line 10990 in H5C\_load\_entry(): unable to load entry  
> > \>  
> > \> major: Object cache  
> > \> minor: Unable to load metadata into cache  
> > \>  
> > \> #011: H5Gcache.c line 176 in H5G\_node\_load(): bad symbol table node  
> > \>  
> > \> signature  
> > \>  
> > \> major: Symbol table  
> > \> minor: Unable to load metadata into cache  
> > \>  
> > \> I'm at a loss now, what am I doing wrong here?  
> > \>  
> > \> Cheers,  
> > \> Thorben  
> > \>  
> > \> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > \> Hdf-forum is for HDF software users discussion.  
> > \> Hdf-forum@hdfgroup.org  
> > \> http://\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)

---

<div class="post-metadata">

**Author:** ![miller86](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/miller86/32/124_2.png) [@miller86](https://forum.hdfgroup.org/u/miller86)\
**Post date:** [March 3, 2010, 5:49pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/19 "2010-03-03T17:49:09Z")

</div>

I think the comment in h5test.c is misleading.

The documentation for H5Pset\_fapl\_core says the inc argument is the  
number of BYTES, not MEGAbytes.

I'd bet the initial malloc is failing for reading your whole file.  
Sorry. Didn't expect that. But, I was thinking in terms of 64 bits and  
30 gigs 5 bits too many for a 32 bit number. So, if you are running in  
truly 64 bit, then I am completely puzzled. Otherwise, if you are  
running 32 bit, then I think the problem has presented itself; we can't  
address 30 gig buffer in a 32 bit application.

I'd suggest looking at some of the other properties such as

H5Pset\_small\_data\_block\_size  
HP5set\_buffer (used during read/xform only)  
H5Pset\_cache (relevant only for chunked datasets)  
H5Pset\_meta\_block\_size( hid\_t fapl\_id, hsize\_t size )

and see if setting any/all of these to large numbers; 100's of megabytes  
has any positive impact.

Mark

> **···**
>
> On Wed, 2010-03-03 at 09:30, Thorben Kr�ger wrote:
> 
> > On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:  
> > \> I think the '5' in the HP5set\_fapl\_core means to allocate only 5 more  
> > \> bytes each time it needs to allocate. I think that is too small and  
> > \> maybe a problem since its an odd number and not a power of 2.  
> > \>  
> > \> Although I would not necessarily expect that to have failed in the way  
> > \> you are seeing either.  
> > \>  
> > \> Try '5\<\<20' instead of '5'
> > 
> > I found this in h5test.c, function h5\_fileaccess()
> > 
> > /\* In-core temporary file with 1MB increment \*/  
> > &nbsp;&nbsp;if (H5Pset\_fapl\_core(fapl, (size\_t)1, TRUE)\<0) return -1;
> > 
> > Anyway, using a number like 5\<\<20 did not make a difference, it still  
> > aborts...
> > 
> > Cheers,  
> > Thorben
> > 
> > \>  
> > \> Other than that, I don't see anything wrong with what your are doing.  
> > \> Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your  
> > \> file into core is failing due to too large a request. I guess that is  
> > \> possible. To see if that is happening, if your HDF5 installation was  
> > \> compiled with tracing/debugging features turned on, then try setting  
> > \> HDF5\_DEBUG=all in your enviornment where you are running your  
> > \> application.  
> > \>  
> > \> Mark  
> > \>  
> > \> On Wed, 2010-03-03 at 09:07, Thorben Kr�ger wrote:  
> > \> \> \> \> On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:  
> > \> \> \> \> \> Hello Thorben,  
> > \> \> \> \> \>  
> > \> \> \> \> \> However, did you in fact try using HDF5-1.8.4  
> > \> \> \> \> \> and reading the file with the CORE vfd (H5Pset\_fapl\_core)  
> > \> \>  
> > \> \> instead of opening my file like this:  
> > \> \>  
> > \> \> hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY,  
> > \> \> H5P\_DEFAULT);  
> > \> \>  
> > \> \> I am now doing this:  
> > \> \>  
> > \> \> hid\_t fapl = H5Pcreate(H5P\_FILE\_ACCESS);  
> > \> \> if(fapl \< 0) {throw std::runtime\_error("fapl \< 0");}  
> > \> \> hid\_t ret = H5Pset\_fapl\_core(fapl, 5/\*MB?\*/, 0 /\*no backing store\*/);  
> > \> \> if(ret \< 0) {throw std::runtime\_error("could not create fapl core");}  
> > \> \> hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, fapl);  
> > \> \>  
> > \> \> But using the second way, it cannot open any of my datasets:  
> > \> \>  
> > \> \> HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:  
> > \> \> #000: H5D.c line 321 in H5Dopen2(): not found  
> > \> \>  
> > \> \> major: Dataset  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #001: H5Gloc.c line 468 in H5G\_loc\_find(): can't find object  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #002: H5Gtraverse.c line 877 in H5G\_traverse(): internal path traversal  
> > \> \>  
> > \> \> failed  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #003: H5Gtraverse.c line 665 in H5G\_traverse\_real(): can't look up  
> > \> \> component  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #004: H5Gobj.c line 1047 in H5G\_obj\_lookup(): can't locate object  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #005: H5Gstab.c line 860 in H5G\_stab\_lookup(): not found  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #006: H5B.c line 360 in H5B\_find(): can't lookup key in leaf node  
> > \> \>  
> > \> \> major: B-Tree node  
> > \> \> minor: Object not found  
> > \> \>  
> > \> \> #007: H5Gnode.c line 522 in H5G\_node\_found(): unable to protect symbol  
> > \> \> table  
> > \> \>  
> > \> \> node  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Unable to load metadata into cache  
> > \> \>  
> > \> \> #008: H5AC.c line 1831 in H5AC\_protect(): H5C\_protect() failed.  
> > \> \>  
> > \> \> major: Object cache  
> > \> \> minor: Unable to protect metadata  
> > \> \>  
> > \> \> #009: H5C.c line 6160 in H5C\_protect(): can't load entry  
> > \> \>  
> > \> \> major: Object cache  
> > \> \> minor: Unable to load metadata into cache  
> > \> \>  
> > \> \> #010: H5C.c line 10990 in H5C\_load\_entry(): unable to load entry  
> > \> \>  
> > \> \> major: Object cache  
> > \> \> minor: Unable to load metadata into cache  
> > \> \>  
> > \> \> #011: H5Gcache.c line 176 in H5G\_node\_load(): bad symbol table node  
> > \> \>  
> > \> \> signature  
> > \> \>  
> > \> \> major: Symbol table  
> > \> \> minor: Unable to load metadata into cache  
> > \> \>  
> > \> \> I'm at a loss now, what am I doing wrong here?  
> > \> \>  
> > \> \> Cheers,  
> > \> \> Thorben  
> > \> \>  
> > \> \> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > \> \> Hdf-forum is for HDF software users discussion.  
> > \> \> Hdf-forum@hdfgroup.org  
> > \> \> http://\*\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > http://\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> 
> --  
> Mark C. Miller, Lawrence Livermore National Laboratory  
> ================!!LLNL BUSINESS ONLY!!================  
> miller86@llnl.gov urgent: miller86@pager.llnl.gov  
> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

---

<div class="post-metadata">

**Author:** ![Quincey\_Koziol](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/q/dbc845/32.png) [@Quincey\_Koziol](https://forum.hdfgroup.org/u/Quincey_Koziol)\
**Post date:** [March 3, 2010, 9:02pm UTC](https://forum.hdfgroup.org/t/loading-chunks-or-even-the-whole-hdf5-file-into-memory/1637/20 "2010-03-03T21:02:42Z")

</div>

> I think the comment in h5test.c is misleading.
> 
> The documentation for H5Pset\_fapl\_core says the inc argument is the  
> number of BYTES, not MEGAbytes.
> 
> I'd bet the initial malloc is failing for reading your whole file.  
> Sorry. Didn't expect that. But, I was thinking in terms of 64 bits and  
> 30 gigs 5 bits too many for a 32 bit number. So, if you are running in  
> truly 64 bit, then I am completely puzzled. Otherwise, if you are  
> running 32 bit, then I think the problem has presented itself; we can't  
> address 30 gig buffer in a 32 bit application.
> 
> I'd suggest looking at some of the other properties such as
> 
> H5Pset\_small\_data\_block\_size  
> HP5set\_buffer (used during read/xform only)  
> H5Pset\_cache (relevant only for chunked datasets)  
> H5Pset\_meta\_block\_size( hid\_t fapl\_id, hsize\_t size )
> 
> and see if setting any/all of these to large numbers; 100's of megabytes  
> has any positive impact.

&nbsp;&nbsp;Those probably won't have any effect, I'm guessing that you are correct that the machine ran out of memory (or Thorben has a quota about the amount of RAM his process can use).

&nbsp;&nbsp;Quincey

> **···**
>
> On Mar 3, 2010, at 11:49 AM, Mark Miller wrote:
> 
> > Mark
> > 
> > On Wed, 2010-03-03 at 09:30, Thorben Kröger wrote:
> > 
> > > On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:
> > > 
> > > > I think the '5' in the HP5set\_fapl\_core means to allocate only 5 more  
> > > > bytes each time it needs to allocate. I think that is too small and  
> > > > maybe a problem since its an odd number and not a power of 2.
> > > > 
> > > > Although I would not necessarily expect that to have failed in the way  
> > > > you are seeing either.
> > > > 
> > > > Try '5\<\<20' instead of '5'
> > > 
> > > I found this in h5test.c, function h5\_fileaccess()
> > > 
> > > /\* In-core temporary file with 1MB increment \*/  
> > > &nbsp;&nbsp;if (H5Pset\_fapl\_core(fapl, (size\_t)1, TRUE)\<0) return -1;
> > > 
> > > Anyway, using a number like 5\<\<20 did not make a difference, it still  
> > > aborts...
> > > 
> > > Cheers,  
> > > Thorben
> > > 
> > > > Other than that, I don't see anything wrong with what your are doing.  
> > > > Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your  
> > > > file into core is failing due to too large a request. I guess that is  
> > > > possible. To see if that is happening, if your HDF5 installation was  
> > > > compiled with tracing/debugging features turned on, then try setting  
> > > > HDF5\_DEBUG=all in your enviornment where you are running your  
> > > > application.
> > > > 
> > > > Mark
> > > > 
> > > > On Wed, 2010-03-03 at 09:07, Thorben Kröger wrote:
> > > > 
> > > > > > > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> > > > > > > 
> > > > > > > > Hello Thorben,
> > > > > > > > 
> > > > > > > > However, did you in fact try using HDF5-1.8.4  
> > > > > > > > and reading the file with the CORE vfd (H5Pset\_fapl\_core)
> > > > > 
> > > > > instead of opening my file like this:
> > > > > 
> > > > > hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY,  
> > > > > H5P\_DEFAULT);
> > > > > 
> > > > > I am now doing this:
> > > > > 
> > > > > hid\_t fapl = H5Pcreate(H5P\_FILE\_ACCESS);  
> > > > > if(fapl \< 0) {throw std::runtime\_error("fapl \< 0");}  
> > > > > hid\_t ret = H5Pset\_fapl\_core(fapl, 5/\*MB?\*/, 0 /\*no backing store\*/);  
> > > > > if(ret \< 0) {throw std::runtime\_error("could not create fapl core");}  
> > > > > hdf5geometryFile\_ = H5Fopen(filename.c\_str(), H5F\_ACC\_RDONLY, fapl);
> > > > > 
> > > > > But using the second way, it cannot open any of my datasets:
> > > > > 
> > > > > HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:  
> > > > > #000: H5D.c line 321 in H5Dopen2(): not found
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Dataset  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #001: H5Gloc.c line 468 in H5G\_loc\_find(): can't find object
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #002: H5Gtraverse.c line 877 in H5G\_traverse(): internal path traversal
> > > > > 
> > > > > failed
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #003: H5Gtraverse.c line 665 in H5G\_traverse\_real(): can't look up  
> > > > > component
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #004: H5Gobj.c line 1047 in H5G\_obj\_lookup(): can't locate object
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #005: H5Gstab.c line 860 in H5G\_stab\_lookup(): not found
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #006: H5B.c line 360 in H5B\_find(): can't lookup key in leaf node
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: B-Tree node  
> > > > > &nbsp;&nbsp;&nbsp;minor: Object not found
> > > > > 
> > > > > #007: H5Gnode.c line 522 in H5G\_node\_found(): unable to protect symbol  
> > > > > table
> > > > > 
> > > > > node
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache
> > > > > 
> > > > > #008: H5AC.c line 1831 in H5AC\_protect(): H5C\_protect() failed.
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Object cache  
> > > > > &nbsp;&nbsp;&nbsp;minor: Unable to protect metadata
> > > > > 
> > > > > #009: H5C.c line 6160 in H5C\_protect(): can't load entry
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Object cache  
> > > > > &nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache
> > > > > 
> > > > > #010: H5C.c line 10990 in H5C\_load\_entry(): unable to load entry
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Object cache  
> > > > > &nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache
> > > > > 
> > > > > #011: H5Gcache.c line 176 in H5G\_node\_load(): bad symbol table node
> > > > > 
> > > > > signature
> > > > > 
> > > > > &nbsp;&nbsp;&nbsp;major: Symbol table  
> > > > > &nbsp;&nbsp;&nbsp;minor: Unable to load metadata into cache
> > > > > 
> > > > > I'm at a loss now, what am I doing wrong here?
> > > > > 
> > > > > Cheers,  
> > > > > Thorben
> > > > > 
> > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > > > > Hdf-forum is for HDF software users discussion.  
> > > > > Hdf-forum@hdfgroup.org  
> > > > > http://\*\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> > > 
> > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > > Hdf-forum is for HDF software users discussion.  
> > > Hdf-forum@hdfgroup.org  
> > > http://\*[mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> > 
> > --  
> > Mark C. Miller, Lawrence Livermore National Laboratory  
> > ================!!LLNL BUSINESS ONLY!!================  
> > miller86@llnl.gov urgent: miller86@pager.llnl.gov  
> > T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)