loading chunks or even the whole hdf5 file into memory

Hdf-forum archives
1

Hello,
We have a ~30GB HDF5 file with something like 100 million small datasets in
it. We need to iterate through all of them, and doing so is very slow as each
one has to be loaded from disk. I also don't know if it is possible to find
out a proper ordering to go through them, so I suspect that there might also
be a lot of disk seeks necessary.

Maybe it isn't such a good idea to have so many small objects in the file, but
I'm stuck with this format now. What options do I have?

I'm working now on a machine with 128GB of RAM, so my file would fit
comfortably inside. Is it possible to load the file completely into memory to
avoid all of the above problems?

Thanks,
Thorben

2

On a related note, I've just found this piece of information which might
accelerate our program as well:

···

---

http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html

H5Pset_libver_bounds( hid_t fapl_id, H5F_libver_t low, H5F_libver_t high )

Compact-or-indexed groups enable much-compressed link storage for groups with
very few members and improved efficiency and performance for groups with very
large numbers of members. The efficiency and performance impacts are most
noticeable at the extremes: all unnecessary overhead is eliminated for groups
with zero members; groups with tens of thousands of members may see as much
as a 100-fold performance gain.

H5Pset_libver_bounds( hid_t fapl_id, H5F_libver_t low, H5F_libver_t high )
H5Pget_libver_bounds( hid_t fapl_id, H5F_libver_t* low, H5F_libver_t* high )

Default behavior: If H5Pset_libver_bounds is not called with low equal to
HDF_LIBVER_LATEST, then the HDF5 Library provides the greatest-possible
format compatibility. It does this by creating objects with the earliest
opssible format that will handle the data being stored and accommodate
the action being taken.

---

Though the 30GB file I was talking of was written using HDF5 1.8.4, if I
understand correctly, it will not make use of these new features because it
tries to maintain downward compatibility to 1.6. Correct?

Is there a tool available that converts an existing file to a new file version
to make use of all these performance improvements? Or should I hack this into
h5repack.c myself? I'd like to avoid that...

Cheers,
Thorben

On Wednesday 03 March 2010 11:47:18 Thorben Kröger wrote:

Hello,
We have a ~30GB HDF5 file with something like 100 million small datasets in
it. We need to iterate through all of them, and doing so is very slow as
each one has to be loaded from disk. I also don't know if it is possible
to find out a proper ordering to go through them, so I suspect that there
might also be a lot of disk seeks necessary.

Maybe it isn't such a good idea to have so many small objects in the file,
but I'm stuck with this format now. What options do I have?

I'm working now on a machine with 128GB of RAM, so my file would fit
comfortably inside. Is it possible to load the file completely into memory
to avoid all of the above problems?

Thanks,
Thorben

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

3

Yes, having many small objects is inefficient. H5iterate() just traverses
all objects as found by name, which is alphabetical ordering, which may or
may not be what you want. If you want a particular order, you could create
a table with references/links to the specific datasets, and iterate over
this table instead.

Are you using Linux as operating system? If so, then you can just copy
your 30GB file to

  /dev/shm

which (usually) is a tmpfs, i.e. a ramdisk under Linux.

A more portable solution is to use the memory virtual file driver,
where you can read an HDF5 file from an RAM image.

  Werner

···

On Wed, 03 Mar 2010 11:47:18 +0100, Thorben Kröger <thorben.kroeger@iwr.uni-heidelberg.de> wrote:

Hello,
We have a ~30GB HDF5 file with something like 100 million small datasets in
it. We need to iterate through all of them, and doing so is very slow as each
one has to be loaded from disk. I also don't know if it is possible to find
out a proper ordering to go through them, so I suspect that there might also
be a lot of disk seeks necessary.

Maybe it isn't such a good idea to have so many small objects in the file, but
I'm stuck with this format now. What options do I have?

I'm working now on a machine with 128GB of RAM, so my file would fit
comfortably inside. Is it possible to load the file completely into memory to
avoid all of the above problems?

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

4

> Hello,
> We have a ~30GB HDF5 file with something like 100 million small datasets
> in it. We need to iterate through all of them, and doing so is very slow
> as each one has to be loaded from disk. I also don't know if it is
> possible to find out a proper ordering to go through them, so I suspect
> that there might also be a lot of disk seeks necessary.
>
> Maybe it isn't such a good idea to have so many small objects in the
> file, but I'm stuck with this format now. What options do I have?
>
> I'm working now on a machine with 128GB of RAM, so my file would fit
> comfortably inside. Is it possible to load the file completely into
> memory to avoid all of the above problems?

Yes, having many small objects is inefficient. H5iterate() just traverses
all objects as found by name, which is alphabetical ordering, which may or
may not be what you want. If you want a particular order, you could create
a table with references/links to the specific datasets, and iterate over
this table instead.

Are you using Linux as operating system? If so, then you can just copy
your 30GB file to

  /dev/shm

which (usually) is a tmpfs, i.e. a ramdisk under Linux.

I also thought of using a ramdisk, but thought that you needed to have root
rights for that. As it turns out, I can copy stuff to /dev/shm :slight_smile:
Thanks, as a temporary solution this is really good :slight_smile:

···

On Wednesday 03 March 2010 13:43:38 Werner Benger wrote:

On Wed, 03 Mar 2010 11:47:18 +0100, Thorben Kröger <thorben.kroeger@iwr.uni- heidelberg.de> wrote:

A more portable solution is to use the memory virtual file driver,
where you can read an HDF5 file from an RAM image.

  Werner

5

A Wednesday 03 March 2010 13:18:02 Thorben Kröger escrigué:

On a related note, I've just found this piece of information which might
accelerate our program as well:

---

http://www.hdfgroup.org/HDF5/doc/ADGuide/CompatFormat180.html

H5Pset_libver_bounds( hid_t fapl_id, H5F_libver_t low, H5F_libver_t high )

Compact-or-indexed groups enable much-compressed link storage for groups
with very few members and improved efficiency and performance for groups
with very large numbers of members. The efficiency and performance impacts
are most noticeable at the extremes: all unnecessary overhead is
eliminated for groups with zero members; groups with tens of thousands of
members may see as much as a 100-fold performance gain.

H5Pset_libver_bounds( hid_t fapl_id, H5F_libver_t low, H5F_libver_t high )
H5Pget_libver_bounds( hid_t fapl_id, H5F_libver_t* low, H5F_libver_t* high
)

Default behavior: If H5Pset_libver_bounds is not called with low equal to
HDF_LIBVER_LATEST, then the HDF5 Library provides the greatest-possible
format compatibility. It does this by creating objects with the earliest
opssible format that will handle the data being stored and accommodate
the action being taken.

---

Though the 30GB file I was talking of was written using HDF5 1.8.4, if I
understand correctly, it will not make use of these new features because it
tries to maintain downward compatibility to 1.6. Correct?

Yes, you should enable the new link storage by calling H5Pset_libver_bounds()
explicitly. IMO, this would be the best path to improve performance for your
scenario (except if you don't want to follow the safer path of creating large
datasets and references to parts of them, as Werner suggested).

···

--
Francesc Alted

6

A Wednesday 03 March 2010 14:03:16 Thorben Kröger escrigué:

> Are you using Linux as operating system? If so, then you can just copy
> your 30GB file to
>
> /dev/shm
>
> which (usually) is a tmpfs, i.e. a ramdisk under Linux.

I also thought of using a ramdisk, but thought that you needed to have root
rights for that. As it turns out, I can copy stuff to /dev/shm :slight_smile:
Thanks, as a temporary solution this is really good :slight_smile:

I don't think you are going to have a decent speed-up by just copying to
/dev/shm because your 30 GB file probably was probably completely in your OS
filesystem cache already, which is memory too. Having said this, in case you
get some noticeable speed-up please tell us: I like to hear how wrong I am :wink:

···

--
Francesc Alted

7

> Though the 30GB file I was talking of was written using HDF5 1.8.4, if I
> understand correctly, it will not make use of these new features because
> it tries to maintain downward compatibility to 1.6. Correct?

Yes, you should enable the new link storage by calling
H5Pset_libver_bounds() explicitly. IMO, this would be the best path to
improve performance for your scenario

So can I convert my existing file to the 1.8-only file version without having
to rerun the program that generated it? That would take a week I think :frowning:

(except if you don't want to follow
the safer path of creating large datasets and references to parts of them,
as Werner suggested).

I would, but I do not want to rerun the program that generated the big file,
and don't want to write a conversion tool atm. either, so I'm stuck with the
data layout...

Cheers,
Thorben

8

Hello,

I'd like to create a null dataspace dataset in java.

For this, the java doc
http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S_NULL
mentions the HS5_NULL constant but this
constant is missing in the source file
(hdf-java/ncsa/hdf/hdf5lib/HDF5Constants.java).

Is there another way to create a null dataspace in java ?

Best regards (thanks a lot for this awesome HDF5 format / library),

Cyril.

9

I don't think you are going to have a decent speed-up by just copying to
/dev/shm because your 30 GB file probably was probably completely in your
OS filesystem cache already, which is memory too. Having said this, in
case you get some noticeable speed-up please tell us: I like to hear how
wrong I am :wink:

I'm going to have to disappoint you then:
I just tried it, and as you said, there really is _no_ speed difference at
all.

So I guess my only option now is to try to get my existing file over to the
new 1.8-only file format.

Cheers,
Thorben

10

So, I've found the "h5repack" command line tool now that accepts the "--
latest" options, and am running this now on my file like this:

h5repack --latest oldfile.h5 newfile.h5

Will report back about possible performance improvements...

Cheers,
Thorben

···

On Wednesday 03 March 2010 14:25:17 Thorben Kröger wrote:

> > Though the 30GB file I was talking of was written using HDF5 1.8.4, if
> > I understand correctly, it will not make use of these new features
> > because it tries to maintain downward compatibility to 1.6. Correct?
>
> Yes, you should enable the new link storage by calling
> H5Pset_libver_bounds() explicitly. IMO, this would be the best path to
> improve performance for your scenario

So can I convert my existing file to the 1.8-only file version without
having to rerun the program that generated it? That would take a week I
think :frowning:

11

The current hdf-java release does not support any HDF5 1.8 features (including constants).
We are working on supporting HDF5 1.8 in hdf-java. H5S_NULL will be in our future release.

Thanks
--pc

Giraudon Cyril wrote:

···

Hello,

I'd like to create a null dataspace dataset in java.

For this, the java doc
http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S_NULL
mentions the HS5_NULL constant but this
constant is missing in the source file
(hdf-java/ncsa/hdf/hdf5lib/HDF5Constants.java).

Is there another way to create a null dataspace in java ?

Best regards (thanks a lot for this awesome HDF5 format / library),

Cyril.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

12

Thanks Peter for your response.

Le 11/03/2010 15:01, Peter Cao a �crit :

···

The current hdf-java release does not support any HDF5 1.8 features
(including constants).
We are working on supporting HDF5 1.8 in hdf-java. H5S_NULL will be in
our future release.

Thanks
--pc

Giraudon Cyril wrote:

Hello,

I'd like to create a null dataspace dataset in java.

For this, the java doc
http://www.hdfgroup.org/hdf-java-html/javadocs/ncsa/hdf/hdf5lib/HDF5Constants.html#H5S_NULL

mentions the HS5_NULL constant but this
constant is missing in the source file
(hdf-java/ncsa/hdf/hdf5lib/HDF5Constants.java).

Is there another way to create a null dataspace in java ?

Best regards (thanks a lot for this awesome HDF5 format / library),

Cyril.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

13

Hello Thorben,

So, I am not sure I completely understood the part of this thread
dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and
reading the file with the CORE vfd (H5Pset_fapl_core) and setting your
allocation inc to say 32*(1<<32) (thats 32 gig and just a little bigger
than your file) and then opening the file with that fapl? I'd expect
that to have helped a lot.

Mark

···

On Wed, 2010-03-03 at 06:13, Thorben Kr�ger wrote:

On Wednesday 03 March 2010 14:25:17 Thorben Kr�ger wrote:
> > > Though the 30GB file I was talking of was written using HDF5 1.8.4, if
> > > I understand correctly, it will not make use of these new features
> > > because it tries to maintain downward compatibility to 1.6. Correct?
> >
> > Yes, you should enable the new link storage by calling
> > H5Pset_libver_bounds() explicitly. IMO, this would be the best path to
> > improve performance for your scenario
>
> So can I convert my existing file to the 1.8-only file version without
> having to rerun the program that generated it? That would take a week I
> think :frowning:

So, I've found the "h5repack" command line tool now that accepts the "--
latest" options, and am running this now on my file like this:

h5repack --latest oldfile.h5 newfile.h5

Will report back about possible performance improvements...

Cheers,
Thorben

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

14

Hello Thorben,

So, I am not sure I completely understood the part of this thread
dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and
reading the file with the CORE vfd (H5Pset_fapl_core)

I'm sorry, obviously I'm not deep into HDF5 yet or maybe I would have known
that function, but I didn't. I now read up on it
(for reference: http://www.hdfgroup.org/HDF5/doc/UG/08_TheFile.html )
and I will try this out now...

and setting your
allocation inc to say 32*(1<<32) (thats 32 gig and just a little bigger
than your file)

What's an allocation inc?

and then opening the file with that fapl? I'd expect
that to have helped a lot.

I hope you're right; it will probably take me at least till tomorrow to get a
result, but will report back

Cheers,
Thorben

···

On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:

15

Hello Thorben,

> Hello Thorben,
>
> So, I am not sure I completely understood the part of this thread
> dealing with /dev/shm. However, did you in fact try using HDF5-1.8.4 and
> reading the file with the CORE vfd (H5Pset_fapl_core)

I'm sorry, obviously I'm not deep into HDF5 yet or maybe I would have known
that function, but I didn't. I now read up on it
(for reference: http://*www.*hdfgroup.org/HDF5/doc/UG/08_TheFile.html )
and I will try this out now...

No worries. The interface has A LOT of options.

> and setting your
> allocation inc to say 32*(1<<32) (thats 32 gig and just a little bigger
> than your file)

What's an allocation inc?

Well, I may have been a bit misleading here. Since you really only need
to READ this file, then the allocation increment may not be that
relevant. But, if you are creating or appending a file the allocation
incriment tells the HDF5 library how much MORE memory to grab each time
the 'file' (in memory) grows beyond the size of the currently allocated
buffer. If you set this number too small, then you can wind up spending
a lot of time in re-allocs (and subsequent memcpys) because you exceed
the current size too often. If you set it too large, you can wind up
allocating too much more memory than you need.

The ability to READ file from the core vfd is a relatively new feature
to HDF5. And, when you are READING, I am pretty sure the library is
smart enough to set the initial buffer allocated to the size of the file
that gets read. So, it may be that you actually DO NOT WANT to set the
allocation increment to 32*(1<<32) as I suggested above but instead
something quite small, say (1<<12) (4 kilobytes). Also, if you are ONLY
reading the file, you probably want to set H5_ACC_RDONLY in the modes to
the open call.

Good luck.

···

On Wed, 2010-03-03 at 08:08, Thorben Kr�ger wrote:

On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:

> and then opening the file with that fapl? I'd expect
> that to have helped a lot.

I hope you're right; it will probably take me at least till tomorrow to get a
result, but will report back

Cheers,
Thorben

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

16

instead of opening my file like this:

hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, H5P_DEFAULT);

I am now doing this:

hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
if(fapl < 0) {throw std::runtime_error("fapl < 0");}
hid_t ret = H5Pset_fapl_core(fapl, 5/*MB?*/, 0 /*no backing store*/);
if(ret < 0) {throw std::runtime_error("could not create fapl core");}
hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, fapl);

But using the second way, it cannot open any of my datasets:

HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: H5D.c line 321 in H5Dopen2(): not found
    major: Dataset
    minor: Object not found
  #001: H5Gloc.c line 468 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #002: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal
failed
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 665 in H5G_traverse_real(): can't look up component
    major: Symbol table
    minor: Object not found
  #004: H5Gobj.c line 1047 in H5G_obj_lookup(): can't locate object
    major: Symbol table
    minor: Object not found
  #005: H5Gstab.c line 860 in H5G_stab_lookup(): not found
    major: Symbol table
    minor: Object not found
  #006: H5B.c line 360 in H5B_find(): can't lookup key in leaf node
    major: B-Tree node
    minor: Object not found
  #007: H5Gnode.c line 522 in H5G_node_found(): unable to protect symbol table
node
    major: Symbol table
    minor: Unable to load metadata into cache
  #008: H5AC.c line 1831 in H5AC_protect(): H5C_protect() failed.
    major: Object cache
    minor: Unable to protect metadata
  #009: H5C.c line 6160 in H5C_protect(): can't load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #010: H5C.c line 10990 in H5C_load_entry(): unable to load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #011: H5Gcache.c line 176 in H5G_node_load(): bad symbol table node
signature
    major: Symbol table
    minor: Unable to load metadata into cache

I'm at a loss now, what am I doing wrong here?

Cheers,
Thorben

···

> On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> > Hello Thorben,

> > However, did you in fact try using HDF5-1.8.4
> > and reading the file with the CORE vfd (H5Pset_fapl_core)

17

I think the '5' in the HP5set_fapl_core means to allocate only 5 more
bytes each time it needs to allocate. I think that is too small and
maybe a problem since its an odd number and not a power of 2.

Although I would not necessarily expect that to have failed in the way
you are seeing either.

Try '5<<20' instead of '5'

Other than that, I don't see anything wrong with what your are doing.
Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your
file into core is failing due to too large a request. I guess that is
possible. To see if that is happening, if your HDF5 installation was
compiled with tracing/debugging features turned on, then try setting
HDF5_DEBUG=all in your enviornment where you are running your
application.

Mark

···

On Wed, 2010-03-03 at 09:07, Thorben Kr�ger wrote:

> > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> > > Hello Thorben,

> > > However, did you in fact try using HDF5-1.8.4
> > > and reading the file with the CORE vfd (H5Pset_fapl_core)

instead of opening my file like this:

hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, H5P_DEFAULT);

I am now doing this:

hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
if(fapl < 0) {throw std::runtime_error("fapl < 0");}
hid_t ret = H5Pset_fapl_core(fapl, 5/*MB?*/, 0 /*no backing store*/);
if(ret < 0) {throw std::runtime_error("could not create fapl core");}
hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, fapl);

But using the second way, it cannot open any of my datasets:

HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
  #000: H5D.c line 321 in H5Dopen2(): not found
    major: Dataset
    minor: Object not found
  #001: H5Gloc.c line 468 in H5G_loc_find(): can't find object
    major: Symbol table
    minor: Object not found
  #002: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal
failed
    major: Symbol table
    minor: Object not found
  #003: H5Gtraverse.c line 665 in H5G_traverse_real(): can't look up component
    major: Symbol table
    minor: Object not found
  #004: H5Gobj.c line 1047 in H5G_obj_lookup(): can't locate object
    major: Symbol table
    minor: Object not found
  #005: H5Gstab.c line 860 in H5G_stab_lookup(): not found
    major: Symbol table
    minor: Object not found
  #006: H5B.c line 360 in H5B_find(): can't lookup key in leaf node
    major: B-Tree node
    minor: Object not found
  #007: H5Gnode.c line 522 in H5G_node_found(): unable to protect symbol table
node
    major: Symbol table
    minor: Unable to load metadata into cache
  #008: H5AC.c line 1831 in H5AC_protect(): H5C_protect() failed.
    major: Object cache
    minor: Unable to protect metadata
  #009: H5C.c line 6160 in H5C_protect(): can't load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #010: H5C.c line 10990 in H5C_load_entry(): unable to load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #011: H5Gcache.c line 176 in H5G_node_load(): bad symbol table node
signature
    major: Symbol table
    minor: Unable to load metadata into cache

I'm at a loss now, what am I doing wrong here?

Cheers,
Thorben

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

18

I think the '5' in the HP5set_fapl_core means to allocate only 5 more
bytes each time it needs to allocate. I think that is too small and
maybe a problem since its an odd number and not a power of 2.

Although I would not necessarily expect that to have failed in the way
you are seeing either.

Try '5<<20' instead of '5'

I found this in h5test.c, function h5_fileaccess()

/* In-core temporary file with 1MB increment */
  if (H5Pset_fapl_core(fapl, (size_t)1, TRUE)<0) return -1;

Anyway, using a number like 5<<20 did not make a difference, it still
aborts...

Cheers,
Thorben

···

On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:

Other than that, I don't see anything wrong with what your are doing.
Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your
file into core is failing due to too large a request. I guess that is
possible. To see if that is happening, if your HDF5 installation was
compiled with tracing/debugging features turned on, then try setting
HDF5_DEBUG=all in your enviornment where you are running your
application.

Mark

On Wed, 2010-03-03 at 09:07, Thorben Kröger wrote:
> > > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> > > > Hello Thorben,
> > > >
> > > > However, did you in fact try using HDF5-1.8.4
> > > > and reading the file with the CORE vfd (H5Pset_fapl_core)
>
> instead of opening my file like this:
>
> hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY,
> H5P_DEFAULT);
>
> I am now doing this:
>
> hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
> if(fapl < 0) {throw std::runtime_error("fapl < 0");}
> hid_t ret = H5Pset_fapl_core(fapl, 5/*MB?*/, 0 /*no backing store*/);
> if(ret < 0) {throw std::runtime_error("could not create fapl core");}
> hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, fapl);
>
> But using the second way, it cannot open any of my datasets:
>
> HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
> #000: H5D.c line 321 in H5Dopen2(): not found
>
> major: Dataset
> minor: Object not found
>
> #001: H5Gloc.c line 468 in H5G_loc_find(): can't find object
>
> major: Symbol table
> minor: Object not found
>
> #002: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal
>
> failed
>
> major: Symbol table
> minor: Object not found
>
> #003: H5Gtraverse.c line 665 in H5G_traverse_real(): can't look up
> component
>
> major: Symbol table
> minor: Object not found
>
> #004: H5Gobj.c line 1047 in H5G_obj_lookup(): can't locate object
>
> major: Symbol table
> minor: Object not found
>
> #005: H5Gstab.c line 860 in H5G_stab_lookup(): not found
>
> major: Symbol table
> minor: Object not found
>
> #006: H5B.c line 360 in H5B_find(): can't lookup key in leaf node
>
> major: B-Tree node
> minor: Object not found
>
> #007: H5Gnode.c line 522 in H5G_node_found(): unable to protect symbol
> table
>
> node
>
> major: Symbol table
> minor: Unable to load metadata into cache
>
> #008: H5AC.c line 1831 in H5AC_protect(): H5C_protect() failed.
>
> major: Object cache
> minor: Unable to protect metadata
>
> #009: H5C.c line 6160 in H5C_protect(): can't load entry
>
> major: Object cache
> minor: Unable to load metadata into cache
>
> #010: H5C.c line 10990 in H5C_load_entry(): unable to load entry
>
> major: Object cache
> minor: Unable to load metadata into cache
>
> #011: H5Gcache.c line 176 in H5G_node_load(): bad symbol table node
>
> signature
>
> major: Symbol table
> minor: Unable to load metadata into cache
>
> I'm at a loss now, what am I doing wrong here?
>
> Cheers,
> Thorben
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

19

I think the comment in h5test.c is misleading.

The documentation for H5Pset_fapl_core says the inc argument is the
number of BYTES, not MEGAbytes.

I'd bet the initial malloc is failing for reading your whole file.
Sorry. Didn't expect that. But, I was thinking in terms of 64 bits and
30 gigs 5 bits too many for a 32 bit number. So, if you are running in
truly 64 bit, then I am completely puzzled. Otherwise, if you are
running 32 bit, then I think the problem has presented itself; we can't
address 30 gig buffer in a 32 bit application.

I'd suggest looking at some of the other properties such as

H5Pset_small_data_block_size
HP5set_buffer (used during read/xform only)
H5Pset_cache (relevant only for chunked datasets)
H5Pset_meta_block_size( hid_t fapl_id, hsize_t size )

and see if setting any/all of these to large numbers; 100's of megabytes
has any positive impact.

Mark

···

On Wed, 2010-03-03 at 09:30, Thorben Kr�ger wrote:

On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:
> I think the '5' in the HP5set_fapl_core means to allocate only 5 more
> bytes each time it needs to allocate. I think that is too small and
> maybe a problem since its an odd number and not a power of 2.
>
> Although I would not necessarily expect that to have failed in the way
> you are seeing either.
>
> Try '5<<20' instead of '5'

I found this in h5test.c, function h5_fileaccess()

/* In-core temporary file with 1MB increment */
  if (H5Pset_fapl_core(fapl, (size_t)1, TRUE)<0) return -1;

Anyway, using a number like 5<<20 did not make a difference, it still
aborts...

Cheers,
Thorben

>
> Other than that, I don't see anything wrong with what your are doing.
> Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your
> file into core is failing due to too large a request. I guess that is
> possible. To see if that is happening, if your HDF5 installation was
> compiled with tracing/debugging features turned on, then try setting
> HDF5_DEBUG=all in your enviornment where you are running your
> application.
>
> Mark
>
> On Wed, 2010-03-03 at 09:07, Thorben Kr�ger wrote:
> > > > On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:
> > > > > Hello Thorben,
> > > > >
> > > > > However, did you in fact try using HDF5-1.8.4
> > > > > and reading the file with the CORE vfd (H5Pset_fapl_core)
> >
> > instead of opening my file like this:
> >
> > hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY,
> > H5P_DEFAULT);
> >
> > I am now doing this:
> >
> > hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
> > if(fapl < 0) {throw std::runtime_error("fapl < 0");}
> > hid_t ret = H5Pset_fapl_core(fapl, 5/*MB?*/, 0 /*no backing store*/);
> > if(ret < 0) {throw std::runtime_error("could not create fapl core");}
> > hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, fapl);
> >
> > But using the second way, it cannot open any of my datasets:
> >
> > HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
> > #000: H5D.c line 321 in H5Dopen2(): not found
> >
> > major: Dataset
> > minor: Object not found
> >
> > #001: H5Gloc.c line 468 in H5G_loc_find(): can't find object
> >
> > major: Symbol table
> > minor: Object not found
> >
> > #002: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal
> >
> > failed
> >
> > major: Symbol table
> > minor: Object not found
> >
> > #003: H5Gtraverse.c line 665 in H5G_traverse_real(): can't look up
> > component
> >
> > major: Symbol table
> > minor: Object not found
> >
> > #004: H5Gobj.c line 1047 in H5G_obj_lookup(): can't locate object
> >
> > major: Symbol table
> > minor: Object not found
> >
> > #005: H5Gstab.c line 860 in H5G_stab_lookup(): not found
> >
> > major: Symbol table
> > minor: Object not found
> >
> > #006: H5B.c line 360 in H5B_find(): can't lookup key in leaf node
> >
> > major: B-Tree node
> > minor: Object not found
> >
> > #007: H5Gnode.c line 522 in H5G_node_found(): unable to protect symbol
> > table
> >
> > node
> >
> > major: Symbol table
> > minor: Unable to load metadata into cache
> >
> > #008: H5AC.c line 1831 in H5AC_protect(): H5C_protect() failed.
> >
> > major: Object cache
> > minor: Unable to protect metadata
> >
> > #009: H5C.c line 6160 in H5C_protect(): can't load entry
> >
> > major: Object cache
> > minor: Unable to load metadata into cache
> >
> > #010: H5C.c line 10990 in H5C_load_entry(): unable to load entry
> >
> > major: Object cache
> > minor: Unable to load metadata into cache
> >
> > #011: H5Gcache.c line 176 in H5G_node_load(): bad symbol table node
> >
> > signature
> >
> > major: Symbol table
> > minor: Unable to load metadata into cache
> >
> > I'm at a loss now, what am I doing wrong here?
> >
> > Cheers,
> > Thorben
> >
> > _______________________________________________
> > Hdf-forum is for HDF software users discussion.
> > Hdf-forum@hdfgroup.org
> > http://**mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

20

I think the comment in h5test.c is misleading.

The documentation for H5Pset_fapl_core says the inc argument is the
number of BYTES, not MEGAbytes.

I'd bet the initial malloc is failing for reading your whole file.
Sorry. Didn't expect that. But, I was thinking in terms of 64 bits and
30 gigs 5 bits too many for a 32 bit number. So, if you are running in
truly 64 bit, then I am completely puzzled. Otherwise, if you are
running 32 bit, then I think the problem has presented itself; we can't
address 30 gig buffer in a 32 bit application.

I'd suggest looking at some of the other properties such as

H5Pset_small_data_block_size
HP5set_buffer (used during read/xform only)
H5Pset_cache (relevant only for chunked datasets)
H5Pset_meta_block_size( hid_t fapl_id, hsize_t size )

and see if setting any/all of these to large numbers; 100's of megabytes
has any positive impact.

  Those probably won't have any effect, I'm guessing that you are correct that the machine ran out of memory (or Thorben has a quota about the amount of RAM his process can use).

  Quincey

···

On Mar 3, 2010, at 11:49 AM, Mark Miller wrote:

Mark

On Wed, 2010-03-03 at 09:30, Thorben Kröger wrote:

On Wednesday 03 March 2010 18:17:31 Mark Miller wrote:

I think the '5' in the HP5set_fapl_core means to allocate only 5 more
bytes each time it needs to allocate. I think that is too small and
maybe a problem since its an odd number and not a power of 2.

Although I would not necessarily expect that to have failed in the way
you are seeing either.

Try '5<<20' instead of '5'

I found this in h5test.c, function h5_fileaccess()

/* In-core temporary file with 1MB increment */
  if (H5Pset_fapl_core(fapl, (size_t)1, TRUE)<0) return -1;

Anyway, using a number like 5<<20 did not make a difference, it still
aborts...

Cheers,
Thorben

Other than that, I don't see anything wrong with what your are doing.
Except that maybe allocating the 30 gigabyte buffer in HDF5 to read your
file into core is failing due to too large a request. I guess that is
possible. To see if that is happening, if your HDF5 installation was
compiled with tracing/debugging features turned on, then try setting
HDF5_DEBUG=all in your enviornment where you are running your
application.

Mark

On Wed, 2010-03-03 at 09:07, Thorben Kröger wrote:

On Wednesday 03 March 2010 16:47:20 Mark Miller wrote:

Hello Thorben,

However, did you in fact try using HDF5-1.8.4
and reading the file with the CORE vfd (H5Pset_fapl_core)

instead of opening my file like this:

hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY,
H5P_DEFAULT);

I am now doing this:

hid_t fapl = H5Pcreate(H5P_FILE_ACCESS);
if(fapl < 0) {throw std::runtime_error("fapl < 0");}
hid_t ret = H5Pset_fapl_core(fapl, 5/*MB?*/, 0 /*no backing store*/);
if(ret < 0) {throw std::runtime_error("could not create fapl core");}
hdf5geometryFile_ = H5Fopen(filename.c_str(), H5F_ACC_RDONLY, fapl);

But using the second way, it cannot open any of my datasets:

HDF5-DIAG: Error detected in HDF5 (1.8.4) thread 0:
#000: H5D.c line 321 in H5Dopen2(): not found

   major: Dataset
   minor: Object not found

#001: H5Gloc.c line 468 in H5G_loc_find(): can't find object

   major: Symbol table
   minor: Object not found

#002: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal

failed

   major: Symbol table
   minor: Object not found

#003: H5Gtraverse.c line 665 in H5G_traverse_real(): can't look up
component

   major: Symbol table
   minor: Object not found

#004: H5Gobj.c line 1047 in H5G_obj_lookup(): can't locate object

   major: Symbol table
   minor: Object not found

#005: H5Gstab.c line 860 in H5G_stab_lookup(): not found

   major: Symbol table
   minor: Object not found

#006: H5B.c line 360 in H5B_find(): can't lookup key in leaf node

   major: B-Tree node
   minor: Object not found

#007: H5Gnode.c line 522 in H5G_node_found(): unable to protect symbol
table

node

   major: Symbol table
   minor: Unable to load metadata into cache

#008: H5AC.c line 1831 in H5AC_protect(): H5C_protect() failed.

   major: Object cache
   minor: Unable to protect metadata

#009: H5C.c line 6160 in H5C_protect(): can't load entry

   major: Object cache
   minor: Unable to load metadata into cache

#010: H5C.c line 10990 in H5C_load_entry(): unable to load entry

   major: Object cache
   minor: Unable to load metadata into cache

#011: H5Gcache.c line 176 in H5G_node_load(): bad symbol table node

signature

   major: Symbol table
   minor: Unable to load metadata into cache

I'm at a loss now, what am I doing wrong here?

Cheers,
Thorben

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://**mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org