hdf5 header files issues

Seems like I had found an issue with header files in hdf5 and it
explains why examples files are not compiled property

C:\Program Files (x86)\HDF5 1.8.6\include>grep H5DOpen2 *.h

C:\Program Files (x86)\HDF5 1.8.6\include>grep H5Dopen2 *.h
H5Dpublic.h:H5_DLL hid_t H5Dopen2(hid_t file_id, const char *name, hid_t
dapl_id
);
H5public.h: * if((dset = H5Dopen2(file, name)) < 0)
H5version.h:#define H5Dopen H5Dopen2

Why are these macro calls inconsistent??

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of
hdf-forum-request@hdfgroup.org
Sent: Wed., May. 04, 2011 2:03 PM
To: hdf-forum@hdfgroup.org
Subject: Hdf-forum Digest, Vol 23, Issue 5

Send Hdf-forum mailing list submissions to
  hdf-forum@hdfgroup.org

To subscribe or unsubscribe via the World Wide Web, visit
  http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
or, via email, send a message with subject or body 'help' to
  hdf-forum-request@hdfgroup.org

You can reach the person managing the list at
  hdf-forum-owner@hdfgroup.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."

Today's Topics:

   1. Re: How does chunked reading/writing interact with setting
      hyperslabs in datasets? (Quincey Koziol)
   2. Re: chunks of different sizes (Quincey Koziol)
   3. Re: behavior of H5Sselect_elements() (Quincey Koziol)
   4. Re: Attributes vs Datasets (Quincey Koziol)
   5. Re: Problem with getting hdf5 file size (Quincey Koziol)
   6. Re: Attributes vs Datasets (Rob Latham)
   7. Re: Attributes vs Datasets (Quincey Koziol)
   8. Re: Problem with getting hdf5 file size (dsentenac@interfree.it)

----------------------------------------------------------------------

Message: 1
Date: Wed, 4 May 2011 11:06:00 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] How does chunked reading/writing interact
  with setting hyperslabs in datasets?
Message-ID: <8E67111E-15AF-4A26-8988-C0C613A95368@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi Kent,

On May 3, 2011, at 11:20 AM, Williams, Norman K wrote:

OK.

I'm not entirely clear how to coordinate different memory and disk
footprints, particularly when using the C++ interface.

The steps I know about:

1. Open the file
2. Open the DataSet.
3. Obtain the DataSpace from the Dataset.
4. Select a hyperslab to read data into a voxel buffer.

I select a hyperslab from the dataset to read based on the image

region

specified by the user. It's a little more complicated than that in

ITK,

but that summarizes the process.

At what stage in the process do I specify a different in-memory

dataspace?

  The voxel buffer is the buffer that you want to describe with
the in-memory dataspace. So, you'd define a new dataspace (with
H5Screate) that specified the correct dimensions for your buffer.

And how do I specify it's position in the larger on-disk dataspace?

  You specify a selection in the on-disk dataspace (probably with
H5Sselect_all, H5Sselect_hyperslab, H5Sselect_elements, etc) and a
selection in the in-memory dataspace (in the same way) and when you call
H5Dread, the elements are transferred from the on-disk dataset into the
memory buffer.

If there's example code in the manual, just point me there.

  I think Binh-Minh already pointed you to the C++ examples, and
you can search for H5Sselect_hyperslab in the 'examples' subdirectory of
the HDF5 distribution.

  Quincey

On 5/3/11 10:56 AM, "Quincey Koziol" <koziol@hdfgroup.org> wrote:

Hi Kent,

On May 3, 2011, at 10:16 AM, Williams, Norman K wrote:

I am using the H5Sselect_hyperslab method for both reading and

writing

already. Additionally when I write out an image, I 'chunk' the

output

and
compress it.

The question I have is what I can do to minimize the memory

footprint.

Based on my reading in the documentation, the Hyperslab interface
actually
scatters/gathers to/from an in-memory dataset, leading me to believe

the

entire dataset will be allocated in system memory.

So the question is how would I use HDF5 in such a way as to minimize

the

memory footprint in this context?

  You are allowed to create a memory dataspace that is different from
the dataset's dataspace in the file. That will allow you to tune the
memory footprint.

________________________________
Notice: This UI Health Care e-mail (including attachments) is covered

by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is
confidential and may be legally privileged. If you are not the intended
recipient, you are hereby notified that any retention, dissemination,
distribution, or copying of this communication is strictly prohibited.
Please reply to the sender that you have received the message in error,
then delete it. Thank you.

________________________________

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

------------------------------

Message: 2
Date: Wed, 4 May 2011 11:08:00 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] chunks of different sizes
Message-ID: <E71A76D7-8D94-493E-A8EA-F2A51577133B@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi Eduardo,

On May 3, 2011, at 12:36 PM, Eduardo wrote:

Different chunk sizes would be interesting to avoid IO contention

when

the hyperslabs are not regular.

So, if my hyperslabs are not regular (similarly to my previous
example), does my application suffer IO contention? I mean, should I
avoid irregular hyperslabs, even if there is no overlap or gaps?

  Yes, independent access I/O performance will be best when there
is a direct mapping of each process's selection to 1+ chunks and those
chunks aren't accessed by other processes. Enabling collective I/O
access may reduce or eliminate the penalty for accessing one chunk with
multiple processes however.

  Quincey

------------------------------

Message: 3
Date: Wed, 4 May 2011 11:09:25 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Cc: Prabhat <prabhat@hpcrd.lbl.gov>
Subject: Re: [Hdf-forum] behavior of H5Sselect_elements()
Message-ID: <21B43F56-53D0-4BF4-A9E1-23D36C7C7C70@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi Mark,

On May 3, 2011, at 4:22 PM, Mark Howison wrote:

Hi all,

We use this call in H5hut to allow the user to do a point selection on
a dataset. Previously, I thought it worked like in this example:

1) I call H5Sselect_elements() with the H5S_SELECT_SET operation, an
"coord" array of [0,3,6,9] and num_elements=4
2) I do a H5Dread to a buffer with enough space for 4 elements, and I
get back a *packed* array with elements 0,3,6,9 and nothing in between

Recently though our regression test for this in H5hut failed with
1.8.5 and 1.8.6, and when I looked closer I realized that H5Dread is
giving me a sparsely packed array that wants to be 10 elements long
with only 0,3,6,9 filled in and the others left untouched.

Did the behavior of H5Sselect_elements() change in a recent release,
or has there always been something wrong with our regression test, and
I never should have gotten back a packed array from H5Dread?

  Nope, H5Sselect_elements has never returned a packed array. I'd
lean toward an incorrect regression test. (Although it's always
possible to have a bug in the HDF5 library, also)

  Quincey

------------------------------

Message: 4
Date: Wed, 4 May 2011 11:13:08 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Attributes vs Datasets
Message-ID: <98478D4D-8F6F-4EAD-B52C-8EED2F23F1AB@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi John,

On May 4, 2011, at 2:10 AM, Biddiscombe, John A. wrote:

If I know that I want to attach a couple of arrays as attributes, and

these arrays may be say 1024 elements of double precision max. (say
8kB). The datasets themselves might be xGB.

1) Can I tell hdf5 to reserve a minimum size in the object header so

that I know in advance that the attributes will fit (or does the object
header only contain pointers to other structures anyway)

  This feature isn't available through the public API currently.
(I have thought about exposing the internal part of the library that
would allow it, but it seemed like it would only be used by a very small
portion of the user base)

2) Is there any performance advantage in terms of fewer metadata

transaction to using attributes over datasets in a parallel IO context.
(file is written in parallel, only rank zero needs to actually write the
attributes)

  Hmm, as usual, it depends. :slight_smile: Are all the processes going to
be accessing the attribute? If not, you could create an attribute with
an object reference to an auxiliary dataset, and then read that in
dataset in when needed.

  Quincey

Question motivated by ....

From the docs ...
"The HDF5 format and I/O library are designed with the assumption that

attributes are small datasets. They are always stored in the object
header of the object they are attached to. Because of this, large
datasets should not be stored as attributes. How large is "large" is not
defined by the library and is up to the user's interpretation. (Large
datasets with metadata can be stored as supplemental datasets in a group
with the primary dataset.)"

Thanks

JB

--
John Biddiscombe, email:biddisco @ cscs.ch
http://www.cscs.ch/
CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

------------------------------

Message: 5
Date: Wed, 4 May 2011 11:16:29 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Problem with getting hdf5 file size
Message-ID: <FBEB3430-C0BC-455B-901E-D4251E89DF86@hdfgroup.org>
Content-Type: text/plain; charset="us-ascii"

Hi Daniel,

On May 4, 2011, at 3:13 AM, Sentenac Daniel wrote:

Hello,
I am currently doing a very small test to retreive the size of a just

created HDF5 file using the function

H5Fget_filesize and cannot understand why the size returned is

actually different from the real one.

Here is the simple example piece of code:

hsize_t size;
hid_t file_id = H5Fcreate("file.hdf5", H5F_ACC_TRUNC, H5P_DEFAULT,

H5P_DEFAULT);

H5Fget_filesize( file_id, &size );

The real size of the file appears to be 800 bytes, while the size

returned fromH5Fget_filesize is 2144 bytes.

  The library allocates a 2KB buffer to sub-allocate metadata from
(in addition to the ~100 byte superblock for the file), but then
truncates the file's size down to the actual amount of data used in the
file, when the file is closed.

I would like to use the function H5Fget_vfd_handle to get a (char*)

buffer copy of the file. Can someone help me to do that correctly ?

  I would recommend against this sort of mucking with the bytes -
the HDF5 library does a lot of work behind the scenes and you probably
won't get a coherent picture of the file by just looking at the bytes.

  That said, why do you want to do this? Possibly the "core" VFD
(H5Pset_fapl_core) would meet your needs?

  Quincey

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/2
0110504/1a06b74b/attachment-0001.html>

------------------------------

Message: 6
Date: Wed, 4 May 2011 11:21:30 -0500
From: Rob Latham <robl@mcs.anl.gov>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Attributes vs Datasets
Message-ID: <20110504162130.GD15956@mcs.anl.gov>
Content-Type: text/plain; charset=us-ascii

On Wed, May 04, 2011 at 11:13:08AM -0500, Quincey Koziol wrote:

> 1) Can I tell hdf5 to reserve a minimum size in the object header so

that I know in advance that the attributes will fit (or does the object
header only contain pointers to other structures anyway)

  This feature isn't available through the public API currently.

(I have thought about exposing the internal part of the library that
would allow it, but it seemed like it would only be used by a very small
portion of the user base)

Seems like something the NetCDF-4 folks could use to good effect, if
they don't already. After exiting NetCDF define mode, the size of the
attributes and objects will be known. NetCDF callers are familiar with
the potential pain of re-entering define mode.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

------------------------------

Message: 7
Date: Wed, 4 May 2011 11:32:28 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Attributes vs Datasets
Message-ID: <46648BAC-3223-4F90-8883-5237FA3DBDEB@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

On May 4, 2011, at 11:21 AM, Rob Latham wrote:

On Wed, May 04, 2011 at 11:13:08AM -0500, Quincey Koziol wrote:

1) Can I tell hdf5 to reserve a minimum size in the object header so

that I know in advance that the attributes will fit (or does the object
header only contain pointers to other structures anyway)

  This feature isn't available through the public API currently.

(I have thought about exposing the internal part of the library that
would allow it, but it seemed like it would only be used by a very small
portion of the user base)

Seems like something the NetCDF-4 folks could use to good effect, if
they don't already. After exiting NetCDF define mode, the size of the
attributes and objects will be known. NetCDF callers are familiar with
the potential pain of re-entering define mode.

  Ah, good point. (Although adding new attributes to an existing
HDF5 object can be done at any point, without any of the pain that
netCDF has for re-entering define mode)

  Quincey

------------------------------

Message: 8
Date: 4 May 2011 18:02:45 -0000
From: dsentenac@interfree.it
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Problem with getting hdf5 file size
Message-ID: <20110504180245.7961.qmail@community1.interfree.it>
Content-Type: text/plain; charset="us-ascii"

An HTML attachment was scrubbed...
URL:
<http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/2
0110504/91d6d1a7/attachment.html>

------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

End of Hdf-forum Digest, Vol 23, Issue 5
****************************************

Hello Stanislav,

Seems like I had found an issue with header files in hdf5 and it
explains why examples files are not compiled property

C:\Program Files (x86)\HDF5 1.8.6\include>grep H5DOpen2 *.h

C:\Program Files (x86)\HDF5 1.8.6\include>grep H5Dopen2 *.h
H5Dpublic.h:H5_DLL hid_t H5Dopen2(hid_t file_id, const char *name, hid_t
dapl_id

This (above) is the correct signature for H5Dopen2

);
H5public.h: * if((dset = H5Dopen2(file, name)) < 0)

This statement is actually in a comment on how to test status
return values, so is not part of the code. We could change
that name to something else so that it does not get confused
with actual APIs (H5Xxxx ?).

What kind of problems are you encountering with the example
programs?

-Barbara

···

On Wed, 4 May 2011, Stanislav.Seltser@sungard.com wrote:

H5version.h:#define H5Dopen H5Dopen2

Why are these macro calls inconsistent??

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of
hdf-forum-request@hdfgroup.org
Sent: Wed., May. 04, 2011 2:03 PM
To: hdf-forum@hdfgroup.org
Subject: Hdf-forum Digest, Vol 23, Issue 5

Send Hdf-forum mailing list submissions to
  hdf-forum@hdfgroup.org

To subscribe or unsubscribe via the World Wide Web, visit
  http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
or, via email, send a message with subject or body 'help' to
  hdf-forum-request@hdfgroup.org

You can reach the person managing the list at
  hdf-forum-owner@hdfgroup.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."

Today's Topics:

  1. Re: How does chunked reading/writing interact with setting
     hyperslabs in datasets? (Quincey Koziol)
  2. Re: chunks of different sizes (Quincey Koziol)
  3. Re: behavior of H5Sselect_elements() (Quincey Koziol)
  4. Re: Attributes vs Datasets (Quincey Koziol)
  5. Re: Problem with getting hdf5 file size (Quincey Koziol)
  6. Re: Attributes vs Datasets (Rob Latham)
  7. Re: Attributes vs Datasets (Quincey Koziol)
  8. Re: Problem with getting hdf5 file size (dsentenac@interfree.it)

----------------------------------------------------------------------

Message: 1
Date: Wed, 4 May 2011 11:06:00 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] How does chunked reading/writing interact
  with setting hyperslabs in datasets?
Message-ID: <8E67111E-15AF-4A26-8988-C0C613A95368@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi Kent,

On May 3, 2011, at 11:20 AM, Williams, Norman K wrote:

OK.

I'm not entirely clear how to coordinate different memory and disk
footprints, particularly when using the C++ interface.

The steps I know about:

1. Open the file
2. Open the DataSet.
3. Obtain the DataSpace from the Dataset.
4. Select a hyperslab to read data into a voxel buffer.

I select a hyperslab from the dataset to read based on the image

region

specified by the user. It's a little more complicated than that in

ITK,

but that summarizes the process.

At what stage in the process do I specify a different in-memory

dataspace?

  The voxel buffer is the buffer that you want to describe with
the in-memory dataspace. So, you'd define a new dataspace (with
H5Screate) that specified the correct dimensions for your buffer.

And how do I specify it's position in the larger on-disk dataspace?

  You specify a selection in the on-disk dataspace (probably with
H5Sselect_all, H5Sselect_hyperslab, H5Sselect_elements, etc) and a
selection in the in-memory dataspace (in the same way) and when you call
H5Dread, the elements are transferred from the on-disk dataset into the
memory buffer.

If there's example code in the manual, just point me there.

  I think Binh-Minh already pointed you to the C++ examples, and
you can search for H5Sselect_hyperslab in the 'examples' subdirectory of
the HDF5 distribution.

  Quincey

On 5/3/11 10:56 AM, "Quincey Koziol" <koziol@hdfgroup.org> wrote:

Hi Kent,

On May 3, 2011, at 10:16 AM, Williams, Norman K wrote:

I am using the H5Sselect_hyperslab method for both reading and

writing

already. Additionally when I write out an image, I 'chunk' the

output

and
compress it.

The question I have is what I can do to minimize the memory

footprint.

Based on my reading in the documentation, the Hyperslab interface
actually
scatters/gathers to/from an in-memory dataset, leading me to believe

the

entire dataset will be allocated in system memory.

So the question is how would I use HDF5 in such a way as to minimize

the

memory footprint in this context?

  You are allowed to create a memory dataspace that is different from
the dataset's dataspace in the file. That will allow you to tune the
memory footprint.

________________________________
Notice: This UI Health Care e-mail (including attachments) is covered

by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is
confidential and may be legally privileged. If you are not the intended
recipient, you are hereby notified that any retention, dissemination,
distribution, or copying of this communication is strictly prohibited.
Please reply to the sender that you have received the message in error,
then delete it. Thank you.

________________________________

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

------------------------------

Message: 2
Date: Wed, 4 May 2011 11:08:00 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] chunks of different sizes
Message-ID: <E71A76D7-8D94-493E-A8EA-F2A51577133B@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi Eduardo,

On May 3, 2011, at 12:36 PM, Eduardo wrote:

Different chunk sizes would be interesting to avoid IO contention

when

the hyperslabs are not regular.

So, if my hyperslabs are not regular (similarly to my previous
example), does my application suffer IO contention? I mean, should I
avoid irregular hyperslabs, even if there is no overlap or gaps?

  Yes, independent access I/O performance will be best when there
is a direct mapping of each process's selection to 1+ chunks and those
chunks aren't accessed by other processes. Enabling collective I/O
access may reduce or eliminate the penalty for accessing one chunk with
multiple processes however.

  Quincey

------------------------------

Message: 3
Date: Wed, 4 May 2011 11:09:25 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Cc: Prabhat <prabhat@hpcrd.lbl.gov>
Subject: Re: [Hdf-forum] behavior of H5Sselect_elements()
Message-ID: <21B43F56-53D0-4BF4-A9E1-23D36C7C7C70@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi Mark,

On May 3, 2011, at 4:22 PM, Mark Howison wrote:

Hi all,

We use this call in H5hut to allow the user to do a point selection on
a dataset. Previously, I thought it worked like in this example:

1) I call H5Sselect_elements() with the H5S_SELECT_SET operation, an
"coord" array of [0,3,6,9] and num_elements=4
2) I do a H5Dread to a buffer with enough space for 4 elements, and I
get back a *packed* array with elements 0,3,6,9 and nothing in between

Recently though our regression test for this in H5hut failed with
1.8.5 and 1.8.6, and when I looked closer I realized that H5Dread is
giving me a sparsely packed array that wants to be 10 elements long
with only 0,3,6,9 filled in and the others left untouched.

Did the behavior of H5Sselect_elements() change in a recent release,
or has there always been something wrong with our regression test, and
I never should have gotten back a packed array from H5Dread?

  Nope, H5Sselect_elements has never returned a packed array. I'd
lean toward an incorrect regression test. (Although it's always
possible to have a bug in the HDF5 library, also)

  Quincey

------------------------------

Message: 4
Date: Wed, 4 May 2011 11:13:08 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Attributes vs Datasets
Message-ID: <98478D4D-8F6F-4EAD-B52C-8EED2F23F1AB@hdfgroup.org>
Content-Type: text/plain; charset=us-ascii

Hi John,

On May 4, 2011, at 2:10 AM, Biddiscombe, John A. wrote:

If I know that I want to attach a couple of arrays as attributes, and

these arrays may be say 1024 elements of double precision max. (say
8kB). The datasets themselves might be xGB.

1) Can I tell hdf5 to reserve a minimum size in the object header so

that I know in advance that the attributes will fit (or does the object
header only contain pointers to other structures anyway)

  This feature isn't available through the public API currently.
(I have thought about exposing the internal part of the library that
would allow it, but it seemed like it would only be used by a very small
portion of the user base)

2) Is there any performance advantage in terms of fewer metadata

transaction to using attributes over datasets in a parallel IO context.
(file is written in parallel, only rank zero needs to actually write the
attributes)

  Hmm, as usual, it depends. :slight_smile: Are all the processes going to
be accessing the attribute? If not, you could create an attribute with
an object reference to an auxiliary dataset, and then read that in
dataset in when needed.

  Quincey

Question motivated by ....

From the docs ...
"The HDF5 format and I/O library are designed with the assumption that

attributes are small datasets. They are always stored in the object
header of the object they are attached to. Because of this, large
datasets should not be stored as attributes. How large is "large" is not
defined by the library and is up to the user's interpretation. (Large
datasets with metadata can be stored as supplemental datasets in a group
with the primary dataset.)"

Thanks

JB

--
John Biddiscombe, email:biddisco @ cscs.ch
http://www.cscs.ch/
CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07
Via Cantonale, 6928 Manno, Switzerland | Fax: +41 (91) 610.82.82

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

------------------------------

Message: 5
Date: Wed, 4 May 2011 11:16:29 -0500
From: Quincey Koziol <koziol@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Problem with getting hdf5 file size
Message-ID: <FBEB3430-C0BC-455B-901E-D4251E89DF86@hdfgroup.org>
Content-Type: text/plain; charset="us-ascii"

Hi Daniel,

On May 4, 2011, at 3:13 AM, Sentenac Daniel wrote:

Hello,
I am currently doing a very small test to retreive the size of a just

created HDF5 file using the function

H5Fget_filesize and cannot understand why the size returned is

actually different from the real one.

Here is the simple example piece of code:

hsize_t size;
hid_t file_id = H5Fcreate("file.hdf5", H5F_ACC_TRUNC, H5P_DEFAULT,

H5P_DEFAULT);

H5Fget_filesize( file_id, &size );

The real size of the file appears to be 800 bytes, while the size

returned fromH5Fget_filesize is 2144 bytes.

  The library allocates a 2KB buffer to sub-allocate metadata from
(in addition to the ~100 byte superblock for the file), but then
truncates the file's size down to the actual amount of data used in the
file, when the file is closed.

I would like to use the function H5Fget_vfd_handle to get a (char*)

buffer copy of the file. Can someone help me to do that correctly ?

  I would recommend against this sort of mucking with the bytes -
the HDF5 library does a lot of work behind the scenes and you probably
won't get a coherent picture of the file by just looking at the bytes.

  That said, why do you want to do this? Possibly the "core" VFD
(H5Pset_fapl_core) would meet your needs?

  Quincey

-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.hdfgroup.org/pipermail/hdf-forum_hdfgroup.org/attachments/2
0110504/1a06b74b/attachment-0001.html>

------------------------------

Message: 6
Date: Wed, 4 May 2011 11:21:30 -0500
From: Rob Latham <robl@mcs.anl.gov>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Attributes vs Datasets
Message-ID: <20110504162130.GD15956@mcs.anl.gov>
Content-Type: text/plain; charset=us-ascii

On Wed, May 04, 2011 at 11:13:08AM -0500, Quincey Koziol wrote:

1) Can I tell hdf5 to reserve a minimum size in the object header so

that I know in advance that the attributes will fit (or does the object
header only contain pointers to other structures anyway)

  This feature isn't available through the public API currently.

(I have thought about exposing the internal part of the library that
would allow it, but it seemed like it would only be used by a very small
portion of the user base)

Seems like something the NetCDF-4 folks could use to good effect, if
they don't already. After exiting NetCDF define mode, the size of the
attributes and objects will be known. NetCDF callers are familiar with
the potential pain of re-entering define mode.

==rob

--
Barbara Jones
bljones@hdfgroup.org