writing part of a compound datatype

Hdf-forum archives
1

I'd like to pursue a particular detail of the compound datatype thread from
yesterday. Specifically, the ability to write part of a compound datatype,
rather than all of it in one go. I've spent some time today trying to
figure out what's involved, but haven't got a clear idea. I'm assuming that
the writing would be accomplished by calling *H5Dwrite* and passing it the
id of a dataset created using *H5Dcreate2*() - (I'm cribbing from the
listing of the sample program *h5_compound*). *H5Dwrite*() has a variety of
arguments, but none that obviously give me a mechanism for selecting what
part of the dataset is to be written. I've got a couple of guesses:
1. there's some work to be done on the dataset prior to the write that
selects what is supposed to be written, or
2. the property list argument can somehow convey the information.

Any guidance would be gratefully received.

-Josiah

2

Hi,
i'm not quite sure that i understand your question?

You are trying to:
1. write part of a compound datatype. - "the ability to write part of a
compound datatype, rather than all of it in one go."
2. write part of a dataset. - "but none that obviously
give me a mechanism for selecting what part of the dataset is to be
written."

As to the first one, i'm not familiar with a way to do it. (Maybe there is
a way, i don't know)

The second one:
The signature of the H5Dwrite function is:
herr_t H5Dwrite(hid_t dataset_id, hid_t mem_type_id, hid_t mem_space_id,
hid_t file_space_id, hid_t xfer_plist_id, const void * buf )

In order to specify which part of the dataset is the passed data referenced
by the buf pointer (essentially where in the dataset to store the provided
data), you should call
herr_t H5Sselect_hyperslab(hid_t space_id, H5S_seloper_t op, const hsize_t
*start, const hsize_t *stride, const hsize_t *count, const hsize_t *block )
before H5Dwrite with the same space_id argument as file_space_id
in H5Dwrite.

Eexample:
Assuming you want to store an array([500]) of compound datatype
(structures) from the beginning of the dataset to the 500th element.

// Create file_space, mem_space, mem_type, dataset, etc.

hsize_t start = 0;
hsize_t stride = 0;
hsize_t count = 500;
hsize_t block = 0;

H5Sselect_hyperslab( file_space_id, H5S_SELECT_SET, &start, &stride,
&count, &block );
H5Dwrite( dataset_id, mem_type_id, mem_space_id, file_space_id,
xfer_plist_id, data );

More info about H5Sselect_hyperslab at:
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5S.html#Dataspace-SelectHyperslab

Best Regards,
V. Daskalov

I'd like to pursue a particular detail of the compound datatype thread

from yesterday. Specifically, the ability to write part of a compound
datatype, rather than all of it in one go. I've spent some time today
trying to figure out what's involved, but haven't got a clear idea. I'm
assuming that the writing would be accomplished by calling H5Dwrite and
passing it the id of a dataset created using H5Dcreate2() - (I'm cribbing
from the listing of the sample program h5_compound). H5Dwrite() has a
variety of arguments, but none that obviously give me a mechanism for
selecting what part of the dataset is to be written. I've got a couple of
guesses:

1. there's some work to be done on the dataset prior to the write that

selects what is supposed to be written, or

2. the property list argument can somehow convey the information.

Any guidance would be gratefully received.

-Josiah

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

···

On 18 July 2013 00:13, Josiah Slack <josiahnmi@gmail.com> wrote:

3

Reading or writing a compound type partially can be controlled via the
mem_type_id argument. Let's say you have a compound type in the file
with components A, B, and C. If you'd like to read only A or write to A,
just construct an in memory compound type with just an A component.
The library will match components by name between the file type and the
memory type.

G.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
Josiah Slack
Sent: Thursday, July 18, 2013 7:30 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] writing part of a compound datatype

On Thu, Jul 18, 2013 at 5:05 AM, Vladimir Daskalov <valado.89@gmail.com> wrote:
Hi,
i'm not quite sure that i understand your question?

Thanks for the response. The question doesn't stand alone very well; it's
related to a broader question I had asked earlier. The summary is that I'm
interested in constructing compound datatypes on the fly. I am trying to
figure out if it's feasible to do something along the lines of
1) get next datum
2) Insert its description into the compound datatype
3) write the datum into the compound datatype,

You are trying to:
1. write part of a compound datatype. - "the ability to write part of a
compound datatype, rather than all of it in one go."
2. write part of a dataset. - "but none that obviously
give me a mechanism for selecting what part of the dataset is to be
written."

As to the first one, i'm not familiar with a way to do it. (Maybe there is a
way, i don't know)

The second one:
The signature of the H5Dwrite function is:
herr_t H5Dwrite(hid_t dataset_id, hid_t mem_type_id, hid_t mem_space_id,
hid_t file_space_id, hid_t xfer_plist_id, const void * buf )

In order to specify which part of the dataset is the passed data referenced
by the buf pointer (essentially where in the dataset to store the provided
data), you should call
herr_t H5Sselect_hyperslab(hid_t space_id, H5S_seloper_t op, const hsize_t
*start, const hsize_t *stride, const hsize_t *count, const hsize_t *block )
before H5Dwrite with the same space_id argument as file_space_id
in H5Dwrite.

Eexample:
Assuming you want to store an array([500]) of compound datatype (structures)
from the beginning of the dataset to the 500th element.

// Create file_space, mem_space, mem_type, dataset, etc.

hsize_t start = 0;
hsize_t stride = 0;
hsize_t count = 500;
hsize_t block = 0;

H5Sselect_hyperslab( file_space_id, H5S_SELECT_SET, &start, &stride, &count,
&block );
H5Dwrite( dataset_id, mem_type_id, mem_space_id, file_space_id,
xfer_plist_id, data );

More info about H5Sselect_hyperslab at:
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5S.html#Dataspace-SelectHyperslab

Thanks for this, I'll have a close look at it.

Best Regards,
V. Daskalov

On 18 July 2013 00:13, Josiah Slack <josiahnmi@gmail.com> wrote:

I'd like to pursue a particular detail of the compound datatype thread

from yesterday. Specifically, the ability to write part of a compound
datatype, rather than all of it in one go. I've spent some time today trying
to figure out what's involved, but haven't got a clear idea. I'm assuming
that the writing would be accomplished by calling H5Dwrite and passing it
the id of a dataset created using H5Dcreate2() - (I'm cribbing from the
listing of the sample program h5_compound). H5Dwrite() has a variety of
arguments, but none that obviously give me a mechanism for selecting what
part of the dataset is to be written. I've got a couple of guesses:

1. there's some work to be done on the dataset prior to the write that

selects what is supposed to be written, or

2. the property list argument can somehow convey the information.

Any guidance would be gratefully received.

-Josiah

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

4

Hi,
i'm not quite sure that i understand your question?

Thanks for the response. The question doesn't stand alone very well; it's
related to a broader question I had asked earlier. The summary is that I'm
interested in constructing compound datatypes on the fly. I am trying to
figure out if it's feasible to do something along the lines of
1) get next datum
2) Insert its description into the compound datatype
3) write the datum into the compound datatype,

You are trying to:
1. write part of a compound datatype. - "the ability to write part of a
compound datatype, rather than all of it in one go."
2. write part of a dataset. - "but none that obviously
give me a mechanism for selecting what part of the dataset is to be
written."

As to the first one, i'm not familiar with a way to do it. (Maybe there is
a way, i don't know)

The second one:
The signature of the H5Dwrite function is:
herr_t H5Dwrite(hid_t dataset_id, hid_t mem_type_id, hid_t mem_space_id,
hid_t file_space_id, hid_t xfer_plist_id, const void * buf )

In order to specify which part of the dataset is the passed data
referenced by the buf pointer (essentially where in the dataset to store
the provided data), you should call
herr_t H5Sselect_hyperslab(hid_t space_id, H5S_seloper_t op, const hsize_t
*start, const hsize_t *stride, const hsize_t *count, const hsize_t *block )
before H5Dwrite with the same space_id argument as file_space_id
in H5Dwrite.

Eexample:
Assuming you want to store an array([500]) of compound datatype
(structures) from the beginning of the dataset to the 500th element.

// Create file_space, mem_space, mem_type, dataset, etc.

hsize_t start = 0;
hsize_t stride = 0;
hsize_t count = 500;
hsize_t block = 0;

H5Sselect_hyperslab( file_space_id, H5S_SELECT_SET, &start, &stride,
&count, &block );
H5Dwrite( dataset_id, mem_type_id, mem_space_id, file_space_id,
xfer_plist_id, data );

More info about H5Sselect_hyperslab at:
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5S.html#Dataspace-SelectHyperslab

Thanks for this, I'll have a close look at it.

···

On Thu, Jul 18, 2013 at 5:05 AM, Vladimir Daskalov <valado.89@gmail.com>wrote:

Best Regards,
V. Daskalov

On 18 July 2013 00:13, Josiah Slack <josiahnmi@gmail.com> wrote:
>
> I'd like to pursue a particular detail of the compound datatype thread
from yesterday. Specifically, the ability to write part of a compound
datatype, rather than all of it in one go. I've spent some time today
trying to figure out what's involved, but haven't got a clear idea. I'm
assuming that the writing would be accomplished by calling H5Dwrite and
passing it the id of a dataset created using H5Dcreate2() - (I'm cribbing
from the listing of the sample program h5_compound). H5Dwrite() has a
variety of arguments, but none that obviously give me a mechanism for
selecting what part of the dataset is to be written. I've got a couple of
guesses:
> 1. there's some work to be done on the dataset prior to the write that
selects what is supposed to be written, or
> 2. the property list argument can somehow convey the information.
>
> Any guidance would be gratefully received.
>
> -Josiah
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

5

Thanks - that seems pretty straightforward.

-Josiah

···

On Thu, Jul 18, 2013 at 8:57 AM, Gerd Heber <gheber@hdfgroup.org> wrote:

Reading or writing a compound type partially can be controlled via the
mem_type_id argument. Let's say you have a compound type in the file
with components A, B, and C. If you'd like to read only A or write to A,
just construct an in memory compound type with just an A component.
The library will match components by name between the file type and the
memory type.

G.

6

I'm not having much success. Let me be pretty concrete about the steps I'm
taking, and maybe it will be clear where I'm going wrong.

fileId = H5Fcreate(<filename>, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
compoundType = H5Tcreate(HST_COMPOUND, <overall message size>);
H5Tinsert(compoundType, "dummy", 0, HST_NATIVE_INT); // hack so that
H5Dcreate2 doesn't fail
ctSpace = H5Screate_simple(1, dim, NULL);
ctDataSet = H5Dcreate2(fileId, "messageData", compoundType, ctSpace,
H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
status = H5Dwrite(ctDataSet, compoundType, H5S_ALL, H5S_ALL, H5P_DEFAULT,
&dummyData);
.
.
.
// for each message field
valueDT = H5Tcreate(H5T_COMPOUND, dtSize);
H5Tinsert(valueDT, valueName, 0, dataType);
H5Tinsert(compoundType, valueName, runningOffset, dataType);
runningOffset += dtSize;
status = H5Dwrite(ctDataSet, valueDT, H5S_ALL, ctSpace, H5P_DEFAULT,
&value);
status = H5Tclose(valueDT);
.
.
.
status = H5Fflush(fileId, H5F_SCOPE_LOCAL);
status = H5Tclose(compoundType);
status = H5Sclose(ctSpace);
status = H5Dclose(ctDataSet);
status = H5Fclose(fileId);

I've left out error checking and initializations of a few variables for
clarity.

There are a dozen or so values that I'm attempting to write out, but the
only value that shows up in the output is the dummy that I put in initially.

···

On Thu, Jul 18, 2013 at 9:26 AM, Josiah Slack <josiahnmi@gmail.com> wrote:

Thanks - that seems pretty straightforward.

-Josiah

On Thu, Jul 18, 2013 at 8:57 AM, Gerd Heber <gheber@hdfgroup.org> wrote:

Reading or writing a compound type partially can be controlled via the
mem_type_id argument. Let's say you have a compound type in the file
with components A, B, and C. If you'd like to read only A or write to A,
just construct an in memory compound type with just an A component.
The library will match components by name between the file type and the
memory type.

G.

7

Josiah, maybe I'm misreading your code, but your compound type
in the file has only one member ("dummy").
Later you are trying to write a member whose name is contained
in the valueName variable. Unless that name is "dummy", the
H5Dwrite call is not going to find anything suitable to write to.

G.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Josiah Slack
Sent: Friday, July 19, 2013 12:44 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] writing part of a compound datatype

I'm not having much success. Let me be pretty concrete about the steps I'm taking, and maybe it will be clear where I'm going wrong.
fileId = H5Fcreate(<filename>, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);
compoundType = H5Tcreate(HST_COMPOUND, <overall message size>);
H5Tinsert(compoundType, "dummy", 0, HST_NATIVE_INT); // hack so that H5Dcreate2 doesn't fail
ctSpace = H5Screate_simple(1, dim, NULL);
ctDataSet = H5Dcreate2(fileId, "messageData", compoundType, ctSpace, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
status = H5Dwrite(ctDataSet, compoundType, H5S_ALL, H5S_ALL, H5P_DEFAULT, &dummyData);
.
.
.
// for each message field
valueDT = H5Tcreate(H5T_COMPOUND, dtSize);
H5Tinsert(valueDT, valueName, 0, dataType);
H5Tinsert(compoundType, valueName, runningOffset, dataType);
runningOffset += dtSize;
status = H5Dwrite(ctDataSet, valueDT, H5S_ALL, ctSpace, H5P_DEFAULT, &value);
status = H5Tclose(valueDT);
.
.
.
status = H5Fflush(fileId, H5F_SCOPE_LOCAL);
status = H5Tclose(compoundType);
status = H5Sclose(ctSpace);
status = H5Dclose(ctDataSet);
status = H5Fclose(fileId);
I've left out error checking and initializations of a few variables for clarity.
There are a dozen or so values that I'm attempting to write out, but the only value that shows up in the output is the dummy that I put in initially.

On Thu, Jul 18, 2013 at 9:26 AM, Josiah Slack <josiahnmi@gmail.com<mailto:josiahnmi@gmail.com>> wrote:
Thanks - that seems pretty straightforward.
-Josiah

On Thu, Jul 18, 2013 at 8:57 AM, Gerd Heber <gheber@hdfgroup.org<mailto:gheber@hdfgroup.org>> wrote:
Reading or writing a compound type partially can be controlled via the
mem_type_id argument. Let's say you have a compound type in the file
with components A, B, and C. If you'd like to read only A or write to A,
just construct an in memory compound type with just an A component.
The library will match components by name between the file type and the
memory type.

G.

8

Josiah, maybe I’m misreading your code, but your compound type ****

in the file has only one member (“dummy”).****

Later you are trying to write a member whose name is contained****

in the valueName variable. Unless that name is “dummy”, the****

H5Dwrite call is not going to find anything suitable to write to.****

** **

G.****

** **
*I've got a compound data type that I intended to be associated with the
file (the one I'm calling compoundType), and another compound type in a
loop that is intended to be an in-memory type (what I'm calling valueDT).
In my loop, I do H5Tinsert() operations on both compound datatypes. Do I
need to make a new dataset for compoundType inside the loop?*

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

···

On Fri, Jul 19, 2013 at 2:06 PM, Gerd Heber <gheber@hdfgroup.org> wrote:

9

Ok, let's go back to the beginning. Maybe I didn't understand your problem
to begin with.

You've created a dataset D with compound file type FT that has members A, B, C.
Later you'd like to write component A to D.
You create a memory compound type MT with component A and write
your buffer to D.

Once you've created a dataset with a certain file type, you cannot change that type.
That is, if I'd like to write component E to D, I'd be out of luck.
There is no "ADD COULMN" - like construct with compound datatypes.

G.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Josiah Slack
Sent: Friday, July 19, 2013 1:32 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] writing part of a compound datatype

On Fri, Jul 19, 2013 at 2:06 PM, Gerd Heber <gheber@hdfgroup.org<mailto:gheber@hdfgroup.org>> wrote:
Josiah, maybe I'm misreading your code, but your compound type
in the file has only one member ("dummy").
Later you are trying to write a member whose name is contained
in the valueName variable. Unless that name is "dummy", the
H5Dwrite call is not going to find anything suitable to write to.

G.

I've got a compound data type that I intended to be associated with the file (the one I'm calling compoundType), and another compound type in a loop that is intended to be an in-memory type (what I'm calling valueDT). In my loop, I do H5Tinsert() operations on both compound datatypes. Do I need to make a new dataset for compoundType inside the loop?

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

10

Thanks for walking me through this. I'll need to go back to the drawing
board - there are various ways that I can discover the structure that I'm
trying to capture in the compound datatype. Assuming that I've populated
the datatype with all the necessary fields, do the other arguments to
H5Dwrite()
look appropriate?

···

On Fri, Jul 19, 2013 at 2:40 PM, Gerd Heber <gheber@hdfgroup.org> wrote:

Ok, let’s go back to the beginning. Maybe I didn’t understand your
problem****

to begin with.****

** **

You’ve created a dataset D with compound file type FT that has members A,
B, C.****

Later you’d like to write component A to D.****

You create a memory compound type MT with component A and write****

your buffer to D.****

** **

Once you’ve created a dataset with a certain file type, you cannot change
that type.****

That is, if I’d like to write component E to D, I’d be out of luck.****

There is no “ADD COULMN” – like construct with compound datatypes.****

** **

G.****

** **

** **

** **

** **

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Josiah Slack
*Sent:* Friday, July 19, 2013 1:32 PM

*To:* HDF Users Discussion List
*Subject:* Re: [Hdf-forum] writing part of a compound datatype****

** **

** **

** **

On Fri, Jul 19, 2013 at 2:06 PM, Gerd Heber <gheber@hdfgroup.org> wrote:**
**

Josiah, maybe I’m misreading your code, but your compound type ****

in the file has only one member (“dummy”).****

Later you are trying to write a member whose name is contained****

in the valueName variable. Unless that name is “dummy”, the****

H5Dwrite call is not going to find anything suitable to write to.****

****

G.****

****

*I've got a compound data type that I intended to be associated with the
file (the one I'm calling compoundType), and another compound type in a
loop that is intended to be an in-memory type (what I'm calling valueDT).
In my loop, I do H5Tinsert() operations on both compound datatypes. Do I
need to make a new dataset for compoundType inside the loop?* ****

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
****

** **

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org