Appending to a dataset in Hdf5 .NET

Hdf-forum archives
1

Hi,

Is there any good way to append data to an existing dataset in Hdf5 .NET, preferably without having to read the existing data back first? Something like the H5PT_append functionality would be ideal, but unfortunately it is not implemented with the .NET wrappers. Additionally H5D_extend does not seem to exist either, which was the other option I considered for doing this. My data is quickly becoming much to large to store in an array and dump into the hdf5 file all at once, so if there was some method of adding the data to the dataset as it arrives instead of all at once after all the data has come in, that would be great.

Thanks for the help,
Mike

···

________________________________
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

2

Hi Mike,

···

On Aug 4, 2010, at 7:59 AM, Rosensweig, Mike wrote:

Hi,

Is there any good way to append data to an existing dataset in Hdf5 .NET, preferably without having to read the existing data back first? Something like the H5PT_append functionality would be ideal, but unfortunately it is not implemented with the .NET wrappers. Additionally H5D_extend does not seem to exist either, which was the other option I considered for doing this. My data is quickly becoming much to large to store in an array and dump into the hdf5 file all at once, so if there was some method of adding the data to the dataset as it arrives instead of all at once after all the data has come in, that would be great.

  H5Dextend was deprecated in favor of H5Dset_extent, does that exist in the .NET wrappers?

    Quincey

3

Hi Mike,

···

On Aug 4, 2010, at 7:59 AM, Rosensweig, Mike wrote:

Hi,

Is there any good way to append data to an existing dataset in Hdf5 .NET,
preferably without having to read the existing data back first? Something
like the H5PT_append functionality would be ideal, but unfortunately it is
not implemented with the .NET wrappers. Additionally H5D_extend does not
seem to exist either, which was the other option I considered for doing
this. My data is quickly becoming much to large to store in an array and
dump into the hdf5 file all at once, so if there was some method of adding
the data to the dataset as it arrives instead of all at once after all the
data has come in, that would be great.

            H5Dextend was deprecated in favor of H5Dset_extent, does that
exist in the .NET wrappers?

Quincey, no, it doesn't either.

Mike, you can call H5Dset_extent directly using interop for unmanaged code.

Binh-Minh

                        Quincey

4

Whenever I try to use interop to call the hdf5 stuff directly, executing the program results in the following error:

PInvokeStackImbalance was detected
Message: A call to PInvoke function 'H5Dset_extent' has unbalanced the stack. This is likely because the managed PInvoke signature does not match the unmanaged target signature. Check that the calling convention and parameters of the PInvoke signature match the target unmanaged signature.

Despite this error, everything seems to work totally as intended, and the program executes fine. Has anyone encountered this or a similar error before? And can it just be ignored since the program works?

Thanks,
Mike

···

________________________________
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Binh-Minh Ribler
Sent: Wednesday, August 04, 2010 9:14 AM
To: 'HDF Users Discussion List'
Subject: Re: [Hdf-forum] Appending to a dataset in Hdf5 .NET

Hi Mike,

On Aug 4, 2010, at 7:59 AM, Rosensweig, Mike wrote:

Hi,

Is there any good way to append data to an existing dataset in Hdf5 .NET, preferably without having to read the existing data back first? Something like the H5PT_append functionality would be ideal, but unfortunately it is not implemented with the .NET wrappers. Additionally H5D_extend does not seem to exist either, which was the other option I considered for doing this. My data is quickly becoming much to large to store in an array and dump into the hdf5 file all at once, so if there was some method of adding the data to the dataset as it arrives instead of all at once after all the data has come in, that would be great.

            H5Dextend was deprecated in favor of H5Dset_extent, does that exist in the .NET wrappers?

Quincey, no, it doesn't either.
Mike, you can call H5Dset_extent directly using interop for unmanaged code.

Binh-Minh

                        Quincey

________________________________
IMPORTANT: The information contained in this email and/or its attachments is confidential. If you are not the intended recipient, please notify the sender immediately by reply and immediately delete this message and all its attachments. Any review, use, reproduction, disclosure or dissemination of this message or any attachment by an unintended recipient is strictly prohibited. Neither this message nor any attachment is intended as or should be construed as an offer, solicitation or recommendation to buy or sell any security or other financial instrument. Neither the sender, his or her employer nor any of their respective affiliates makes any warranties as to the completeness or accuracy of any of the information contained herein or that this message or any of its attachments is free of viruses.

5

Hi Mike,

Please double check the parameters of the call to match the parameter list
of 'H5Dset_extent' in the documentation. It is not a good idea to ignore
the error since it might cause you problem in the future.

Binh-Minh

···

_____

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Rosensweig, Mike
Sent: Tuesday, August 10, 2010 11:55 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Appending to a dataset in Hdf5 .NET

Whenever I try to use interop to call the hdf5 stuff directly, executing the
program results in the following error:

PInvokeStackImbalance was detected

Message: A call to PInvoke function 'H5Dset_extent' has unbalanced the
stack. This is likely because the managed PInvoke signature does not match
the unmanaged target signature. Check that the calling convention and
parameters of the PInvoke signature match the target unmanaged signature.

Despite this error, everything seems to work totally as intended, and the
program executes fine. Has anyone encountered this or a similar error
before? And can it just be ignored since the program works?

Thanks,

Mike

  _____

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Binh-Minh Ribler
Sent: Wednesday, August 04, 2010 9:14 AM
To: 'HDF Users Discussion List'
Subject: Re: [Hdf-forum] Appending to a dataset in Hdf5 .NET

Hi Mike,

On Aug 4, 2010, at 7:59 AM, Rosensweig, Mike wrote:

Hi,

Is there any good way to append data to an existing dataset in Hdf5 .NET,
preferably without having to read the existing data back first? Something
like the H5PT_append functionality would be ideal, but unfortunately it is
not implemented with the .NET wrappers. Additionally H5D_extend does not
seem to exist either, which was the other option I considered for doing
this. My data is quickly becoming much to large to store in an array and
dump into the hdf5 file all at once, so if there was some method of adding
the data to the dataset as it arrives instead of all at once after all the
data has come in, that would be great.

            H5Dextend was deprecated in favor of H5Dset_extent, does that
exist in the .NET wrappers?

Quincey, no, it doesn't either.

Mike, you can call H5Dset_extent directly using interop for unmanaged code.

Binh-Minh

                        Quincey

  _____

IMPORTANT: The information contained in this email and/or its attachments is
confidential. If you are not the intended recipient, please notify the
sender immediately by reply and immediately delete this message and all its
attachments. Any review, use, reproduction, disclosure or dissemination of
this message or any attachment by an unintended recipient is strictly
prohibited. Neither this message nor any attachment is intended as or should
be construed as an offer, solicitation or recommendation to buy or sell any
security or other financial instrument. Neither the sender, his or her
employer nor any of their respective affiliates makes any warranties as to
the completeness or accuracy of any of the information contained herein or
that this message or any of its attachments is free of viruses.

6

Mike,

I am replying off list intentionally as I don't know how helpful it is
generally. But, we are using HDF from .Net just using standard
inter-op calls. Basically, our assessment of the "stock" .Net wrappers
was that they were quite limited and did not represent a significant
value added. It is very easy to call the HDF library directly using
Pinvoke calls, it all just works. I would be happy to give an example
or two if you are interested.

Matt

···

On Tue, Aug 10, 2010 at 11:54 AM, Rosensweig, Mike <Mike.Rosensweig@sig.com> wrote:

Whenever I try to use interop to call the hdf5 stuff directly, executing the
program results in the following error:

PInvokeStackImbalance was detected

Message: A call to PInvoke function 'H5Dset_extent' has unbalanced the
stack. This is likely because the managed PInvoke signature does not match
the unmanaged target signature. Check that the calling convention and
parameters of the PInvoke signature match the target unmanaged signature.

Despite this error, everything seems to work totally as intended, and the
program executes fine. Has anyone encountered this or a similar error
before? And can it just be ignored since the program works?

Thanks,

Mike

________________________________

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Binh-Minh Ribler
Sent: Wednesday, August 04, 2010 9:14 AM
To: 'HDF Users Discussion List'
Subject: Re: [Hdf-forum] Appending to a dataset in Hdf5 .NET

Hi Mike,

On Aug 4, 2010, at 7:59 AM, Rosensweig, Mike wrote:

Hi,

Is there any good way to append data to an existing dataset in Hdf5 .NET,
preferably without having to read the existing data back first? Something
like the H5PT_append functionality would be ideal, but unfortunately it is
not implemented with the .NET wrappers. Additionally H5D_extend does not
seem to exist either, which was the other option I considered for doing
this. My data is quickly becoming much to large to store in an array and
dump into the hdf5 file all at once, so if there was some method of adding
the data to the dataset as it arrives instead of all at once after all the
data has come in, that would be great.

        H5Dextend was deprecated in favor of H5Dset\_extent, does that

exist in the .NET wrappers?

Quincey, no, it doesn’t either.

Mike, you can call H5Dset_extent directly using interop for unmanaged code.

Binh-Minh

                    Quincey

________________________________
IMPORTANT: The information contained in this email and/or its attachments is
confidential. If you are not the intended recipient, please notify the
sender immediately by reply and immediately delete this message and all its
attachments. Any review, use, reproduction, disclosure or dissemination of
this message or any attachment by an unintended recipient is strictly
prohibited. Neither this message nor any attachment is intended as or should
be construed as an offer, solicitation or recommendation to buy or sell any
security or other financial instrument. Neither the sender, his or her
employer nor any of their respective affiliates makes any warranties as to
the completeness or accuracy of any of the information contained herein or
that this message or any of its attachments is free of viruses.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org