How does one append to Datasets?

Petr_KLAPKA · August 10, 2015, 7:09pm

Hello all,

Now that I'm able to write types of pretty much unlimited complexity into
the HDF5 files, the next step is writing additional elements of data to
existing datasets.

I'll go diving through the manuals again, but if anyone has a "quick start
guide" link for how to create a dataset and append to (stream to) it
throughout the life of the application, I'd be most grateful.

My next goal is to add elements to the "myMajorCycles" dataset that is
pictured below with just a single element. I need to accomplish this in a
way where the dataset is kept open and can accept new data as it comes in,
but it must not remain completely in memory because the final dataset on
disk will be ~2GB.

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myMajorCycles" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "UniqueID";
         H5T_STD_U32LE "MsgCount";
         H5T_ARRAY { [5] H5T_COMPOUND {
            H5T_STD_U32LE "MsgID";
            H5T_STD_U64LE "MsgHWTimestamp";
            H5T_STD_U32LE "MsgLengthBytes";
            H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
         } } "Messages";
      }
      DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): {
            23,
            5,
            [ {
                  10,
                  18446744073709551615,
                  8,
                  [ 0, 1, 2, 3, 4, 5, 6, 7 ]
               },
                {
                  11,
                  18446744073709551615,
                  8,
                  [ 10, 11, 12, 13, 14, 15, 16, 17 ]
               },
                {
                  12,
                  18446744073709551615,
                  8,
                  [ 20, 21, 22, 23, 24, 25, 26, 27 ]
               },
                {
                  13,
                  18446744073709551615,
                  8,
                  [ 30, 31, 32, 33, 34, 35, 36, 37 ]
               },
                {
                  14,
                  18446744073709551615,
                  8,
                  [ 40, 41, 42, 43, 44, 45, 46, 47 ]
               } ]
         }
      }
   }
}
}

Many thanks!

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

···

--

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

nevion · August 10, 2015, 8:46pm

Hi Petr,

I know I've said this before and I'm not sure how first-class this is in C#
(you probably will have to drop to the c-api) but packet tables do this.
They keep the table api backend open so you can keep appending but
otherwise are just another table (existing tools will treat it as a normal
HDF5 table dataset).

-Jason

···

On Mon, Aug 10, 2015 at 12:09 PM, Petr KLAPKA <petr.klapka@valeo.com> wrote:

Hello all,

Now that I'm able to write types of pretty much unlimited complexity into
the HDF5 files, the next step is writing additional elements of data to
existing datasets.

I'll go diving through the manuals again, but if anyone has a "quick start
guide" link for how to create a dataset and append to (stream to) it
throughout the life of the application, I'd be most grateful.

My next goal is to add elements to the "myMajorCycles" dataset that is
pictured below with just a single element. I need to accomplish this in a
way where the dataset is kept open and can accept new data as it comes in,
but it must not remain completely in memory because the final dataset on
disk will be ~2GB.

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myMajorCycles" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "UniqueID";
         H5T_STD_U32LE "MsgCount";
         H5T_ARRAY { [5] H5T_COMPOUND {
            H5T_STD_U32LE "MsgID";
            H5T_STD_U64LE "MsgHWTimestamp";
            H5T_STD_U32LE "MsgLengthBytes";
            H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
         } } "Messages";
      }
      DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): {
            23,
            5,
            [ {
                  10,
                  18446744073709551615,
                  8,
                  [ 0, 1, 2, 3, 4, 5, 6, 7 ]
               },
                {
                  11,
                  18446744073709551615,
                  8,
                  [ 10, 11, 12, 13, 14, 15, 16, 17 ]
               },
                {
                  12,
                  18446744073709551615,
                  8,
                  [ 20, 21, 22, 23, 24, 25, 26, 27 ]
               },
                {
                  13,
                  18446744073709551615,
                  8,
                  [ 30, 31, 32, 33, 34, 35, 36, 37 ]
               },
                {
                  14,
                  18446744073709551615,
                  8,
                  [ 40, 41, 42, 43, 44, 45, 46, 47 ]
               } ]
         }
      }
   }
}
}

Many thanks!

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Petr_KLAPKA · August 10, 2015, 8:53pm

Hi Jason,

Unfortunately, Packet Tables aren't even mentioned in the HDF5DotNet
documentation.

I've received advice from a few other folks to do this with expanding
datasets. I've been reading through the C API documentation on this, and
it looks like it will work as long as I can find the analogs to the
documented function in the C# wrapper... Getting burnt out on this today,
so I will pick it up again in the AM.

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

···

On Mon, Aug 10, 2015 at 4:46 PM, Jason Newton <nevion@gmail.com> wrote:

Hi Petr,

I know I've said this before and I'm not sure how first-class this is in
C# (you probably will have to drop to the c-api) but packet tables do
this. They keep the table api backend open so you can keep appending but
otherwise are just another table (existing tools will treat it as a normal
HDF5 table dataset).

-Jason

On Mon, Aug 10, 2015 at 12:09 PM, Petr KLAPKA <petr.klapka@valeo.com> > wrote:

Hello all,

Now that I'm able to write types of pretty much unlimited complexity into
the HDF5 files, the next step is writing additional elements of data to
existing datasets.

I'll go diving through the manuals again, but if anyone has a "quick
start guide" link for how to create a dataset and append to (stream to) it
throughout the life of the application, I'd be most grateful.

My next goal is to add elements to the "myMajorCycles" dataset that is
pictured below with just a single element. I need to accomplish this in a
way where the dataset is kept open and can accept new data as it comes in,
but it must not remain completely in memory because the final dataset on
disk will be ~2GB.

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myMajorCycles" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "UniqueID";
         H5T_STD_U32LE "MsgCount";
         H5T_ARRAY { [5] H5T_COMPOUND {
            H5T_STD_U32LE "MsgID";
            H5T_STD_U64LE "MsgHWTimestamp";
            H5T_STD_U32LE "MsgLengthBytes";
            H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
         } } "Messages";
      }
      DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): {
            23,
            5,
            [ {
                  10,
                  18446744073709551615,
                  8,
                  [ 0, 1, 2, 3, 4, 5, 6, 7 ]
               },
                {
                  11,
                  18446744073709551615,
                  8,
                  [ 10, 11, 12, 13, 14, 15, 16, 17 ]
               },
                {
                  12,
                  18446744073709551615,
                  8,
                  [ 20, 21, 22, 23, 24, 25, 26, 27 ]
               },
                {
                  13,
                  18446744073709551615,
                  8,
                  [ 30, 31, 32, 33, 34, 35, 36, 37 ]
               },
                {
                  14,
                  18446744073709551615,
                  8,
                  [ 40, 41, 42, 43, 44, 45, 46, 47 ]
               } ]
         }
      }
   }
}
}

Many thanks!

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

faltet · August 11, 2015, 6:25pm

Hi Petr,

For appending data to an existing dataset in HDF5 you need a chunked (or
extensible) dataset instead of a simple (or contiguous) one. There is
plenty of info about HDF5 chunked datasets out there, but the basics are
here:

https://www.hdfgroup.org/HDF5/doc/H5.user/Chunking.html

Now, you simply need to create a chunked dataset and append data to it, so
the next tutorial:

https://www.hdfgroup.org/HDF5/Tutor/extend.html

should basically be all you need.

Also, and to make things a bit simpler, the H5TB is a 'high-level' API that
uses the regular HDF5 API to create primitives so as to make the calls
easier (although somewhat less flexible). Even if there is not a .NET API
for H5TB, what you can do is to have a look at the sources and port what
you need and include your own 'simple enough' H5TB version to your
project. For example, PyTables is using this route, and you might just
grab the H5TBOmake_table, H5TBOread_records and H5TBOappend_records from:

github.com

PyTables/PyTables/blob/develop/src/H5TB-opt.c

/****************************************************************************
 * NCSA HDF                                                                 *
 * Scientific Data Technologies                                             *
 * National Center for Supercomputing Applications                          *
 * University of Illinois at Urbana-Champaign                               *
 * 605 E. Springfield, Champaign IL 61820                                   *
 *                                                                          *
 * For conditions of distribution and use, see the accompanying             *
 * hdf/COPYING file.                                                        *
 *                                                                          *
 ****************************************************************************/

/* WARNING: This is a highly stripped down and modified version of the
   original H5TB.c that comes with the HDF5 library. These
   modifications has been done in order to serve the needs of
   PyTables, and specially for supporting nested datatypes. In
   particular, the VERSION attribute is out of sync so it is not
   guaranteed that the resulting PyTables objects will be identical
   with those generated with HDF5_HL, although they should remain
   largely compatibles.

This file has been truncated. show original

convert them to .NET calls, and you are ready to go and use them in your
project.

Francesc

···

2015-08-10 21:09 GMT+02:00 Petr KLAPKA <petr.klapka@valeo.com>:

Hello all,

Now that I'm able to write types of pretty much unlimited complexity into
the HDF5 files, the next step is writing additional elements of data to
existing datasets.

I'll go diving through the manuals again, but if anyone has a "quick start
guide" link for how to create a dataset and append to (stream to) it
throughout the life of the application, I'd be most grateful.

My next goal is to add elements to the "myMajorCycles" dataset that is
pictured below with just a single element. I need to accomplish this in a
way where the dataset is kept open and can accept new data as it comes in,
but it must not remain completely in memory because the final dataset on
disk will be ~2GB.

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myMajorCycles" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "UniqueID";
         H5T_STD_U32LE "MsgCount";
         H5T_ARRAY { [5] H5T_COMPOUND {
            H5T_STD_U32LE "MsgID";
            H5T_STD_U64LE "MsgHWTimestamp";
            H5T_STD_U32LE "MsgLengthBytes";
            H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
         } } "Messages";
      }
      DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): {
            23,
            5,
            [ {
                  10,
                  18446744073709551615,
                  8,
                  [ 0, 1, 2, 3, 4, 5, 6, 7 ]
               },
                {
                  11,
                  18446744073709551615,
                  8,
                  [ 10, 11, 12, 13, 14, 15, 16, 17 ]
               },
                {
                  12,
                  18446744073709551615,
                  8,
                  [ 20, 21, 22, 23, 24, 25, 26, 27 ]
               },
                {
                  13,
                  18446744073709551615,
                  8,
                  [ 30, 31, 32, 33, 34, 35, 36, 37 ]
               },
                {
                  14,
                  18446744073709551615,
                  8,
                  [ 40, 41, 42, 43, 44, 45, 46, 47 ]
               } ]
         }
      }
   }
}
}

Many thanks!

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Francesc Alted

nevion · August 10, 2015, 9:15pm

OK but make sure you take a look at the documentation and contrast it with
your use cases/requirements:
https://www.hdfgroup.org/HDF5/doc/HL/H5PT_Intro.html . I know the learning
curve is steep for doing what you want to do, having looked at it myself
when I started using HDF5 for these same purposes a few years ago too (in
C++/C). Let me assure you it is the right tool for the job though - both
HDF5 and packet tables.

Another word of caution: C is the only first class citizen of HDF5 - C++,
Java, and C# all have just subsets of it exposed for what they deemed was
common to wrap and can have some some conveniences and issues the C-api
doesn't. This is why I'd suggest just using the C-api for packet table as
it itself is not a first class HDF citizen, nor are language wrappers.
Documentation and knowledge of things like this suffers a bit also again
because the C part is the only first class citizen.. and even then when
there's no example or clarity on an issue, I just jump down into the
library sourcecode to figure out what it's doing... not that I expect or
hope you do But the benefits are worth the pain: long term achievable
and completely portable and good methods for compression, robust to changes
of datatypes, FAST - and tools like hdfview and hdfdump. It solves the
problem so well it is used by many other different api's these days
including Matlab's version 7+ .mat format and NetCDF4. So please be
encouraged to keep with it - I'm sure you'll eventually crack it.

-Jason

···

On Mon, Aug 10, 2015 at 1:53 PM, Petr KLAPKA <petr.klapka@valeo.com> wrote:

Hi Jason,

Unfortunately, Packet Tables aren't even mentioned in the HDF5DotNet
documentation.

I've received advice from a few other folks to do this with expanding
datasets. I've been reading through the C API documentation on this, and
it looks like it will work as long as I can find the analogs to the
documented function in the C# wrapper... Getting burnt out on this today,
so I will pick it up again in the AM.

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

On Mon, Aug 10, 2015 at 4:46 PM, Jason Newton <nevion@gmail.com> wrote:

Hi Petr,

I know I've said this before and I'm not sure how first-class this is in
C# (you probably will have to drop to the c-api) but packet tables do
this. They keep the table api backend open so you can keep appending but
otherwise are just another table (existing tools will treat it as a normal
HDF5 table dataset).

-Jason

On Mon, Aug 10, 2015 at 12:09 PM, Petr KLAPKA <petr.klapka@valeo.com> >> wrote:

Hello all,

Now that I'm able to write types of pretty much unlimited complexity
into the HDF5 files, the next step is writing additional elements of data
to existing datasets.

I'll go diving through the manuals again, but if anyone has a "quick
start guide" link for how to create a dataset and append to (stream to) it
throughout the life of the application, I'd be most grateful.

My next goal is to add elements to the "myMajorCycles" dataset that is
pictured below with just a single element. I need to accomplish this in a
way where the dataset is kept open and can accept new data as it comes in,
but it must not remain completely in memory because the final dataset on
disk will be ~2GB.

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myMajorCycles" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "UniqueID";
         H5T_STD_U32LE "MsgCount";
         H5T_ARRAY { [5] H5T_COMPOUND {
            H5T_STD_U32LE "MsgID";
            H5T_STD_U64LE "MsgHWTimestamp";
            H5T_STD_U32LE "MsgLengthBytes";
            H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
         } } "Messages";
      }
      DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): {
            23,
            5,
            [ {
                  10,
                  18446744073709551615,
                  8,
                  [ 0, 1, 2, 3, 4, 5, 6, 7 ]
               },
                {
                  11,
                  18446744073709551615,
                  8,
                  [ 10, 11, 12, 13, 14, 15, 16, 17 ]
               },
                {
                  12,
                  18446744073709551615,
                  8,
                  [ 20, 21, 22, 23, 24, 25, 26, 27 ]
               },
                {
                  13,
                  18446744073709551615,
                  8,
                  [ 30, 31, 32, 33, 34, 35, 36, 37 ]
               },
                {
                  14,
                  18446744073709551615,
                  8,
                  [ 40, 41, 42, 43, 44, 45, 46, 47 ]
               } ]
         }
      }
   }
}
}

Many thanks!

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Petr_KLAPKA · August 11, 2015, 2:20pm

Thanks Jason,

After I check out the Packet Table C documentation, I'll have a look at
what it will take to expand the .net wrapper to include the PT "class."

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

···

On Mon, Aug 10, 2015 at 5:15 PM, Jason Newton <nevion@gmail.com> wrote:

OK but make sure you take a look at the documentation and contrast it with
your use cases/requirements:
https://www.hdfgroup.org/HDF5/doc/HL/H5PT_Intro.html . I know the
learning curve is steep for doing what you want to do, having looked at it
myself when I started using HDF5 for these same purposes a few years ago
too (in C++/C). Let me assure you it is the right tool for the job though
- both HDF5 and packet tables.

Another word of caution: C is the only first class citizen of HDF5 - C++,
Java, and C# all have just subsets of it exposed for what they deemed was
common to wrap and can have some some conveniences and issues the C-api
doesn't. This is why I'd suggest just using the C-api for packet table as
it itself is not a first class HDF citizen, nor are language wrappers.
Documentation and knowledge of things like this suffers a bit also again
because the C part is the only first class citizen.. and even then when
there's no example or clarity on an issue, I just jump down into the
library sourcecode to figure out what it's doing... not that I expect or
hope you do But the benefits are worth the pain: long term achievable
and completely portable and good methods for compression, robust to changes
of datatypes, FAST - and tools like hdfview and hdfdump. It solves the
problem so well it is used by many other different api's these days
including Matlab's version 7+ .mat format and NetCDF4. So please be
encouraged to keep with it - I'm sure you'll eventually crack it.

-Jason

On Mon, Aug 10, 2015 at 1:53 PM, Petr KLAPKA <petr.klapka@valeo.com> > wrote:

Hi Jason,

Unfortunately, Packet Tables aren't even mentioned in the HDF5DotNet
documentation.

I've received advice from a few other folks to do this with expanding
datasets. I've been reading through the C API documentation on this, and
it looks like it will work as long as I can find the analogs to the
documented function in the C# wrapper... Getting burnt out on this today,
so I will pick it up again in the AM.

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

On Mon, Aug 10, 2015 at 4:46 PM, Jason Newton <nevion@gmail.com> wrote:

Hi Petr,

I know I've said this before and I'm not sure how first-class this is in
C# (you probably will have to drop to the c-api) but packet tables do
this. They keep the table api backend open so you can keep appending but
otherwise are just another table (existing tools will treat it as a normal
HDF5 table dataset).

-Jason

On Mon, Aug 10, 2015 at 12:09 PM, Petr KLAPKA <petr.klapka@valeo.com> >>> wrote:

Hello all,

Now that I'm able to write types of pretty much unlimited complexity
into the HDF5 files, the next step is writing additional elements of data
to existing datasets.

I'll go diving through the manuals again, but if anyone has a "quick
start guide" link for how to create a dataset and append to (stream to) it
throughout the life of the application, I'd be most grateful.

My next goal is to add elements to the "myMajorCycles" dataset that is
pictured below with just a single element. I need to accomplish this in a
way where the dataset is kept open and can accept new data as it comes in,
but it must not remain completely in memory because the final dataset on
disk will be ~2GB.

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myMajorCycles" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "UniqueID";
         H5T_STD_U32LE "MsgCount";
         H5T_ARRAY { [5] H5T_COMPOUND {
            H5T_STD_U32LE "MsgID";
            H5T_STD_U64LE "MsgHWTimestamp";
            H5T_STD_U32LE "MsgLengthBytes";
            H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
         } } "Messages";
      }
      DATASPACE SIMPLE { ( 1 ) / ( 1 ) }
      DATA {
      (0): {
            23,
            5,
            [ {
                  10,
                  18446744073709551615,
                  8,
                  [ 0, 1, 2, 3, 4, 5, 6, 7 ]
               },
                {
                  11,
                  18446744073709551615,
                  8,
                  [ 10, 11, 12, 13, 14, 15, 16, 17 ]
               },
                {
                  12,
                  18446744073709551615,
                  8,
                  [ 20, 21, 22, 23, 24, 25, 26, 27 ]
               },
                {
                  13,
                  18446744073709551615,
                  8,
                  [ 30, 31, 32, 33, 34, 35, 36, 37 ]
               },
                {
                  14,
                  18446744073709551615,
                  8,
                  [ 40, 41, 42, 43, 44, 45, 46, 47 ]
               } ]
         }
      }
   }
}
}

Many thanks!

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *