Compound Dataset from C# Struct to HDF File

Hdf-forum archives
1

Hello folks,

I've been beating my head against this one all day... All I'm trying to do
is write a simple struct to a dataset. Using C# and the .NET wrapper, if I
remove the array from the struct, all is well, but once I introduce the
array, the data in the file is mangled (as shown). Any clues? Also, is
there a way to get h5dump to output the values in hex? I found nothing in
the documentation and it would be super useful for debugging. Thanks in
advance!

        [StructLayoutAttribute(LayoutKind.Sequential, Pack = 1)]
        struct cStyleMessage
        {
            public uint MsgID;
            public ulong MsgHWTimestamp;
            public uint MsgLenthBytes;
            [MarshalAs(UnmanagedType.ByValArray,
ArraySubType=UnmanagedType.U1, SizeConst=8)]
            public byte[] MsgBody;
        }

        private void btn_WriteCustomType_Click(object sender,
RoutedEventArgs e)
        {
            //Create a C friendly struct full of easily walkable data...
            cStyleMessage myCStyleMsg = new cStyleMessage();
            myCStyleMsg.MsgID = 0x04030201; //dec: 67305985
            myCStyleMsg.MsgHWTimestamp = 0x0C0B0A0908070605; //dec:
867798387104613893
            myCStyleMsg.MsgLenthBytes = 0x100F0E0D; //dec: 269422093
            myCStyleMsg.MsgBody = new byte[] { 0x11, 0x12, 0x13, 0x14,
0x15, 0x16, 0x17, 0x18 };

            //Define HDF5 compound type
            H5DataSpaceId myDataSpace = H5S.create(H5S.H5SClass.SCALAR);
            int structSize = Marshal.SizeOf(myCStyleMsg);
            H5DataTypeId myH5MessageType =
H5T.create(H5T.CreateClass.COMPOUND, structSize);
            H5T.insert(myH5MessageType, "MsgID",0, new
H5DataTypeId(H5T.H5Type.STD_U32LE));
            H5T.insert(myH5MessageType, "MsgHWTimestamp",4, new
H5DataTypeId(H5T.H5Type.STD_U64LE));
            H5T.insert(myH5MessageType, "MsgLengthBytes", 12, new
H5DataTypeId(H5T.H5Type.STD_U32LE));
            H5T.insert(myH5MessageType, "MsgBody", 16, H5T.create_array(new
H5DataTypeId(H5T.H5Type.STD_U8LE), new long[1] {8}));
            //Create the compound type
            H5DataSetId myDataSet = H5D.create(h5FileHandle,
"/myCustomData", myH5MessageType, myDataSpace);

            //Write to file
            H5D.writeScalar<cStyleMessage>(myDataSet, myH5MessageType, ref
myCStyleMsg);
            //clean up localy created overhead
            H5D.close(myDataSet);
            H5S.close(myDataSpace);
        }

The data in the file is mangled as shown by H5dump:

D:\Programs\HDF5Utils>h5dump c:\temp\myfirstfile.h5
HDF5 "c:\temp\myfirstfile.h5" {
GROUP "/" {
   DATASET "myCustomData" {
      DATATYPE H5T_COMPOUND {
         H5T_STD_U32LE "MsgID";
         H5T_STD_U64LE "MsgHWTimestamp";
         H5T_STD_U32LE "MsgLengthBytes";
         H5T_ARRAY { [8] H5T_STD_U8LE } "MsgBody";
      }
      DATASPACE SCALAR
      DATA {
      (0): {
            134678021,
            181141207983524361,
            67305985,
            [ 13, 14, 15, 16, 144, 254, 123, 2 ]
         }
      }
   }
}
}

The first four bytes are being lost, the data is being shifted by those
four bytes, and then the last four bytes in the file are garbage.

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

···

--

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *

2

I am not a Windows person, so don't know how the MSC# is dealing with the
alignment of structures, but I would try:

      struct cStyleMessage
        {
            public uint MsgID;
            public uint MsgLenthBytes;
            public ulong MsgHWTimestamp;
            [keep the rest the same]
       }

If that does not help, then my advice is to start with a compound type of a
single field, see if that works, and then proceed with 2 fields. When you
reach this point you will be able to figure out how to generalize more
easily.

···

2015-08-06 22:20 GMT+02:00 Petr KLAPKA <petr.klapka@valeo.com>:

Hello folks,

I've been beating my head against this one all day... All I'm trying to
do is write a simple struct to a dataset. Using C# and the .NET wrapper,
if I remove the array from the struct, all is well, but once I introduce
the array, the data in the file is mangled (as shown). Any clues? Also,
is there a way to get h5dump to output the values in hex? I found nothing
in the documentation and it would be super useful for debugging. Thanks in
advance!

        [StructLayoutAttribute(LayoutKind.Sequential, Pack = 1)]
        struct cStyleMessage
        {
            public uint MsgID;
            public ulong MsgHWTimestamp;
            public uint MsgLenthBytes;
            [MarshalAs(UnmanagedType.ByValArray,
ArraySubType=UnmanagedType.U1, SizeConst=8)]
            public byte[] MsgBody;
        }

--
Francesc Alted

3

I cracked the nut late last night. The problem was indeed on the C# side.
By forcing it to use "unsafe" code, I was able to define the struct in a C
native way and avoid .NET marshaling between it. I'm now getting exactly
what I expect in the file according to H5dump. Next step is an array of
200 of these. Then an ever expanding array of those arrays, and I'll have
my HDF5 data structure figured out. Thanks for your inputs!

Here is the solution:

        [StructLayoutAttribute(LayoutKind.Sequential, Pack = 1)]
        unsafe struct cStyleMessage
        {
            public uint MsgID;
            public ulong MsgHWTimestamp;
            public uint MsgLenthBytes;
            public fixed byte MsgBody[8];
        }

        private void btn_WriteCustomType_Click(object sender,
RoutedEventArgs e)
        {
            //Create a C friendly struct full of easily walkable data...
            cStyleMessage myCStyleMsg = new cStyleMessage();
            myCStyleMsg.MsgID = 0x04030201; //dec: 67305985
            myCStyleMsg.MsgHWTimestamp = 0x0C0B0A0908070605; //dec:
867798387104613893
            myCStyleMsg.MsgLenthBytes = 0x100F0E0D; //dec: 269422093
            unsafe
            {
                for (int i = 0; i < 8; i++)
                {
                    myCStyleMsg.MsgBody[i] = (byte)(0x11 + i);
                }
            }

            //Define HDF5 compound type
            H5DataSpaceId myDataSpace = H5S.create(H5S.H5SClass.SCALAR);
            int structSize = Marshal.SizeOf(myCStyleMsg);
            H5DataTypeId myH5MessageType =
H5T.create(H5T.CreateClass.COMPOUND, structSize);
            H5T.insert(myH5MessageType, "MsgID",0, new
H5DataTypeId(H5T.H5Type.STD_U32LE));
            H5T.insert(myH5MessageType, "MsgHWTimestamp",4, new
H5DataTypeId(H5T.H5Type.STD_U64LE));
            H5T.insert(myH5MessageType, "MsgLengthBytes", 12, new
H5DataTypeId(H5T.H5Type.STD_U32LE));
            H5T.insert(myH5MessageType, "MsgBody", 16, H5T.create_array(new
H5DataTypeId(H5T.H5Type.STD_U8LE), new long[1] {8}));
            //Create the compound type
            H5DataSetId myDataSet = H5D.create(h5FileHandle,
"/myCustomData", myH5MessageType, myDataSpace);

            //Write to file
            H5D.writeScalar<cStyleMessage>(myDataSet, myH5MessageType, ref
myCStyleMsg);
            //clean up localy created overhead
            H5D.close(myDataSet);
            H5S.close(myDataSpace);
        }

Best regards,

Petr Klapka
System Tools Engineer
*Valeo* Radar Systems
46 River Rd
Hudson, NH 03051
Mobile: (603) 921-4440
Office: (603) 578-8045
*"Festina lente."*

···

On Fri, Aug 7, 2015 at 7:29 AM, Francesc Alted <faltet@gmail.com> wrote:

2015-08-06 22:20 GMT+02:00 Petr KLAPKA <petr.klapka@valeo.com>:

Hello folks,

I've been beating my head against this one all day... All I'm trying to
do is write a simple struct to a dataset. Using C# and the .NET wrapper,
if I remove the array from the struct, all is well, but once I introduce
the array, the data in the file is mangled (as shown). Any clues? Also,
is there a way to get h5dump to output the values in hex? I found nothing
in the documentation and it would be super useful for debugging. Thanks in
advance!

        [StructLayoutAttribute(LayoutKind.Sequential, Pack = 1)]
        struct cStyleMessage
        {
            public uint MsgID;
            public ulong MsgHWTimestamp;
            public uint MsgLenthBytes;
            [MarshalAs(UnmanagedType.ByValArray,
ArraySubType=UnmanagedType.U1, SizeConst=8)]
            public byte[] MsgBody;
        }

I am not a Windows person, so don't know how the MSC# is dealing with the
alignment of structures, but I would try:

      struct cStyleMessage
        {
            public uint MsgID;
            public uint MsgLenthBytes;
            public ulong MsgHWTimestamp;
            [keep the rest the same]
       }

If that does not help, then my advice is to start with a compound type of
a single field, see if that works, and then proceed with 2 fields. When
you reach this point you will be able to figure out how to generalize more
easily.

--
Francesc Alted

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--

*This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender
at the above address and destroy it. *