Is H5LT cross-platform compatible (due to endianness)

Hdf-forum archives
1

Hello,

I noticed that you cannot set the endianness of data sets or attributes when you are using the H5LT interface. The tutorials suggest using a NATIVE datatype for this. But it is not clear which endianness the files will be written in.

So I am concerned that if my hdf5 files are made of an LE machine, that they may not be read correctly by a BE machine. I do not currently have a BE machine to test this on, but as my project is meant to run on large clusters it is feasible that this could be the case in the future. So currently I am implementing using the Low-Level c routines so that I can guarantee this.

Thank you,
Justin

2

Justin,

NATIVE will be whatever the endian-ness of the platform the code is running on. You do not need to worry about the ability of other machines to read the data correctly--the HDF5 library takes care of that. In fact, that is one of the most basic benefits of using HDF5.

From https://www.hdfgroup.org/HDF5/doc/UG/UG_frame11Datatypes.html

    For example, when reading from data stored as 32-bit signed
    integers, big-endian into 32-bit signed integers, little-endian, the
    HDF5 Library will automatically swap the bytes.

This conversion obviously uses CPU cycles and where you could get into trouble in terms of performance (but not, importantly, correctness) is if you had a large amount of data written in in an endian-ness didn't match the target platform *and* which was being read on the target platform a significant number of times or on target with limited resources like a smart phone or tablet. In that case you might want to force the data to match the endian-ness of the target platform rather than that of that NATIVE endianness of the source platform.

My sense from you email is that you don't (and possibly can't) know enough about the source and target platforms to make a useful choice so I'd say you are fine sticking with the H5LT library. You can deal with performance issues if they come up later.

--dan

···

On 07/07/15 06:03, Justin Erwin wrote:

Hello,

I noticed that you cannot set the endianness of data sets or attributes when you are using the H5LT interface. The tutorials suggest using a NATIVE datatype for this. But it is not clear which endianness the files will be written in.

So I am concerned that if my hdf5 files are made of an LE machine, that they may not be read correctly by a BE machine. I do not currently have a BE machine to test this on, but as my project is meant to run on large clusters it is feasible that this could be the case in the future. So currently I am implementing using the Low-Level c routines so that I can guarantee this.

Thank you,
Justin
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Daniel Kahn
Science Systems and Applications Inc.
301-867-2162