xyz=C++, zyx=Fortran

Hi everyone,

if you create a 3d-array(i,j,k) and use HDF5-library to dump it to disk,
it will look different depending on the application-language:

Fortran-App (=ZYX-style)
  [iii][iii][iii] [iii][iii][iii] [iii][iii][iii]
[ j ][ j ][ j ]
[ k ]

C/C++-App (=XYZ-style)
  [kkk][kkk][kkk] [kkk][kkk][kkk] [kkk][kkk][kkk]
[ j ][ j ][ j ]
[ i ]

Is there any possibility to tell hdf5 what array-style the application
is using?
The same way I can jump from little-endian- to big-endian-machines, I
would like to jump from Fortran-Apps to C++Apps.

Greetings
Jens

···

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

I dont think it is possible. I help myself out with attributes to know the ordering.

Together with the missing compression support in the high level API, those two are the biggest shortcomings of the otherwise great hdf5 library.

-- DOminik

Jens wrote:

···

Hi everyone,

if you create a 3d-array(i,j,k) and use HDF5-library to dump it to disk,
it will look different depending on the application-language:

Fortran-App (=ZYX-style)
  [iii][iii][iii] [iii][iii][iii] [iii][iii][iii]
[ j ][ j ][ j ]
[ k ]

C/C++-App (=XYZ-style)
  [kkk][kkk][kkk] [kkk][kkk][kkk] [kkk][kkk][kkk]
[ j ][ j ][ j ]
[ i ]

Is there any possibility to tell hdf5 what array-style the application
is using?
The same way I can jump from little-endian- to big-endian-machines, I
would like to jump from Fortran-Apps to C++Apps.

Greetings
Jens

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

We also have this problem.
Data are always stored in C order in HDF5 files.
In code where Fortran order is needed, the axes are reversed (thus data are not transposed).
So Fortran code accesses data arrays as [x,y,z], and C++ (or numpy) code as [z,y,x]

Cheers,
Ger

Dominik Szczerba <domi@vision.ee.ethz.ch> 06/05/09 2:37 PM >>>

I dont think it is possible. I help myself out with attributes to know
the ordering.

Together with the missing compression support in the high level API,
those two are the biggest shortcomings of the otherwise great hdf5 library.

-- DOminik

Jens wrote:

···

Hi everyone,

if you create a 3d-array(i,j,k) and use HDF5-library to dump it to disk,
it will look different depending on the application-language:

Fortran-App (=ZYX-style)
  [iii][iii][iii] [iii][iii][iii] [iii][iii][iii]
[ j ][ j ][ j ]
[ k ]

C/C++-App (=XYZ-style)
  [kkk][kkk][kkk] [kkk][kkk][kkk] [kkk][kkk][kkk]
[ j ][ j ][ j ]
[ i ]

Is there any possibility to tell hdf5 what array-style the application
is using?
The same way I can jump from little-endian- to big-endian-machines, I
would like to jump from Fortran-Apps to C++Apps.

Greetings
Jens

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

I remember that there had been plans to introduce this index ordering as
a dataspace property in HDF5, such that for each dataspace one could tell
whether it is Fortran or C ordering.

It probably didn't make it into an implementation due to lack of funding;
it sounds quite complex to mix hyperslab operations between Fortran-ordering
and C-ordering arrays.

While I originally stumbled across this problem myself as well some time
ago, using Fortran convention arrays in my C/C++ code, it didn't turn out
to be much of an issue. Only requirement is to carefully swap the number
of axis when dealing with HDF5 dimensions, and do that consistently on
all operations. The data itself don't need to be touched in any way, and
the only drawback is that the dimensions of a dataset look reversed when
using "h5ls". Once one got used to that, it's not a big deal. I have my
own API on top of HDF5 now which deals with Fortran-convention arrays
consistently (and also supports C-convention arrays via some attribute
as well).

There was once also the idea of a "shared dataspace" that could be
stored to a file like a "named datatype". That one could then carry
attributes itself. Not sure if this shared dataspace ever was implemented,
but if so, that would be a natural place to store such an ordering
attribute. Or more general, would be a nice place to store the names
of axis, like "X","Y","Z" versus "sagittal", "coronal", ... and
eventually dimensions with them "T", "X" -> "seconds", "metres".

  Werner

···

On Fri, 05 Jun 2009 08:08:00 -0500, Ger van Diepen <diepen@astron.nl> wrote:

We also have this problem.
Data are always stored in C order in HDF5 files.
In code where Fortran order is needed, the axes are reversed (thus data are not transposed).
So Fortran code accesses data arrays as [x,y,z], and C++ (or numpy) code as [z,y,x]

Cheers,
Ger

Dominik Szczerba <domi@vision.ee.ethz.ch> 06/05/09 2:37 PM >>>

I dont think it is possible. I help myself out with attributes to know
the ordering.

Together with the missing compression support in the high level API,
those two are the biggest shortcomings of the otherwise great hdf5 library.

-- DOminik

Jens wrote:

Hi everyone,

if you create a 3d-array(i,j,k) and use HDF5-library to dump it to disk,
it will look different depending on the application-language:

Fortran-App (=ZYX-style)
  [iii][iii][iii] [iii][iii][iii] [iii][iii][iii]
[ j ][ j ][ j ]
[ k ]

C/C++-App (=XYZ-style)
  [kkk][kkk][kkk] [kkk][kkk][kkk] [kkk][kkk][kkk]
[ j ][ j ][ j ]
[ i ]

Is there any possibility to tell hdf5 what array-style the application
is using?
The same way I can jump from little-endian- to big-endian-machines, I
would like to jump from Fortran-Apps to C++Apps.

Greetings
Jens

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

I remember that there had been plans to introduce this index ordering as
a dataspace property in HDF5, such that for each dataspace one could tell
whether it is Fortran or C ordering.

It probably didn't make it into an implementation due to lack of funding;
it sounds quite complex to mix hyperslab operations between Fortran-ordering
and C-ordering arrays.

  Yes, you are correct on both accounts (we planned to do it and it was shelved due to lack of demand at the time and lack of funding).

While I originally stumbled across this problem myself as well some time
ago, using Fortran convention arrays in my C/C++ code, it didn't turn out
to be much of an issue. Only requirement is to carefully swap the number
of axis when dealing with HDF5 dimensions, and do that consistently on
all operations. The data itself don't need to be touched in any way, and
the only drawback is that the dimensions of a dataset look reversed when
using "h5ls". Once one got used to that, it's not a big deal. I have my
own API on top of HDF5 now which deals with Fortran-convention arrays
consistently (and also supports C-convention arrays via some attribute
as well).

  Yes, this seems to be how most people do it.

There was once also the idea of a "shared dataspace" that could be
stored to a file like a "named datatype". That one could then carry
attributes itself. Not sure if this shared dataspace ever was implemented,
but if so, that would be a natural place to store such an ordering
attribute. Or more general, would be a nice place to store the names
of axis, like "X","Y","Z" versus "sagittal", "coronal", ... and
eventually dimensions with them "T", "X" -> "seconds", "metres".

  We have not moved forward on the shared dataspace idea, although it was tempting to implement when we were working on features to support the netCDF data model. I think it's still a good concept to put into code.

  The names of axis, etc. can be set with the "dimension scales" high-level API though, which is available in the 1.8.x release series.

  Quincey

···

On Jun 5, 2009, at 9:20 AM, Werner Benger wrote:

  Werner

On Fri, 05 Jun 2009 08:08:00 -0500, Ger van Diepen > <diepen@astron.nl> wrote:

We also have this problem.
Data are always stored in C order in HDF5 files.
In code where Fortran order is needed, the axes are reversed (thus data are not transposed).
So Fortran code accesses data arrays as [x,y,z], and C++ (or numpy) code as [z,y,x]

Cheers,
Ger

Dominik Szczerba <domi@vision.ee.ethz.ch> 06/05/09 2:37 PM >>>

I dont think it is possible. I help myself out with attributes to know
the ordering.

Together with the missing compression support in the high level API,
those two are the biggest shortcomings of the otherwise great hdf5 library.

-- DOminik

Jens wrote:

Hi everyone,

if you create a 3d-array(i,j,k) and use HDF5-library to dump it to disk,
it will look different depending on the application-language:

Fortran-App (=ZYX-style)
[iii][iii][iii] [iii][iii][iii] [iii][iii][iii]
[ j ][ j ][ j ]
[ k ]

C/C++-App (=XYZ-style)
[kkk][kkk][kkk] [kkk][kkk][kkk] [kkk][kkk][kkk]
[ j ][ j ][ j ]
[ i ]

Is there any possibility to tell hdf5 what array-style the application
is using?
The same way I can jump from little-endian- to big-endian-machines, I
would like to jump from Fortran-Apps to C++Apps.

Greetings
Jens

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--
___________________________________________________________________________
Dr. Werner Benger <werner@cct.lsu.edu> Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
239 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.