The reason that I asked about the complex datatype specifically is that complex is a built-in type in Fortran, Python, IDL, and others, and in many other languages is part of the core libraries (C++ std::complex, Perl Math::Complex, etc.). I think in all these cases (correct me if I'm wrong), the type is typically implemented as a Cartesian representation of complex. (though C++ <complex> also gives std::polar).
Expanding the scope to include more general complex numbers, polar representation, etc. certainly does make it much more complex, and I can understand if the scope of that problem would be difficult to grapple with and come up with a standard that works.
···
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Werner Benger
Sent: Wednesday, February 22, 2017 11:53 AM
To: hdf-forum@lists.hdfgroup.org
Cc: Dietmar Hildenbrand <dietmar.hildenbrand@gmail.com>
Subject: Re: [Hdf-forum] Complex numbers (1.10.x update?)
The topic becomes indeed more complex once you think about complex numbers as just special cases of Geometric Algebra (also known as Clifford Algebra), where complex number are the special case in 2D and quaternions are the special case in 3D. But if that scheme is implemented and provided systematically, then rather do it n-dimensional, which automatically leads to Geometric Algebra. And furthermore, doing it systematically would also imply supporting different coordinate representations of the same number (like cartesian vs. polar coordinates, in higher dimensions even more variations). Also we may want to distinguish between different kinds of inner products, which lead to different algebraic properties. For instance a complex quaternion is used in spacetime algebra for Lorentz transformations, useful both in relativistic astrophysics and in quantum mechanics for the Dirac equations, a Dirac spinor is a rotor (a multivector with a scalar and bi-vector component) in 4D Geometric Algebra.
So, it's yet an open end there. A couple of years ago I had a paper on just that topic, including about storing multivector types (the elementary objects of Geometric Algebra, quaternions are a sub-algebra of 3D GA) in HDF5, interested party may find it here:
http://sciviz.cct.lsu.edu/papers/2009/GraVisMa09.pdf
That article is far from a complete scheme or worked out in detail, but it raises some issues on just this topic, food for thought. And it would be good if complex numbers would fit into a bigger scheme yet to come when they are implemented once.
Werner
On 22.02.2017 18:25, Miller, Mark C. wrote:
This has been an interesting thread to follow. Thanks, Jarom, for starting it 
Having said that, in reading here and in the various links contributors have referenced, I didn't necessarily see anything that described the 'issues' with supporting complex types in HDF5 in a first-class (e.g. built in to the library) sort of way. And, now, I am just really curious what those issues are and if/how they are different from any other primitive type HDF5 supports?
I mean, I get that complex numbers represent a 'pair' of scalar values (which in cartesian coords are typically represented as real/imaginary and in polor coords, magnitude/phase) and I get that HDF5's underlying primitive type model is for the most part like standard floating point and integer computer representations including such things as offset, base, bias, mantissa, exponent, etc., but is there anything substantially more comlicated about handling them as first class type in the library than, as Francesc described below? Are there Fortran variants or language extensions that have something other than that basic modality of a pair of either 32 bit or 64 bit floating point numbers? Are there any Fortrans that use magnitude/phase (e.g. polar coords) instead of real/imaginary (cartesian coords).
Question for Francesc...do you define *both* polar and cartesian forms for complex numbers and if so, do you define a conversion method to go between them? If not, would anything like that ever be useful if it existed?
Seems like someone could define a 'complex types' support library extension on github, perhaps refactored from Francesc' code that might have the ability to become the defacto HDF5 standard for complex data in HDF5 files.
Finally, what about quarternions ;}? I am not kidding, just curious. I think those might be relevant in astro-physical contexts wouldn't they?
Mark
"Hdf-forum on behalf of Nelson, Jarom" wrote:
Thanks for all the responses. Very helpful.
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Filipe Maia
Sent: Wednesday, February 22, 2017 6:49 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org><mailto:hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] Complex numbers (1.10.x update?)
I think the main issue is the "almost exactly".
I'm also using h5py's convention, but having an HDF5 "officially endorsed" way of representing complex numbers would greatly help with compatibility across different user groups, instead of having *almost* compatible representations.
And I hope the endorsed way matches the h5py convention, but any convention is better than none.
Cheers,
Filipe
On 22 February 2017 at 15:34, Barbara Jones <bljones@hdfgroup.org<mailto:bljones@hdfgroup.org>> wrote:
Hi Jarom,
In case it might help, there are C and Fortran code examples in the User's Guide that do almost exactly what Francesc mentioned.
See Section 6.5.2.2.1 in the "HDF5 Datatypes" chapter of the HDF5 User's Guide. The link to section 6.5.2 is:
https://support.hdfgroup.org/HDF5/doc/UG/HDF5_Users_Guide-Responsive%20HTML5/index.html#t=HDF5_Users_Guide%2FDatatypes%2FHDF5_Datatypes.htm%23TOC_6_5_2_Definition_ofbc-14&rhtocid=6.3.0_2
Scroll down to 6.5.2.2.1, "Compound Datatypes". Scroll down a little further and you will see "Code Example 6-9" for C and "Code Example 6-10" for Fortran.
-Barbara
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Francesc Altet
Sent: Wednesday, February 22, 2017 2:02 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Complex numbers (1.10.x update?)
Hi Jarom,
For what is worth, in PyTables and h5py (and possibly in others libraries too) we used the convention of declaring the complex type by using a compound type as follows:
"""
The H5T_COMPOUND type class contains two members. Both members must have the H5T_FLOAT atomic datatype class. The name of the first member should be "r" and represents the real part. The name of the second member should be "i" and represents the imaginary part. The precision property of both of the H5T_FLOAT members must be either 32 significant bits (e.g. H5T_NATIVE_FLOAT) or 64 significant bits (e.g. H5T_NATIVE_DOUBLE). They represent Complex32 and Complex64 types respectively.
"""
Perhaps you may want to use this convention until a more formal one is implemented.
Francesc Alted
________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Nelson, Jarom <nelson99@llnl.gov<mailto:nelson99@llnl.gov>>
Sent: Wednesday, February 22, 2017 2:36:23 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Complex numbers (1.10.x update?)
Can we get a FAQ entry on the topic? Bonus points for an example user implementation.
Thanks for your answer and for a great library!
Jarom
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Barbara Jones
Sent: Tuesday, February 21, 2017 1:24 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] Complex numbers (1.10.x update?)
Hi Jarom,
We did originally plan to add support for complex types in HDF5-1.10. However, after the issue was examined closely, we decided to wait until we had a well-defined scope of
what it means to add support for complex types within the library.
The issue is still open but not assigned to be fixed in a specific release.
-Barbara
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Nelson, Jarom
Sent: Thursday, February 09, 2017 10:32 AM
To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Complex numbers (1.10.x update?)
At what point (if any?) will HDF5 support a complex floating point datatype (i.e. pair of real and imaginary floating point numbers) "out-of-the-box"?
This is probably something that comes up frequently, but I haven't found the current status of the question.
Searching the archives and the documentation, it seems that a complex floating point data type was at one point planned for the 1.10 release, however, I don't see anything about continuation of that plan in the documentation.
Ref: these two threads from 2010
https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2010-December/004011.html
https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2010-April/003049.html
Perhaps the answer can be implied from this point in that thread:
To be fair, because there's no one-size-fits-all complex number
solution outside Fortran, a one-size-fits-all cross-language HDF5
helper method to define a complex type would be difficult to get
right. Not technically difficult, just socially difficult.
- Rhys
Also, this seems like a likely topic for a FAQ. (There's one there for boolean datatype)
Jarom Nelson, LLNL
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
