Dimensions of length zero

Hi everyone,

I found out by experimenting that HDF5 does not seem to like empty arrays, i.e. arrays that have a zero in their dimension list. The associated error message is

  Zero sized dimension for non-unlimited dimension

However, I cannot find anything about this in the documentation (though of course I haven't read all of it). Is this a bug or a feature?

I find this restriction quite limiting because there is no obvious way to work around it. If I need to store a list of values and that list just happens to be empty, what can I do? I'd have to define my own representation of an empty list in terms of some non-empty value, perhaps with an attribute saying "this is really empty", but that complicates all application code and the whole file structure.

Konrad.

···

--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

Hi Konrad,

Do you think making the dimension unlimited would work for you?

Binh-Minh

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Konrad Hinsen
Sent: Wednesday, March 30, 2011 6:19 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Dimensions of length zero

Hi everyone,

I found out by experimenting that HDF5 does not seem to like empty arrays,
i.e. arrays that have a zero in their dimension list. The associated error
message is

  Zero sized dimension for non-unlimited dimension

However, I cannot find anything about this in the documentation (though of
course I haven't read all of it). Is this a bug or a feature?

I find this restriction quite limiting because there is no obvious way to
work around it. If I need to store a list of values and that list just
happens to be empty, what can I do? I'd have to define my own representation
of an empty list in terms of some non-empty value, perhaps with an attribute
saying "this is really empty", but that complicates all application code and
the whole file structure.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Internal Virus Database is out-of-date.
Checked by AVG.
Version: 7.5.549 / Virus Database: 270.9.0/1778 - Release Date: 11/9/2008
2:14 PM

I have had the same problem and worked around it by writing an empty
array as a special data type. It's really a nuisance.
You could say it was a design error to use 0 to indicate an unlimited
dimension. -1 (or maybe MAX_INT) would have been better. An axis length
0 conveys information (especially for vectors).

I doubt if the solution of using an unlimited dimension will work when
writing an array as an attribute.

AFAIK the same problem exists when having an empty string for a
variable length string field.

Cheers,
Ger

"Binh-Minh Ribler" <bmribler@hdfgroup.org> 3/30/2011 2:04 PM >>>

Hi Konrad,

Do you think making the dimension unlimited would work for you?

Binh-Minh

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Konrad Hinsen
Sent: Wednesday, March 30, 2011 6:19 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Dimensions of length zero

Hi everyone,

I found out by experimenting that HDF5 does not seem to like empty
arrays,
i.e. arrays that have a zero in their dimension list. The associated
error
message is

Zero sized dimension for non-unlimited dimension

However, I cannot find anything about this in the documentation (though
of
course I haven't read all of it). Is this a bug or a feature?

I find this restriction quite limiting because there is no obvious way
to
work around it. If I need to store a list of values and that list just
happens to be empty, what can I do? I'd have to define my own
representation
of an empty list in terms of some non-empty value, perhaps with an
attribute
saying "this is really empty", but that complicates all application
code and
the whole file structure.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Internal Virus Database is out-of-date.
Checked by AVG.
Version: 7.5.549 / Virus Database: 270.9.0/1778 - Release Date:
11/9/2008
2:14 PM

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

For datasets, yes, but I also (and even more frequently) need emtpy arrays as attributes, for which unlimited dimensions are not possible.

Konrad.

···

On 30 Mar, 2011, at 14:04 , Binh-Minh Ribler wrote:

Do you think making the dimension unlimited would work for you?

--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

Hi all,

I have had the same problem and worked around it by writing an empty array as a special data type. It's really a nuisance.
You could say it was a design error to use 0 to indicate an unlimited dimension. -1 (or maybe MAX_INT) would have been better.

  We don't use 0 for indicating an unlimited dimension (and we do use -1 :-).

An axis length 0 conveys information (especially for vectors).

  Yes, it does and we already have an issue to correct this in our bug tracker. I'll see if we can bump up the priority for the 1.8.7 release.

  One possibility as a workaround in the meanwhile may be to use the H5S_NULL dataspace class, to indicate that an attribute or dataset has no elements.

  Thanks for the feedback,
    Quincey

···

On Mar 30, 2011, at 7:33 AM, Ger van Diepen wrote:

I doubt if the solution of using an unlimited dimension will work when writing an array as an attribute.

AFAIK the same problem exists when having an empty string for a variable length string field.

Cheers,
Ger

>>> "Binh-Minh Ribler" <bmribler@hdfgroup.org> 3/30/2011 2:04 PM >>>
Hi Konrad,

Do you think making the dimension unlimited would work for you?

Binh-Minh

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Konrad Hinsen
Sent: Wednesday, March 30, 2011 6:19 AM
To: HDF Users Discussion List
Subject: [Hdf-forum] Dimensions of length zero

Hi everyone,

I found out by experimenting that HDF5 does not seem to like empty arrays,
i.e. arrays that have a zero in their dimension list. The associated error
message is

Zero sized dimension for non-unlimited dimension

However, I cannot find anything about this in the documentation (though of
course I haven't read all of it). Is this a bug or a feature?

I find this restriction quite limiting because there is no obvious way to
work around it. If I need to store a list of values and that list just
happens to be empty, what can I do? I'd have to define my own representation
of an empty list in terms of some non-empty value, perhaps with an attribute
saying "this is really empty", but that complicates all application code and
the whole file structure.

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Internal Virus Database is out-of-date.
Checked by AVG.
Version: 7.5.549 / Virus Database: 270.9.0/1778 - Release Date: 11/9/2008
2:14 PM

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

An axis length 0 conveys information (especially for vectors).

  Yes, it does and we already have an issue to correct this in our bug tracker. I'll see if we can bump up the priority for the 1.8.7 release.

That would be great :slight_smile:

  One possibility as a workaround in the meanwhile may be to use the H5S_NULL dataspace class, to indicate that an attribute or dataset has no elements.

In principle, this would be a solution for the majority of my use cases, the exception being those where only one of many dimensions of an array has zero length and the length of the others do matter. But the main practical problem I see with H5S_NULL is that popular high-level interfaces to HDF5 (the Java object layer, h5py and PyTables for Python) don't provide access to it. I know that I should complain to the authors of those interfaces...

Konrad.

···

On 30 Mar, 2011, at 14:50 , Quincey Koziol wrote:
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

Konrad,

Our Group asked me to follow up the issue you reported. I understand that you want to declare a dimension size to be zero and you won't write any data to it. A data space with zero dimension size is totally empty. It only shows the number (or rank) of the dimensions. Is this correct?

Thanks.

Ray

···

On Mar 30, 2011, at 10:23 AM, Konrad Hinsen wrote:

On 30 Mar, 2011, at 14:50 , Quincey Koziol wrote:

An axis length 0 conveys information (especially for vectors).

  Yes, it does and we already have an issue to correct this in our bug tracker. I'll see if we can bump up the priority for the 1.8.7 release.

That would be great :slight_smile:

  One possibility as a workaround in the meanwhile may be to use the H5S_NULL dataspace class, to indicate that an attribute or dataset has no elements.

In principle, this would be a solution for the majority of my use cases, the exception being those where only one of many dimensions of an array has zero length and the length of the others do matter. But the main practical problem I see with H5S_NULL is that popular high-level interfaces to HDF5 (the Java object layer, h5py and PyTables for Python) don't provide access to it. I know that I should complain to the authors of those interfaces...

Konrad.
--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Ray,

Our Group asked me to follow up the issue you reported. I understand that you want to declare a dimension size to be zero and you won't write any data to it. A data space with zero dimension size is totally empty. It only shows the number (or rank) of the dimensions. Is this correct?

Almost. Such a dataspace is indeed empty (no data), but it still contains more information than the rank:
1) The datatype
2) The size of each dimension, of which at least one is zero.

Point 2) is the main difference to H5S_EMPTY. It is useful to have the full dimension information to avoid introducing special cases into applications.

An example: suppose my data model specifies that the coordinates of N points are stored in an array of type float and dimension [N 3]. A validator for this data model would then check that the size of the second dimension is 3, which should be possible even if N happens to be zero.

Konrad.

···

--
---------------------------------------------------------------------
Konrad Hinsen
Centre de Biophysique Moléculaire, CNRS Orléans
Synchrotron Soleil - Division Expériences
Saint Aubin - BP 48
91192 Gif sur Yvette Cedex, France
Tel. +33-1 69 35 97 15
E-Mail: research AT khinsen DOT fastmail DOT net
---------------------------------------------------------------------