Boolean values

Is there a way to read/write compact boolean or bitfield values? I see the
bitfield types but I can't understand how it's different than writing ints.
The API examples show hand packing/unpacking of the bits. How is
H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?

Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so
it's easy to work with when loaded but stored compactly.

Thanks,
David

I don’t think the hdf Boolean is any different than an integer. If you create a Boolean dataset and query the type, it comes back that it’s an int. You can’t tell after the fact that the data was specified as Boolean.

I suppose you can create a packed Boolean type, but you’ll have to do the grunt work of muxing and de-muxing the Booleans.

Ultimately, I stuck my bools in separate sbytes.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Wednesday, February 10, 2016 4:12 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Boolean values

Is there a way to read/write compact boolean or bitfield values? I see the bitfield types but I can't understand how it's different than writing ints. The API examples show hand packing/unpacking of the bits. How is H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so it's easy to work with when loaded but stored compactly.

Thanks,
David

You could try making your dataset using an N-bit type based on a copy of an 8-bit type, and then using the N-bit filter. You can then end up handling each Boolean value as a byte in memory, but they’re stored more compactly on disk. Might even compress well too.

Dan

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of SMITCH12@harris.com
Sent: 10 February 2016 21:23
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values

I don’t think the hdf Boolean is any different than an integer. If you create a Boolean dataset and query the type, it comes back that it’s an int. You can’t tell after the fact that the data was specified as Boolean.

I suppose you can create a packed Boolean type, but you’ll have to do the grunt work of muxing and de-muxing the Booleans.

Ultimately, I stuck my bools in separate sbytes.

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Wednesday, February 10, 2016 4:12 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Boolean values

Is there a way to read/write compact boolean or bitfield values? I see the bitfield types but I can't understand how it's different than writing ints. The API examples show hand packing/unpacking of the bits. How is H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so it's easy to work with when loaded but stored compactly.

Thanks,
David

Daniel: What are the N-bit type and filter and types you are referring to?
Do you mean something I would write or are you talking about something in
HDF5?

Also can anyone explain how H5T_NATIVE_B8 is different from
H5T_NATIVE_UINT8.

···

On Thu, Feb 11, 2016 at 1:29 AM, Daniel Tetlow <daniel.tetlow@optasense.com> wrote:

You could try making your dataset using an N-bit type based on a copy of
an 8-bit type, and then using the N-bit filter. You can then end up
handling each Boolean value as a byte in memory, but they’re stored more
compactly on disk. Might even compress well too.

Dan

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *SMITCH12@harris.com
*Sent:* 10 February 2016 21:23
*To:* HDF Users Discussion List
*Subject:* Re: [Hdf-forum] Boolean values

I don’t think the hdf Boolean is any different than an integer. If you
create a Boolean dataset and query the type, it comes back that it’s an
int. You can’t tell after the fact that the data was specified as Boolean.

I suppose you can create a packed Boolean type, but you’ll have to do the
grunt work of muxing and de-muxing the Booleans.

Ultimately, I stuck my bools in separate sbytes.

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org
<hdf-forum-bounces@lists.hdfgroup.org>] *On Behalf Of *David
*Sent:* Wednesday, February 10, 2016 4:12 PM
*To:* HDF Users Discussion List
*Subject:* [Hdf-forum] Boolean values

Is there a way to read/write compact boolean or bitfield values? I see the
bitfield types but I can't understand how it's different than writing ints.
The API examples show hand packing/unpacking of the bits. How is
H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?

Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so
it's easy to work with when loaded but stored compactly.

Thanks,

David

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi,

I was referring to features in HDF5. Looks like these pages in their docs might help:

https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetNbit

https://www.hdfgroup.org/HDF5/doc/UG/HDF5_Users_Guide-Responsive%20HTML5/#t=HDF5_Users_Guide%2FDatasets%2FHDF5_Datasets.htm%23TOC_5_6_1_Using_the_N_bitbc-21&rhtocid=5.4.0_1

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: 11 February 2016 16:10
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values

Daniel: What are the N-bit type and filter and types you are referring to? Do you mean something I would write or are you talking about something in HDF5?
Also can anyone explain how H5T_NATIVE_B8 is different from H5T_NATIVE_UINT8.

On Thu, Feb 11, 2016 at 1:29 AM, Daniel Tetlow <daniel.tetlow@optasense.com<mailto:daniel.tetlow@optasense.com>> wrote:
You could try making your dataset using an N-bit type based on a copy of an 8-bit type, and then using the N-bit filter. You can then end up handling each Boolean value as a byte in memory, but they’re stored more compactly on disk. Might even compress well too.

Dan

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of SMITCH12@harris.com<mailto:SMITCH12@harris.com>
Sent: 10 February 2016 21:23
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values

I don’t think the hdf Boolean is any different than an integer. If you create a Boolean dataset and query the type, it comes back that it’s an int. You can’t tell after the fact that the data was specified as Boolean.

I suppose you can create a packed Boolean type, but you’ll have to do the grunt work of muxing and de-muxing the Booleans.

Ultimately, I stuck my bools in separate sbytes.

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Wednesday, February 10, 2016 4:12 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Boolean values

Is there a way to read/write compact boolean or bitfield values? I see the bitfield types but I can't understand how it's different than writing ints. The API examples show hand packing/unpacking of the bits. How is H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so it's easy to work with when loaded but stored compactly.

Thanks,
David

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

That looks perfect. Thank you so much. Do you know what is the purpose of
the H5T_NATIVE_Bxx types?

Daniel.

It looks like HDF5 data types are all sized in bytes. So while you can
define a 1 bit type it is always stored as a byte. I guess I need to make
my own packer.

I will however use the N-bit stuff for another issue I have which is 24-bit
data. Much more straight forward.

David

···

On Thu, Feb 11, 2016 at 8:39 AM, David <list@aue.org> wrote:

That looks perfect. Thank you so much. Do you know what is the purpose of
the H5T_NATIVE_Bxx types?

David,

That looks perfect. Thank you so much. Do you know what is the purpose of the H5T_NATIVE_Bxx types?

Good question :slight_smile:

B stands for bitfield. See https://www.hdfgroup.org/HDF5/doc1.6/Datatypes.htm, Section 6 and current https://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.html

Elena

···

On Feb 11, 2016, at 10:39 AM, David <list@aue.org> wrote:

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Daniel,

It turns out I just didn't fully understand the n-bit types and filters
yet. The bit types always store in bytes in but then the filter will pack
things further on read/write. So I got what I wanted working with bytes in
memory but bits on disk for bool data sets. Thanks again for your
suggestions.

David

···

On Sat, Feb 13, 2016 at 2:35 PM, David <list@aue.org> wrote:

Daniel.

It looks like HDF5 data types are all sized in bytes. So while you can
define a 1 bit type it is always stored as a byte. I guess I need to make
my own packer.

I will however use the N-bit stuff for another issue I have which is
24-bit data. Much more straight forward.

David

On Thu, Feb 11, 2016 at 8:39 AM, David <list@aue.org> wrote:

That looks perfect. Thank you so much. Do you know what is the purpose of
the H5T_NATIVE_Bxx types?

Hi Elena,

What I don't understand is how H5T_NATIVE_B32 is different from
H5T_NATIVE_UINT32.

David

···

On Wed, Feb 17, 2016 at 10:43 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

David,

On Feb 11, 2016, at 10:39 AM, David <list@aue.org> wrote:

> That looks perfect. Thank you so much. Do you know what is the purpose
of the H5T_NATIVE_Bxx types?
Good question :slight_smile:

B stands for bitfield. See
https://www.hdfgroup.org/HDF5/doc1.6/Datatypes.htm, Section 6 and current
https://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.html

Elena
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

David,

Integer type can be converted by the HDF5 library to other numeric types; bitfield cannot be converted. I hope other HDF folks will chime in if I am wrong.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Feb 17, 2016, at 2:04 PM, David <list@aue.org<mailto:list@aue.org>> wrote:

Hi Elena,

What I don't understand is how H5T_NATIVE_B32 is different from H5T_NATIVE_UINT32.

David

On Wed, Feb 17, 2016 at 10:43 AM, Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>> wrote:
David,

On Feb 11, 2016, at 10:39 AM, David <list@aue.org<mailto:list@aue.org>> wrote:

That looks perfect. Thank you so much. Do you know what is the purpose of the H5T_NATIVE_Bxx types?

Good question :slight_smile:

B stands for bitfield. See https://www.hdfgroup.org/HDF5/doc1.6/Datatypes.htm, Section 6 and current https://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.html

Elena

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi,

Good to know that works!

If you anticipate having a large dataset of bits, you might also want to try enabling the deflate filter *after* the n-bit filter to make the size on disk even smaller.

Cheers,

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: 17 February 2016 06:23
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values

Hi Daniel,
It turns out I just didn't fully understand the n-bit types and filters yet. The bit types always store in bytes in but then the filter will pack things further on read/write. So I got what I wanted working with bytes in memory but bits on disk for bool data sets. Thanks again for your suggestions.
David

On Sat, Feb 13, 2016 at 2:35 PM, David <list@aue.org<mailto:list@aue.org>> wrote:
Daniel.

It looks like HDF5 data types are all sized in bytes. So while you can define a 1 bit type it is always stored as a byte. I guess I need to make my own packer.
I will however use the N-bit stuff for another issue I have which is 24-bit data. Much more straight forward.

David

On Thu, Feb 11, 2016 at 8:39 AM, David <list@aue.org<mailto:list@aue.org>> wrote:
That looks perfect. Thank you so much. Do you know what is the purpose of the H5T_NATIVE_Bxx types?