Is there a way to read/write compact boolean or bitfield values? I see the
bitfield types but I can't understand how it's different than writing ints.
The API examples show hand packing/unpacking of the bits. How is
H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so
it's easy to work with when loaded but stored compactly.
Thanks,
David
I don’t think the hdf Boolean is any different than an integer. If you create a Boolean dataset and query the type, it comes back that it’s an int. You can’t tell after the fact that the data was specified as Boolean.
I suppose you can create a packed Boolean type, but you’ll have to do the grunt work of muxing and de-muxing the Booleans.
Ultimately, I stuck my bools in separate sbytes.
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Wednesday, February 10, 2016 4:12 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Boolean values
Is there a way to read/write compact boolean or bitfield values? I see the bitfield types but I can't understand how it's different than writing ints. The API examples show hand packing/unpacking of the bits. How is H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so it's easy to work with when loaded but stored compactly.
Thanks,
David
You could try making your dataset using an N-bit type based on a copy of an 8-bit type, and then using the N-bit filter. You can then end up handling each Boolean value as a byte in memory, but they’re stored more compactly on disk. Might even compress well too.
Dan
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of SMITCH12@harris.com
Sent: 10 February 2016 21:23
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values
I don’t think the hdf Boolean is any different than an integer. If you create a Boolean dataset and query the type, it comes back that it’s an int. You can’t tell after the fact that the data was specified as Boolean.
I suppose you can create a packed Boolean type, but you’ll have to do the grunt work of muxing and de-muxing the Booleans.
Ultimately, I stuck my bools in separate sbytes.
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Wednesday, February 10, 2016 4:12 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Boolean values
Is there a way to read/write compact boolean or bitfield values? I see the bitfield types but I can't understand how it's different than writing ints. The API examples show hand packing/unpacking of the bits. How is H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so it's easy to work with when loaded but stored compactly.
Thanks,
David
Daniel: What are the N-bit type and filter and types you are referring to?
Do you mean something I would write or are you talking about something in
HDF5?
Also can anyone explain how H5T_NATIVE_B8 is different from
H5T_NATIVE_UINT8.
On Thu, Feb 11, 2016 at 1:29 AM, Daniel Tetlow <daniel.tetlow@optasense.com> wrote:
You could try making your dataset using an N-bit type based on a copy of
an 8-bit type, and then using the N-bit filter. You can then end up
handling each Boolean value as a byte in memory, but they’re stored more
compactly on disk. Might even compress well too.Dan
*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *SMITCH12@harris.com
*Sent:* 10 February 2016 21:23
*To:* HDF Users Discussion List
*Subject:* Re: [Hdf-forum] Boolean valuesI don’t think the hdf Boolean is any different than an integer. If you
create a Boolean dataset and query the type, it comes back that it’s an
int. You can’t tell after the fact that the data was specified as Boolean.I suppose you can create a packed Boolean type, but you’ll have to do the
grunt work of muxing and de-muxing the Booleans.Ultimately, I stuck my bools in separate sbytes.
*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org
<hdf-forum-bounces@lists.hdfgroup.org>] *On Behalf Of *David
*Sent:* Wednesday, February 10, 2016 4:12 PM
*To:* HDF Users Discussion List
*Subject:* [Hdf-forum] Boolean valuesIs there a way to read/write compact boolean or bitfield values? I see the
bitfield types but I can't understand how it's different than writing ints.
The API examples show hand packing/unpacking of the bits. How is
H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so
it's easy to work with when loaded but stored compactly.Thanks,
David
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi,
I was referring to features in HDF5. Looks like these pages in their docs might help:
https://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetNbit
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: 11 February 2016 16:10
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values
Daniel: What are the N-bit type and filter and types you are referring to? Do you mean something I would write or are you talking about something in HDF5?
Also can anyone explain how H5T_NATIVE_B8 is different from H5T_NATIVE_UINT8.
On Thu, Feb 11, 2016 at 1:29 AM, Daniel Tetlow <daniel.tetlow@optasense.com<mailto:daniel.tetlow@optasense.com>> wrote:
You could try making your dataset using an N-bit type based on a copy of an 8-bit type, and then using the N-bit filter. You can then end up handling each Boolean value as a byte in memory, but they’re stored more compactly on disk. Might even compress well too.
Dan
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of SMITCH12@harris.com<mailto:SMITCH12@harris.com>
Sent: 10 February 2016 21:23
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values
I don’t think the hdf Boolean is any different than an integer. If you create a Boolean dataset and query the type, it comes back that it’s an int. You can’t tell after the fact that the data was specified as Boolean.
I suppose you can create a packed Boolean type, but you’ll have to do the grunt work of muxing and de-muxing the Booleans.
Ultimately, I stuck my bools in separate sbytes.
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Wednesday, February 10, 2016 4:12 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] Boolean values
Is there a way to read/write compact boolean or bitfield values? I see the bitfield types but I can't understand how it's different than writing ints. The API examples show hand packing/unpacking of the bits. How is H5T_NATIVE_B8 different from H5T_NATIVE_UINT8?
Ideally I'd like 8-bit bools in memory packed into 1-bit bools on disk so it's easy to work with when loaded but stored compactly.
Thanks,
David
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
That looks perfect. Thank you so much. Do you know what is the purpose of
the H5T_NATIVE_Bxx types?
Daniel.
It looks like HDF5 data types are all sized in bytes. So while you can
define a 1 bit type it is always stored as a byte. I guess I need to make
my own packer.
I will however use the N-bit stuff for another issue I have which is 24-bit
data. Much more straight forward.
David
On Thu, Feb 11, 2016 at 8:39 AM, David <list@aue.org> wrote:
That looks perfect. Thank you so much. Do you know what is the purpose of
the H5T_NATIVE_Bxx types?
David,
That looks perfect. Thank you so much. Do you know what is the purpose of the H5T_NATIVE_Bxx types?
Good question 
B stands for bitfield. See https://www.hdfgroup.org/HDF5/doc1.6/Datatypes.htm, Section 6 and current https://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.html
Elena
On Feb 11, 2016, at 10:39 AM, David <list@aue.org> wrote:
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi Daniel,
It turns out I just didn't fully understand the n-bit types and filters
yet. The bit types always store in bytes in but then the filter will pack
things further on read/write. So I got what I wanted working with bytes in
memory but bits on disk for bool data sets. Thanks again for your
suggestions.
David
On Sat, Feb 13, 2016 at 2:35 PM, David <list@aue.org> wrote:
Daniel.
It looks like HDF5 data types are all sized in bytes. So while you can
define a 1 bit type it is always stored as a byte. I guess I need to make
my own packer.I will however use the N-bit stuff for another issue I have which is
24-bit data. Much more straight forward.David
On Thu, Feb 11, 2016 at 8:39 AM, David <list@aue.org> wrote:
That looks perfect. Thank you so much. Do you know what is the purpose of
the H5T_NATIVE_Bxx types?
Hi Elena,
What I don't understand is how H5T_NATIVE_B32 is different from
H5T_NATIVE_UINT32.
David
On Wed, Feb 17, 2016 at 10:43 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:
David,
On Feb 11, 2016, at 10:39 AM, David <list@aue.org> wrote:
> That looks perfect. Thank you so much. Do you know what is the purpose
of the H5T_NATIVE_Bxx types?
Good questionB stands for bitfield. See
https://www.hdfgroup.org/HDF5/doc1.6/Datatypes.htm, Section 6 and current
https://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.htmlElena
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
David,
Integer type can be converted by the HDF5 library to other numeric types; bitfield cannot be converted. I hope other HDF folks will chime in if I am wrong.
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Feb 17, 2016, at 2:04 PM, David <list@aue.org<mailto:list@aue.org>> wrote:
Hi Elena,
What I don't understand is how H5T_NATIVE_B32 is different from H5T_NATIVE_UINT32.
David
On Wed, Feb 17, 2016 at 10:43 AM, Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>> wrote:
David,
On Feb 11, 2016, at 10:39 AM, David <list@aue.org<mailto:list@aue.org>> wrote:
That looks perfect. Thank you so much. Do you know what is the purpose of the H5T_NATIVE_Bxx types?
Good question
B stands for bitfield. See https://www.hdfgroup.org/HDF5/doc1.6/Datatypes.htm, Section 6 and current https://www.hdfgroup.org/HDF5/doc/RM/PredefDTypes.html
Elena
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi,
Good to know that works!
If you anticipate having a large dataset of bits, you might also want to try enabling the deflate filter *after* the n-bit filter to make the size on disk even smaller.
Cheers,
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: 17 February 2016 06:23
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Boolean values
Hi Daniel,
It turns out I just didn't fully understand the n-bit types and filters yet. The bit types always store in bytes in but then the filter will pack things further on read/write. So I got what I wanted working with bytes in memory but bits on disk for bool data sets. Thanks again for your suggestions.
David
On Sat, Feb 13, 2016 at 2:35 PM, David <list@aue.org<mailto:list@aue.org>> wrote:
Daniel.
It looks like HDF5 data types are all sized in bytes. So while you can define a 1 bit type it is always stored as a byte. I guess I need to make my own packer.
I will however use the N-bit stuff for another issue I have which is 24-bit data. Much more straight forward.
David
On Thu, Feb 11, 2016 at 8:39 AM, David <list@aue.org<mailto:list@aue.org>> wrote:
That looks perfect. Thank you so much. Do you know what is the purpose of the H5T_NATIVE_Bxx types?
