Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

HDFView / Java HDF Object Package
1

I am new to HDF5.

Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

For example, when importing a 720 x 678 JPEG image that contains 114,173 bytes, with HDFView or a Python program, the resulting HDF5 image dataset contains 1.4 MB (720 x 678 x 3 for the RGB values = 1,464,480 bytes). What is the best strategy for compressing this data so that it can still be viewed with HDFView?

Thanks for your kind consideration.

2

Tim, have a look at h5repack

http://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Repack

Before coding anything you can experiment with the chunk size and
compression method/level for the image dataset.

Best, G.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Tim
Sent: Thursday, February 06, 2014 11:43 AM
To: HDF Forum
Subject: [Hdf-forum] Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

I am new to HDF5.

Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

For example, when importing a 720 x 678 JPEG image that contains 114,173 bytes, with HDFView or a Python program, the resulting HDF5 image dataset contains 1.4 MB (720 x 678 x 3 for the RGB values = 1,464,480 bytes). What is the best strategy for compressing this data so that it can still be viewed with HDFView?

Thanks for your kind consideration.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

3

Thanks much for the guidance - h5repack will be a big help.

Is there a tradeoff between storing information in a space-efficient way and the ability to analyze and otherwise manipulate the image?

···

On 2/6/14 9:59 AM, Gerd Heber wrote:

Tim, have a look at h5repack

http://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Repack

Before coding anything you can experiment with the chunk size and
compression method/level for the image dataset.

Best, G.

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Tim
Sent: Thursday, February 06, 2014 11:43 AM
To: HDF Forum
Subject: [Hdf-forum] Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

I am new to HDF5.

Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

For example, when importing a 720 x 678 JPEG image that contains 114,173 bytes, with HDFView or a Python program, the resulting HDF5 image dataset contains 1.4 MB (720 x 678 x 3 for the RGB values = 1,464,480 bytes). What is the best strategy for compressing this data so that it can still be viewed with HDFView?

Thanks for your kind consideration.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

4

Yes, that will be the case. The compression happens at the level
of chunks. When you read or write a (set of) pixel(s), the HDF5 library
will determine the affected chunks and read/write and de-/compress them.
Let's say you chose to shape the chunks like 1D scan lines, but you read
only small contiguous 2D blocks of pixels, the library would still have to read and
decompress all overlapping scan lines. You can fine tune parameters like chunk
cache sizes etc., but it always help to do a little back-of-the-envelope assessment
of what's going on. You might opt to shape the chunks more like squares,
but that won't give you the best performance for row-oriented access.
You'll have to experiment and see what works best for your application.
There's no perfect general solution. The only thing that's certain is
that you should stay away from creating a ton of tiny chunks. That'll
cause your chunk index to balloon to the point that the book keeping takes
more space than the actual dataset.

Have a look at Elena's presentation on things to bear in mind when chunking:

Best, G.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Tim
Sent: Thursday, February 06, 2014 1:03 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

Thanks much for the guidance - h5repack will be a big help.

Is there a tradeoff between storing information in a space-efficient way and the ability to analyze and otherwise manipulate the image?

On 2/6/14 9:59 AM, Gerd Heber wrote:

Tim, have a look at h5repack

http://www.hdfgroup.org/HDF5/doc/RM/Tools.html#Tools-Repack

Before coding anything you can experiment with the chunk size and
compression method/level for the image dataset.

Best, G.

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On
Behalf Of Tim
Sent: Thursday, February 06, 2014 11:43 AM
To: HDF Forum
Subject: [Hdf-forum] Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

I am new to HDF5.

Is there a best practice for storing image data to be viewed with HDFView in a space-efficient way?

For example, when importing a 720 x 678 JPEG image that contains 114,173 bytes, with HDFView or a Python program, the resulting HDF5 image dataset contains 1.4 MB (720 x 678 x 3 for the RGB values = 1,464,480 bytes). What is the best strategy for compressing this data so that it can still be viewed with HDFView?

Thanks for your kind consideration.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgro
up.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgro
up.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org