Direct access to compressed block of data?

Hdf-forum archives
1

I am looking for a way to read in a chunk of data from a HDF5 file without
the decompression step. This would be similar to how H5OCopy works when
copying between two hdf5 files, except I want the data to stay resident in
memory. Is it possible to read the compressed bytes of a chunk into a byte
array?

For additional background, data is stored using the deflate filter. I
define my read dimensions to match my original chunking parameters to read in one exact chunk at a time.

Thanks for the help.

Jebb

2

Jebb,

Would it not work to H5OCopy your array into a memory file? See http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html, section 8.8 on the CORE driver.

--dan

···

On 04/25/14 19:02, Jebb Q Stewart wrote:

I am looking for a way to read in a chunk of data from a HDF5 file without
the decompression step. This would be similar to how H5OCopy works when
copying between two hdf5 files, except I want the data to stay resident in
memory. Is it possible to read the compressed bytes of a chunk into a byte
array?

For additional background, data is stored using the deflate filter. I
define my read dimensions to match my original chunking parameters to read in one exact chunk at a time.

Thanks for the help.

Jebb

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--
Daniel Kahn
Science Systems and Applications Inc.
301-867-2162

3

Dan,

Thanks for the feedback. This isn't quite what I am looking for. While I
can read into a memory file, data is still decompressed when read a
particular hyperslab. I would like to extract the raw bytes of the
compressed hyperslab. My data are 3 dimensional float arrays stored in
compressed (deflate) slabs by vertical level. Right now I extract the
float values for a particular vertical level and compress them before
sending across the wire to a remote location. I would like to directly
read the compressed slab stored within the file and send it, skipping the
extra decompress/compress step. My hyperslab dimensions match my specified
chunk dimensions and in theory the decompression wouldn't be required since
I am not extracting a subset of my chunk dimensions. I was hoping to get
direct access to the deflated hyperslab as a byte array. I don't see a way
to do this with the in memory file. Did I miss something?

Jebb

···

--
Jebb Stewart
NOAA's Earth System Research Laboratory (ESRL)
Global Systems Division
325 Broadway, R/GSD6, Boulder, CO 80305
303-497-6724

On Tue, Apr 29, 2014 at 7:33 AM, Daniel Kahn <daniel.kahn@ssaihq.com> wrote:

Jebb,

Would it not work to H5OCopy your array into a memory file? See
http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html, section 8.8 on
the CORE driver.

--dan

On 04/25/14 19:02, Jebb Q Stewart wrote:

I am looking for a way to read in a chunk of data from a HDF5 file without
the decompression step. This would be similar to how H5OCopy works when
copying between two hdf5 files, except I want the data to stay resident in
memory. Is it possible to read the compressed bytes of a chunk into a byte
array?

For additional background, data is stored using the deflate filter. I
define my read dimensions to match my original chunking parameters to read
in one exact chunk at a time.

Thanks for the help.

Jebb

_______________________________________________
Hdf-forum is for HDF software users discussion.Hdf-forum@lists.hdfgroup.orghttp://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--
Daniel Kahn
Science Systems and Applications Inc.301-867-2162

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

4

Are you looking to do something like this?

Dana

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Jebb Stewart - NOAA Affiliate
Sent: Tuesday, April 29, 2014 12:45 PM
To: daniel.kahn@ssaihq.com; HDF Users Discussion List
Subject: Re: [Hdf-forum] Direct access to compressed block of data?

Dan,

Thanks for the feedback. This isn't quite what I am looking for. While I can read into a memory file, data is still decompressed when read a particular hyperslab. I would like to extract the raw bytes of the compressed hyperslab. My data are 3 dimensional float arrays stored in compressed (deflate) slabs by vertical level. Right now I extract the float values for a particular vertical level and compress them before sending across the wire to a remote location. I would like to directly read the compressed slab stored within the file and send it, skipping the extra decompress/compress step. My hyperslab dimensions match my specified chunk dimensions and in theory the decompression wouldn't be required since I am not extracting a subset of my chunk dimensions. I was hoping to get direct access to the deflated hyperslab as a byte array. I don't see a way to do this with the in memory file. Did I miss something?

Jebb

--
Jebb Stewart
NOAA's Earth System Research Laboratory (ESRL)
Global Systems Division
325 Broadway, R/GSD6, Boulder, CO 80305
303-497-6724

On Tue, Apr 29, 2014 at 7:33 AM, Daniel Kahn <daniel.kahn@ssaihq.com<mailto:daniel.kahn@ssaihq.com>> wrote:
Jebb,

Would it not work to H5OCopy your array into a memory file? See http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html, section 8.8 on the CORE driver.

--dan

On 04/25/14 19:02, Jebb Q Stewart wrote:

I am looking for a way to read in a chunk of data from a HDF5 file without
the decompression step. This would be similar to how H5OCopy works when
copying between two hdf5 files, except I want the data to stay resident in
memory. Is it possible to read the compressed bytes of a chunk into a byte
array?

For additional background, data is stored using the deflate filter. I
define my read dimensions to match my original chunking parameters to read in one exact chunk at a time.

Thanks for the help.

Jebb

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--

Daniel Kahn

Science Systems and Applications Inc.

301-867-2162<tel:301-867-2162>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

5

So close. Exactly in reverse. Something along the lines of
DirectChunkRead. I would like to read the chunk directly from file into
memory for decompression later.

Thanks.

Jebb

···

--
Jebb Stewart
NOAA's Earth System Research Laboratory (ESRL)
Global Systems Division
325 Broadway, R/GSD6, Boulder, CO 80305
303-497-6724

On Tue, Apr 29, 2014 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org>wrote:

Are you looking to do something like this?

http://www.hdfgroup.org/HDF5/doc/Advanced/DirectChunkWrite/UsingDirectChunkWrite.pdf

Dana

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Jebb Stewart - NOAA Affiliate
*Sent:* Tuesday, April 29, 2014 12:45 PM
*To:* daniel.kahn@ssaihq.com; HDF Users Discussion List
*Subject:* Re: [Hdf-forum] Direct access to compressed block of data?

Dan,

Thanks for the feedback. This isn't quite what I am looking for. While I
can read into a memory file, data is still decompressed when read a
particular hyperslab. I would like to extract the raw bytes of the
compressed hyperslab. My data are 3 dimensional float arrays stored in
compressed (deflate) slabs by vertical level. Right now I extract the
float values for a particular vertical level and compress them before
sending across the wire to a remote location. I would like to directly
read the compressed slab stored within the file and send it, skipping the
extra decompress/compress step. My hyperslab dimensions match my specified
chunk dimensions and in theory the decompression wouldn't be required since
I am not extracting a subset of my chunk dimensions. I was hoping to get
direct access to the deflated hyperslab as a byte array. I don't see a way
to do this with the in memory file. Did I miss something?

Jebb

--
Jebb Stewart
NOAA's Earth System Research Laboratory (ESRL)
Global Systems Division
325 Broadway, R/GSD6, Boulder, CO 80305
303-497-6724

On Tue, Apr 29, 2014 at 7:33 AM, Daniel Kahn <daniel.kahn@ssaihq.com> > wrote:

Jebb,

Would it not work to H5OCopy your array into a memory file? See
http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html, section 8.8 on
the CORE driver.

--dan

On 04/25/14 19:02, Jebb Q Stewart wrote:

I am looking for a way to read in a chunk of data from a HDF5 file without
the decompression step. This would be similar to how H5OCopy works when
copying between two hdf5 files, except I want the data to stay resident in
memory. Is it possible to read the compressed bytes of a chunk into a byte
array?

For additional background, data is stored using the deflate filter. I
define my read dimensions to match my original chunking parameters to read
in one exact chunk at a time.

Thanks for the help.

Jebb

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--

Daniel Kahn

Science Systems and Applications Inc.

301-867-2162

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

6

Unfortunately, there is no symmetric H5DOread_chunk() function, though I expect that this could be added if funding for the feature were available.

Dana

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Jebb Stewart - NOAA Affiliate
Sent: Tuesday, April 29, 2014 1:34 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Direct access to compressed block of data?

So close. Exactly in reverse. Something along the lines of DirectChunkRead. I would like to read the chunk directly from file into memory for decompression later.

Thanks.

Jebb

--
Jebb Stewart
NOAA's Earth System Research Laboratory (ESRL)
Global Systems Division
325 Broadway, R/GSD6, Boulder, CO 80305
303-497-6724

On Tue, Apr 29, 2014 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>> wrote:
Are you looking to do something like this?

Dana

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Jebb Stewart - NOAA Affiliate
Sent: Tuesday, April 29, 2014 12:45 PM
To: daniel.kahn@ssaihq.com<mailto:daniel.kahn@ssaihq.com>; HDF Users Discussion List
Subject: Re: [Hdf-forum] Direct access to compressed block of data?

Dan,

Thanks for the feedback. This isn't quite what I am looking for. While I can read into a memory file, data is still decompressed when read a particular hyperslab. I would like to extract the raw bytes of the compressed hyperslab. My data are 3 dimensional float arrays stored in compressed (deflate) slabs by vertical level. Right now I extract the float values for a particular vertical level and compress them before sending across the wire to a remote location. I would like to directly read the compressed slab stored within the file and send it, skipping the extra decompress/compress step. My hyperslab dimensions match my specified chunk dimensions and in theory the decompression wouldn't be required since I am not extracting a subset of my chunk dimensions. I was hoping to get direct access to the deflated hyperslab as a byte array. I don't see a way to do this with the in memory file. Did I miss something?

Jebb

--
Jebb Stewart
NOAA's Earth System Research Laboratory (ESRL)
Global Systems Division
325 Broadway, R/GSD6, Boulder, CO 80305
303-497-6724<tel:303-497-6724>

On Tue, Apr 29, 2014 at 7:33 AM, Daniel Kahn <daniel.kahn@ssaihq.com<mailto:daniel.kahn@ssaihq.com>> wrote:
Jebb,

Would it not work to H5OCopy your array into a memory file? See http://www.hdfgroup.org/HDF5/doc1.6/UG/08_TheFile.html, section 8.8 on the CORE driver.

--dan

On 04/25/14 19:02, Jebb Q Stewart wrote:

I am looking for a way to read in a chunk of data from a HDF5 file without
the decompression step. This would be similar to how H5OCopy works when
copying between two hdf5 files, except I want the data to stay resident in
memory. Is it possible to read the compressed bytes of a chunk into a byte
array?

For additional background, data is stored using the deflate filter. I
define my read dimensions to match my original chunking parameters to read in one exact chunk at a time.

Thanks for the help.

Jebb

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--

Daniel Kahn

Science Systems and Applications Inc.

301-867-2162<tel:301-867-2162>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org