Efficient reading of specific pixels in large HDF file

Hdf-forum archives
1

Hello,
I need to read certain pixels from an HDF file (can be HDF4 or HDF5, we have both) using the FORTRAN interfaces.

The files are quite large (around 40 datasets, each ~44000x2200 pixels) and I wish to only read a specific number of these pixels. The pixels I wish to read are reasonably contiguous (a swatch of satellite data) but are not rectangular or any other regular shape, so I can't just extract a rectangular hyperslab.
I guess it is possible to read pixel-by-pixel until I have all the data I need, but that probably isn't very efficient. Any ideas for the best way to do this?

Cheers,
Simon

2

You can combine/append hyperslabs to select complex regions if that helps, see 1.4.1.1 https://www.hdfgroup.org/HDF5/doc/UG/12_Dataspaces.html, and just have one read call. Otherwise, you can use point selection, see 1.4.1.2, using one read call. In general, if the size of your hyperslabs are “reasonably large” (i.e. you don’t have the number of hyperslabs approaching the number of points you want to read in) and your data is reasonably contiguous then you will likely have better performance with hyperslabs than with point selection.

See examples for adding to current selected regions at (i.e. the use of H5S_SELECT_OR_F):

https://www.physics.ohio-state.edu/~wilkins/computing/HDF/hdf5tutorial/select.html

You definitely don’t want to do a read call for each element.

Scot

···

On Oct 19, 2015, at 9:12 AM, Simon Proud <simon.proud@physics.ox.ac.uk<mailto:simon.proud@physics.ox.ac.uk>> wrote:

Hello,
I need to read certain pixels from an HDF file (can be HDF4 or HDF5, we have both) using the FORTRAN interfaces.

The files are quite large (around 40 datasets, each ~44000x2200 pixels) and I wish to only read a specific number of these pixels. The pixels I wish to read are reasonably contiguous (a swatch of satellite data) but are not rectangular or any other regular shape, so I can't just extract a rectangular hyperslab.
I guess it is possible to read pixel-by-pixel until I have all the data I need, but that probably isn't very efficient. Any ideas for the best way to do this?

Cheers,
Simon
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5