# Shuffle filter for array data

**URL:** <https://forum.hdfgroup.org/t/shuffle-filter-for-array-data/2294>\
**Category:** Hdf-forum archives\
**Created:** [September 7, 2011, 12:46am UTC](https://forum.hdfgroup.org/t/shuffle-filter-for-array-data/2294 "2011-09-07T00:46:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Salnikov\_Andrei\_A](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/s/4da419/32.png) [@Salnikov\_Andrei\_A](https://forum.hdfgroup.org/u/Salnikov_Andrei_A)\
**Post date:** [September 7, 2011, 12:46am UTC](https://forum.hdfgroup.org/t/shuffle-filter-for-array-data/2294/1 "2011-09-07T00:46:01Z")

</div>

Hi all,

I was playing today with the different compression options  
for our data trying to get some optimal numbers. Our data  
are 16-bit images with dynamic range which is limited most  
of the time so I would expect that shuffle filter should  
get us some improvement compared to just using plain zlib  
compression. To my surprise enabling shuffle did not  
change compression factor at all. Looking at the shuffle  
filter code it seems that the reason for that is the structure  
of our data. The dataset which contains the images is a  
1-dimensional dataspace with each element containing another  
2- or 3-dimensional image stack:

DATASET "..." {  
&nbsp;&nbsp;&nbsp;DATATYPE H5T\_ARRAY { [32][185][388] H5T\_STD\_I16LE }  
&nbsp;&nbsp;&nbsp;DATASPACE SIMPLE { ( 2132 ) / ( H5S\_UNLIMITED ) }  
&nbsp;&nbsp;&nbsp;STORAGE\_LAYOUT {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CHUNKED ( 1 )  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SIZE 6608778714 (1.482:1 COMPRESSION)  
&nbsp;&nbsp;&nbsp;&nbsp;}  
&nbsp;&nbsp;&nbsp;FILTERS {  
&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;COMPRESSION DEFLATE { LEVEL 1 }  
&nbsp;&nbsp;&nbsp;}

The size of the image arrays is quite big so the chunks  
fit just one single array most of the time.

My understanding is that shuffle algorithm tries to  
re-order bytes form multiple objects in a chunk but because  
there is just one object (which is array) in this case  
it does not do anything at all. What I would like shuffle  
to do in this case is to shuffle 16-bit words from the array,  
not to treat the array as a single object but look inside it.

I did some experimenting with the code and with a small change  
to the code I managed to convince it to shuffle things correctly.  
The diff is below this message. It does indeed improve  
compression of the image data and the data can be read back  
correctly after de-shuffling with the standard code (h5dump  
shows identical results).

It would be really helpful for us if something like this could  
be added to HDF5 library. I do not particularly care about other  
types such as compounds, but for the datasets whose elements are  
plain arrays it can probably be done without breaking compatibility.  
OTOH if more options could be added to shuffle which control  
shuffling of arrays and other types of data it could become even  
more useful.

Thanks,  
Andy

> **···**
>
> ----------------------------------------------------------------------  
> This is the change applied to 1.8.6 code:
> 
> \*\*\* H5Zshuffle.c.orig 2011-02-14 08:23:19.000000000 -0800  
> --- H5Zshuffle.c 2011-09-06 17:18:13.022259993 -0700  
> \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
> \*\*\* 88,93 \*\*\*\*  
> --- 88,98 ----  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if(H5P\_get\_filter\_by\_id(dcpl\_plist, H5Z\_FILTER\_SHUFFLE, &flags, &cd\_nelmts, cd\_values, (size\_t)0, NULL, NULL) \< 0)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HGOTO\_ERROR(H5E\_PLINE, H5E\_CANTGET, FAIL, "can't get shuffle parameters")  
> &nbsp;&nbsp;  
> + /\* If object is an array use its base type \*/  
> + while (H5T\_get\_class(type, FALSE) == H5T\_ARRAY) {  
> + type = H5T\_get\_super(type);  
> + }  
> +  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;/\* Set "local" parameter for this dataset \*/  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if((cd\_values[H5Z\_SHUFFLE\_PARM\_SIZE] = (unsigned)H5T\_get\_size(type)) == 0)  
> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HGOTO\_ERROR(H5E\_PLINE, H5E\_BADTYPE, FAIL, "bad datatype size")

---

<div class="post-metadata">

**Author:** ![Quincey\_Koziol](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/q/dbc845/32.png) [@Quincey\_Koziol](https://forum.hdfgroup.org/u/Quincey_Koziol)\
**Post date:** [September 7, 2011, 2:12pm UTC](https://forum.hdfgroup.org/t/shuffle-filter-for-array-data/2294/2 "2011-09-07T14:12:13Z")

</div>

Hi Andrei,  
&nbsp;&nbsp;Sounds like a good suggestion to me, I'll put it into our issue tracker and we can get it scheduled for an upcoming release. (It might not make it into the 1.8.8 release in November though)

&nbsp;&nbsp;Thanks for the idea,  
&nbsp;&nbsp;&nbsp;&nbsp;Quincey

> **···**
>
> On Sep 6, 2011, at 7:46 PM, Salnikov, Andrei A. wrote:
> 
> > Hi all,
> > 
> > I was playing today with the different compression options  
> > for our data trying to get some optimal numbers. Our data  
> > are 16-bit images with dynamic range which is limited most  
> > of the time so I would expect that shuffle filter should  
> > get us some improvement compared to just using plain zlib  
> > compression. To my surprise enabling shuffle did not  
> > change compression factor at all. Looking at the shuffle  
> > filter code it seems that the reason for that is the structure  
> > of our data. The dataset which contains the images is a  
> > 1-dimensional dataspace with each element containing another  
> > 2- or 3-dimensional image stack:
> > 
> > DATASET "..." {  
> > &nbsp;&nbsp;DATATYPE H5T\_ARRAY { [32][185][388] H5T\_STD\_I16LE }  
> > &nbsp;&nbsp;DATASPACE SIMPLE { ( 2132 ) / ( H5S\_UNLIMITED ) }  
> > &nbsp;&nbsp;STORAGE\_LAYOUT {  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CHUNKED ( 1 )  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SIZE 6608778714 (1.482:1 COMPRESSION)  
> > &nbsp;&nbsp;&nbsp;}  
> > &nbsp;&nbsp;FILTERS {  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;COMPRESSION DEFLATE { LEVEL 1 }  
> > &nbsp;&nbsp;}
> > 
> > The size of the image arrays is quite big so the chunks  
> > fit just one single array most of the time.
> > 
> > My understanding is that shuffle algorithm tries to  
> > re-order bytes form multiple objects in a chunk but because  
> > there is just one object (which is array) in this case  
> > it does not do anything at all. What I would like shuffle  
> > to do in this case is to shuffle 16-bit words from the array,  
> > not to treat the array as a single object but look inside it.
> > 
> > I did some experimenting with the code and with a small change  
> > to the code I managed to convince it to shuffle things correctly.  
> > The diff is below this message. It does indeed improve  
> > compression of the image data and the data can be read back  
> > correctly after de-shuffling with the standard code (h5dump  
> > shows identical results).
> > 
> > It would be really helpful for us if something like this could  
> > be added to HDF5 library. I do not particularly care about other  
> > types such as compounds, but for the datasets whose elements are  
> > plain arrays it can probably be done without breaking compatibility.  
> > OTOH if more options could be added to shuffle which control  
> > shuffling of arrays and other types of data it could become even  
> > more useful.
> > 
> > Thanks,  
> > Andy
> > 
> > ----------------------------------------------------------------------  
> > This is the change applied to 1.8.6 code:
> > 
> > \*\*\* H5Zshuffle.c.orig 2011-02-14 08:23:19.000000000 -0800  
> > --- H5Zshuffle.c 2011-09-06 17:18:13.022259993 -0700  
> > \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
> > \*\*\* 88,93 \*\*\*\*  
> > --- 88,98 ----  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if(H5P\_get\_filter\_by\_id(dcpl\_plist, H5Z\_FILTER\_SHUFFLE, &flags, &cd\_nelmts, cd\_values, (size\_t)0, NULL, NULL) \< 0)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HGOTO\_ERROR(H5E\_PLINE, H5E\_CANTGET, FAIL, "can't get shuffle parameters")
> > 
> > + /\* If object is an array use its base type \*/  
> > + while (H5T\_get\_class(type, FALSE) == H5T\_ARRAY) {  
> > + type = H5T\_get\_super(type);  
> > + }  
> > +  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;/\* Set "local" parameter for this dataset \*/  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if((cd\_values[H5Z\_SHUFFLE\_PARM\_SIZE] = (unsigned)H5T\_get\_size(type)) == 0)  
> > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HGOTO\_ERROR(H5E\_PLINE, H5E\_BADTYPE, FAIL, "bad datatype size")
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > Hdf-forum is for HDF software users discussion.  
> > Hdf-forum@hdfgroup.org  
> > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)

---

<div class="post-metadata">

**Author:** ![Salnikov\_Andrei\_A](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/s/4da419/32.png) [@Salnikov\_Andrei\_A](https://forum.hdfgroup.org/u/Salnikov_Andrei_A)\
**Post date:** [September 7, 2011, 4:15pm UTC](https://forum.hdfgroup.org/t/shuffle-filter-for-array-data/2294/3 "2011-09-07T16:15:01Z")

</div>

Hi Quincey,

thank you so much, we really appreciate this.

Cheers,  
Andy

> **···**
>
> Quincey Koziol wrote on 2011-09-07:
> 
> > Hi Andrei,  
> > &nbsp;&nbsp;Sounds like a good suggestion to me, I'll put it into our issue  
> > tracker and we can get it scheduled for an upcoming release. (It might  
> > not make it into the 1.8.8 release in November though)
> > 
> > &nbsp;&nbsp;Thanks for the idea,  
> > &nbsp;&nbsp;&nbsp;&nbsp;Quincey
> > 
> > On Sep 6, 2011, at 7:46 PM, Salnikov, Andrei A. wrote:
> > 
> > > Hi all,
> > > 
> > > I was playing today with the different compression options  
> > > for our data trying to get some optimal numbers. Our data  
> > > are 16-bit images with dynamic range which is limited most  
> > > of the time so I would expect that shuffle filter should  
> > > get us some improvement compared to just using plain zlib  
> > > compression. To my surprise enabling shuffle did not  
> > > change compression factor at all. Looking at the shuffle  
> > > filter code it seems that the reason for that is the structure  
> > > of our data. The dataset which contains the images is a  
> > > 1-dimensional dataspace with each element containing another  
> > > 2- or 3-dimensional image stack:
> > > 
> > > DATASET "..." {  
> > > &nbsp;&nbsp;DATATYPE H5T\_ARRAY { [32][185][388] H5T\_STD\_I16LE }  
> > > &nbsp;&nbsp;DATASPACE SIMPLE { ( 2132 ) / ( H5S\_UNLIMITED ) }  
> > > &nbsp;&nbsp;STORAGE\_LAYOUT {  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;CHUNKED ( 1 )  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;SIZE 6608778714 (1.482:1 COMPRESSION)  
> > > &nbsp;&nbsp;&nbsp;}  
> > > &nbsp;&nbsp;FILTERS {  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;COMPRESSION DEFLATE { LEVEL 1 }  
> > > &nbsp;&nbsp;}  
> > > The size of the image arrays is quite big so the chunks  
> > > fit just one single array most of the time.
> > > 
> > > My understanding is that shuffle algorithm tries to  
> > > re-order bytes form multiple objects in a chunk but because  
> > > there is just one object (which is array) in this case  
> > > it does not do anything at all. What I would like shuffle  
> > > to do in this case is to shuffle 16-bit words from the array,  
> > > not to treat the array as a single object but look inside it.
> > > 
> > > I did some experimenting with the code and with a small change  
> > > to the code I managed to convince it to shuffle things correctly.  
> > > The diff is below this message. It does indeed improve  
> > > compression of the image data and the data can be read back  
> > > correctly after de-shuffling with the standard code (h5dump  
> > > shows identical results).
> > > 
> > > It would be really helpful for us if something like this could  
> > > be added to HDF5 library. I do not particularly care about other  
> > > types such as compounds, but for the datasets whose elements are  
> > > plain arrays it can probably be done without breaking compatibility.  
> > > OTOH if more options could be added to shuffle which control  
> > > shuffling of arrays and other types of data it could become even  
> > > more useful.
> > > 
> > > Thanks,  
> > > Andy
> > > 
> > > --------------------------------------------------------------------- -  
> > > This is the change applied to 1.8.6 code:
> > > 
> > > \*\*\* H5Zshuffle.c.orig 2011-02-14 08:23:19.000000000 -0800  
> > > --- H5Zshuffle.c 2011-09-06 17:18:13.022259993 -0700  
> > > \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*  
> > > \*\*\* 88,93 \*\*\*\*  
> > > --- 88,98 ----  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if(H5P\_get\_filter\_by\_id(dcpl\_plist, H5Z\_FILTER\_SHUFFLE, &flags,
> > 
> > &cd\_nelmts, cd\_values, (size\_t)0, NULL, NULL) \< 0)
> > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HGOTO\_ERROR(H5E\_PLINE, H5E\_CANTGET, FAIL, "can't get shuffle
> > 
> > parameters")
> > 
> > > + /\* If object is an array use its base type \*/  
> > > + while (H5T\_get\_class(type, FALSE) == H5T\_ARRAY) {  
> > > + type = H5T\_get\_super(type);  
> > > + }  
> > > +  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;/\* Set "local" parameter for this dataset \*/  
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if((cd\_values[H5Z\_SHUFFLE\_PARM\_SIZE] =
> > 
> > (unsigned)H5T\_get\_size(type)) == 0)
> > 
> > > &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;HGOTO\_ERROR(H5E\_PLINE, H5E\_BADTYPE, FAIL, "bad datatype size")
> > > 
> > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_  
> > > Hdf-forum is for HDF software users discussion.  
> > > Hdf-forum@hdfgroup.org  
> > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
> > 
> > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ Hdf-forum is for HDF  
> > software users discussion. Hdf-forum@hdfgroup.org  
> > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)
