While experimenting with h5repack (HDF5 1.8.22, Windows 10) to apply GZIP compression to single channel Z-stack HDF5 images, I have noticed some unconsistent compression speed (and file size threshold). Basically, repacking a 128 MB file takes less than a second while repacking a 256 MB image takes some 45s. Do you have any idea what might the cause (and ideally the remedy)?
Note: I am using HDF5 1.8.22 because I plan to use a compiled custom filter that is only compatible with HDF5 1.8.
How are you chunking your images?
G.
Here are two dumps following your request, the first one if for the file that gets converted by h5repack in < 1s, the second for the larger file that gets converted in a significantly longer time (it is actually only around 25s for a file only twice as big). The conversion time is not linear at all and quickly gets unpractical for my application (image files of several hundreds of GB).
The only way I managed to keep conversion time linear is to break down the dataset into several channels inside the HDF5 file but it is not ideal and I would like to understand if it is possible to configure h5repack to handle larger files more efficiently.
HDF5 “TestMedium.h5” {
SUPER_BLOCK {
SUPERBLOCK_VERSION 0
FREELIST_VERSION 0
SYMBOLTABLE_VERSION 0
OBJECTHEADER_VERSION 0
OFFSET_SIZE 8
LENGTH_SIZE 8
BTREE_RANK 16
BTREE_LEAF 4
ISTORE_K 32
USER_BLOCK {
USERBLOCK_SIZE 0
}
}
GROUP “/” {
GROUP “DATA_TYPES” {
DATATYPE “Enum_Boolean” H5T_ENUM {
H5T_STD_I8LE;
“FALSE” 0;
“TRUE” 1;
};
DATATYPE “String_VariableLength” H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
};
}
GROUP “t0” {
DATASET “channel0” {
DATATYPE H5T_STD_U16LE
DATASPACE SIMPLE { ( 128, 738, 762 ) / ( H5S_UNLIMITED, H5S_UNLIMITED, H5S_UNLIMITED ) }
STORAGE_LAYOUT {
CHUNKED ( 128, 738, 762 )
SIZE 143963136
}
FILTERS {
NONE
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
ATTRIBUTE “element_size_um” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3 ) / ( 3 ) }
}
}
}
}
}
HDF5 “TestMedium2.h5” {
SUPER_BLOCK {
SUPERBLOCK_VERSION 0
FREELIST_VERSION 0
SYMBOLTABLE_VERSION 0
OBJECTHEADER_VERSION 0
OFFSET_SIZE 8
LENGTH_SIZE 8
BTREE_RANK 16
BTREE_LEAF 4
ISTORE_K 32
USER_BLOCK {
USERBLOCK_SIZE 0
}
}
GROUP “/” {
GROUP “DATA_TYPES” {
DATATYPE “Enum_Boolean” H5T_ENUM {
H5T_STD_I8LE;
“FALSE” 0;
“TRUE” 1;
};
DATATYPE “String_VariableLength” H5T_STRING {
STRSIZE H5T_VARIABLE;
STRPAD H5T_STR_NULLTERM;
CSET H5T_CSET_ASCII;
CTYPE H5T_C_S1;
};
}
GROUP “t0” {
DATASET “channel0” {
DATATYPE H5T_STD_U16LE
DATASPACE SIMPLE { ( 256, 738, 762 ) / ( H5S_UNLIMITED, H5S_UNLIMITED, H5S_UNLIMITED ) }
STORAGE_LAYOUT {
CHUNKED ( 256, 738, 762 )
SIZE 287926272
}
FILTERS {
NONE
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_ALLOC
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
ATTRIBUTE “element_size_um” {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 3 ) / ( 3 ) }
}
}
}
}
}
As you can see from the dumps, there is only one chunk per dataset. I have however tried to first repack the dataset to 64x64x64 chunk size (first step) and then apply compression (second step) and the speed gain for the second step was marginal. There seems to be a clear size threshold (around 128 MB) from which point performance is severely impacted, unless one subdivides the datasets in different groups (not ideal for the application). What parameters could I tweak in h5repack to hopefully observe a significant improvement? I don’t know how to set the chunk cache size from these parameters.
OK, I’m not understanding several things:
With h5repack you can repack any existing dataset. The exisiting layout might be contiguous, or chunked, etc. Rechunking a chunked dataset might be slow, if the old and new layout are very dissimilar (e.g., rank). In that case, the tool might have to read multiple chunks and use only a small portion toward the new chunk, etc.
(The default chunk cache size is 1 MB, which might add to the trouble.)
I don’t quite understand what you mean by first and second steps. With h5repack you specify the new chunk size and the compression on the same command line. There is usually no need to do this in separate steps.
Depending on the data, your chunk size of (128, 738, 762) is may a little on the high side. Is the data mostly constant or zero? If not, a smaller chunk size (~1 MB, e.g., your 64^3) might yeild better results. What compression level are you using? I don’t see any compression ratios in your h5dump output.
You also seem to suggest that you are dealing with stacks of images. Which of the three dimensions is the stack dimension? Does the resolution of your images vary? If not, why did you make the width and height H5S_UNLIMITED? Do you think you can get better compression by having three-dimensional chunks? If not, you should consider a { 1, width_chunk_size, height_chunk_size} or { stack_block_size, width, height} chunk layout.
Maybe you can give us a general description of your workflow? Why aren’t you compressing the images to begin with? You can also compress them outside HDF5 and then just write RAW data chunks straight into the file.
G.
