What happens when chunked dataset read when chunk size is larger than chunk buffer?
I find it programmatically complex to inspect the dataset chunking properties when reading the data into my application. (as opposed to more straightforward known dimensions and chunking choices when writing a dataset ) Fortunately the API does not require any knowledge of the chunking (chunked or not, chunk rank, chunk size, chunk filter, etc.) to do reads. Yay.
I’m looking to improve performance of a 3-4 TB read where all data is read only once and then the file is closed. I’m experimenting with chunking and zlib compression again. I read that the default chunk size is 1MB and yet I think I have reason to try chunk sizes in the ~100MB range. (this is the general size of data that is contiguous in a given MPI PE’s memory space)
Is that chunk size bigger than default chunk buffer size going to be a performance problem for the read and decompress filter?
Only one H5Dread call is made, if that helps. I just have the code read the entire process-local slab at once.