I would like to ask for support for missing data, i.e. to have an array of logical size N_1, which contains only N_2 < N_1 items and the remaining array entries are masked, typically because the data is missing.
This would be helpful for at least two use cases:
Experimental measurements where data is missing, e.g. Electron backscatter diffraction (EBSD). Ping @mike.jackson
Modular simulation tools, where output exists only on part of the domain, e.g. when locally a refined model is used.
NumPys masked array (Masked arrays — NumPy v1.26 Manual) offers related functionality, but the masked data is actually there and can be accessed, so storing a masked array can be done in HDF5 with two arrays (one with values and one with Booleans), potentially lumped together as a compound type. My proposal differs with respect to that in the way that only the existing data is stored to reduce the memory footprint. So the idea could be also seen as some way of storing sparse matrices (see Sparse matrix - Wikipedia).
The advantageous over hand-written solutions would be that HDF5 could use the most appropriate data structure/layout in the backend.
Revise the filter plugin framework for handling using a dynamically loaded filter with a static library. Currently, the HDF5 library has an error stack that is not obvious when this happens, so we get a lot of questions about what is happening. From a discussion with Jordan,
The last I looked at it, my understanding is that the fact that HDF5 plugins are linked in a shared fashion against HDF5 means that use of them with static builds causes two copies of the library to exist in memory, one static and one shared. Since the plugins use public symbols from HDF5 that have global state, such as H5I IDs, this causes issues where the plugins try to use memory from an uninitialized (shared) HDF5, while the IDs they’re trying to lookup exist in the memory space of the static library.
…
I believe most plugin frameworks don’t have this issue because they either don’t expose symbols used by the library a plugin is linking to or expose symbols that have constant values that don’t depend on a library being initialized to look up a value.
…
I’ll have to prototype it, but the ID lookups and the global state are the main issues with this not working correctly. If a plugin could call back into the library to resolve an ID, most problems would be fixed.
from the C-interface macros remove the references to H5CHECK and H5OPEN and require that users explicitly open the interface as in Fortran.
Add a mechanism so that applications can avoid closing the interface inappropriately. For example, developers writing parts of larger application should be able to open and close the interface in their respective parts without stepping on each other.
Would it be possible to open the library to more use cases than file writing? I have HDF5 as a serialization library in mind. Writing/reading to/from memory is possible but the library was not really designed for that use case.
What about adding a HDF5 schema part to the library? This would help to separate content and schema, the later could be transferred separately, especially useful for the serialization use case.
I am thinking out loud and might be asking for HDF6…
