I’d like to build libhdf5.a, but include a few plugins, such as the bitshuffle filter, so you don’t need to dynamically load these plugins (which is a hassle since it mixes static/dynamic linking).
Is this possible? You would have to combine both codebases, but it’s both C, so in theory I don’t see any major hurdles. But I wouldn’t know where to register the plugin so it’s there from the start.
It is likely doable, but HDF Group has not really explored this build configuration so there’s nothing to start from.
I wouldn’t know where to register the plugin
Can you clarify what you mean? What plugin?
Not sure about the nomenclature here, but from what I gathered, in order to have your HDF5 data be “processed” with your own code, you write the processing code and then make the HDF5 library aware of it, also assigning it an ID.
With regular dynamically loaded plugins, this happens during discovery of the .dll/.so file. But I would have to do this more directly.
Check out HDF5 Filter Plugins. Skip the plugin part for the dynamic loading portion. Just call H5Z[un]register in your application, and you should be fine.
OK? G.
If I understand you correctly, you can register a plugin by following the instruction on the Registered HDF5 Filter Plugins page.
We are on the right track here, but it’s still missing the point. @gheber, you’re saying I should register the plugin in the application. I would like to have the plugin “pre-registered” in the library somehow, so that I can just use it in my applications and not worry about forgetting to register something, or setting a HDF5_PLUGIN_PATH variable or copying .so files and whatnot. Just like with certain filters that are built-in to HDF5. At least how I understand it, deflate doesn’t need an external plugin?
More understandable now? I’ve tried looking for how deflate is integrated into libhdf5, but I got lost.
I thought you had your own filter code, and you just compile and link that all together.
I think what you are asking is how to bundle all the pieces together. Right?
I don’t think we have a CMake target for that at the moment. But here’s what you can do:
mkdir -p build/merge
cd build/merge
# throw all the archives and objects you wanna bundle in here
ar -x ../libhdf5.a
ar -x ../libz.a
ar -x ../libmy_lib.a
ar -qc liball.a *.o
ranlib liball.a
liball.a is your proverbial kitchensink.
OK? G.
Ahhh, just (static) linking to a plugin registers it (because of some initialization happening automatically)?
No, not quite. I think there is a bit of terminological confusion going on here.
When the HDF5 library is compiled it contains a minimal set of filters (e.g., N-bit, shuffle, etc.) These filters are “baked into the library.” You can configure the inclusion of additional filters, such a Gzip or SZip compression. However, the binaries for these filters are typically separate from the HDF5 library binary. You would still link your application against libhdf5.[a,so] and libz.[a,so], etc. Yes, these, known at compile-time, filters, would be available to you without further initialization on your part. I wouldn’t call them plugins, though.
The term ‘plugin’ is usually applied to a shared binary object *.[so,dll] that can be loaded dynamically(= at runtime) by an application process. That way you can use a filter with an HDF5 library (provided your execution policy allows the dynamic loading of shared objects) that was build without knowledge of the specifics of that particular filter. All that matters is that it abides by the filter API (i.e., implements the right callbacks, etc.).
OK? But, I’m afraid, I’m still not sure I understand what you are trying to achieve.
Best, G.
Thank you for the clarification! Using this terminology: What I’m trying to achieve is then transforming a plugin (that contains a filter) into a baked-in filter that needs no loading at runtime.
Got it.
filter.[c,h] files. Build a static library (*.a) or just an object file (*.o). You will need that library or object file to link your application. Optional: Build a catch-all library archive as described earlier (liball.a).256-511, for testing only. (Distributing HDF5 files with filter dependencies in that range is asking for trouble!)filter.h header should contain a definition for a struct H5Z_class_t or the callbacks that you would use to populate one. Include that header in your application, and call H5Zregister, passing a pointer to your initialized instance of a struct H5Z_class_t. (Check the return value to ensure that the registration was successful.)H5Pset_filter.H5Zunregister when the filter is no longer needed in the application and to free the associated resources.OK?
G.