Dynamically Loaded Filters on Mac OS


#1

My desktop application uses HDF5 1.10.7 on Mac OS and Windows.

I succeeded in getting dynamically loaded filters (a.k.a., plugins) to work on Windows using hdf5plugin-1.10.7-win10_64-vs15.zip from https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.7/plugins/

I am not able to get them to work on Mac OS. I tried hdf5plugin-1.10.7-osx1013_64-clang.tar.gz and hdf5plugin-1.10.7-osx1013_64-intel.tar.gz. Both provided “.so” files (e.g., libh5blosc.so) but I believe on Macintosh I need .dylib files. I have not been able to find .dylib files.

I have installed the .so files in the right place ("/usr/local/hdf5/lib/plugin") but they are not found by the HDF5 library.

Even with hdf5plugin-1.10.7-osx1013_64-clang.tar, the README file talks about Linux and Windows but nothing about Mac. So I’m starting to think that dynamically loaded filters are not supported on Mac.

Thanks in advance for any enlightenment.


#2

I run on mac too (macOS 10.14) and I would not say that .dylib files are a requirement. That is the macOS native way of doing shared libs but I’ve used .so files for HDF5 plugins fine on macOS.

Have you used the HDF5_PLUGIN_PATH environment variable to inform HDF5 library where to go look for plugins? Also, if you set the HDF5_DEBUG environment variable (https://support.hdfgroup.org/HDF5/doc/H5.user/Debugging.html) to “all”, you might get some hints from the library about why its not finding your plugin shared libs.


#3

Thanks. This was useful.

My program has a command that dumps collected HDF5 library error messages. Using this I saw:

_While executing HDF5LoadData, H5Dread failed. (HDF5 Library Error -1)
_HDF5 library function H5PL__open(): can’t dlopen:dlopen(/usr/local/hdf5/lib/plugin/libh5blosc.so, 5): Library not loaded: @rpath/libhdf5.103.dylib
_ Referenced from: /usr/local/hdf5/lib/plugin/libh5blosc.so
_ Reason: image not found

From that I can tell that the Mac dynamic loader is trying to load the .so file and that the .so file is looking for a library named libhdf5.103.dylib.

My application ships with a library named libhdf5.dylib in the Frameworks folder of the application bundle. After about two hours of floundering and trial-and-error (with rpath and loader_path), I finally found the magic incantation:

install_name_tool -change “@rpath/libhdf5.103.dylib” “@executable_path/…/Frameworks/libhdf5.dylib” “/usr/local/hdf5/lib/plugin/libh5blosc.so”

This tells libh5blosc.so to look for hdf5lib.dylib in the Frameworks folder of the bundle of the currently-running application. And it worked!

I did this for all of the .so files in /usr/local/hdf5/lib/plugin using this command:

for file in /usr/local/hdf5/lib/plugin/*.so; do install_name_tool -change “@rpath/libhdf5.103.dylib” “@executable_path/…/Frameworks/libhdf5.dylib” “$file”; done

Thanks miller86!


#4

I verified that my solution works on Mac OS 10.14.6 (Mojave) and 12.0.1 (Monterey).


#5

To tie up one loose end…

On Unix a .so file is an ELF file.

I believe that Macintosh can not use Linux ELF files but instead uses Mach-O files.

As explained here:

Loadable modules are called “bundles” in Mach-O speak. They have the file type MH_BUNDLE. They can carry any extension; the extension .bundle is recommended by Apple, but most ported software uses .so for the sake of compatibility.

So a .so file that runs on Macintosh is a Mach-O file despite the fact that .so on Unix is an ELF file.

The command “otool -hv” reports the filetype of HDF5 dynamically loaded filter files (e.g., libh5blosc.so) as BUNDLE. This stands for MH_BUNDLE which is described in the Mach-O loader.h file as “dynamically bound bundle file”.

To make matters more confusing, this type of bundle is not the same as a CFBundle which is “is a file directory with a defined structure and file extension, allowing related files to be grouped together as a conceptually single item”.