We’ve been developing a compression filter and have discovered that tools such as h5dump and libraries such as h5py fail to load the filter, while our own C++ tools that read/write datasets using the filter work fine. However, explicitly linking the filter with libpthread allows h5dump to load it, and additionally linking it with libhdf5 allows h5py to load it.
Our C++ tools are already linked with these libraries which would explain why they have no trouble loading it.
I see that someone came across a similar issue before and it was recommended that filters that use HDF5 calls should link with the library (https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015-December/009100.html). I’m not sure I understand why this is necessary, since any code that is capable of opening an HDF5 file would already link with libhdf5.
If this is normal, perhaps it’s worth noting in the filter documentation, especially since implementing a set_local callback would almost certainly involve calling some HDF5 functions (we use it to determine type size, etc.)
I may have a silly question because I am Java-naive but isn't the problem
that the libhdf5 gets statically linked into the JNI? Is there any way to
make libhdf5 dynamically linked into the JNI? If so, then I think a later
dlopen of a libhdf5-dependent shared library (e.g. a compressor) would
find and use the libhdf5 already loaded when the JNI was loaded.
We¹ve been developing a compression filter and have discovered that tools
such as h5dump and libraries such as h5py fail to load the filter, while
our own C++ tools that read/write datasets using the filter work fine.
However, explicitly linking the filter with libpthread allows h5dump to
load it, and additionally linking it with libhdf5 allows h5py to load it.
Our C++ tools are already linked with these libraries which would explain
why they have no trouble loading it.
I see that someone came across a similar issue before and it was
recommended that filters that use HDF5 calls should link with the library
(https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015-De
cember/009100.html). I¹m not sure I understand why this is necessary,
since any code that is capable of opening an HDF5 file would already link
with libhdf5.
If this is normal, perhaps it¹s worth noting in the filter documentation,
especially since implementing a set_local callback would almost certainly
involve calling some HDF5 functions (we use it to determine type size,
etc.)
Wouldn't you want the libhdf5 to dynamically load filters itself, so it doesn't have to be rebuilt for any one specific filter. I would make the analogy to video codecs where you can have multiple and add them as needed.
Of course that creates a whole new security loophole that has to be controlled. I'm not sure how much of an issue that would be though.
Scott
···
-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: Wednesday, May 18, 2016 12:53 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Shared library problems with plugin filters
I may have a silly question because I am Java-naive but isn't the problem that the libhdf5 gets statically linked into the JNI? Is there any way to make libhdf5 dynamically linked into the JNI? If so, then I think a later dlopen of a libhdf5-dependent shared library (e.g. a compressor) would find and use the libhdf5 already loaded when the JNI was loaded.
On 5/18/16, 9:26 AM, "Hdf-forum on behalf of Derick Swanepoel"
<hdf-forum-bounces@lists.hdfgroup.org on behalf of dswanepoel@gmail.com>
wrote:
Good day,
We¹ve been developing a compression filter and have discovered that
tools such as h5dump and libraries such as h5py fail to load the
filter, while our own C++ tools that read/write datasets using the filter work fine.
However, explicitly linking the filter with libpthread allows h5dump to
load it, and additionally linking it with libhdf5 allows h5py to load it.
Our C++ tools are already linked with these libraries which would
explain why they have no trouble loading it.
I see that someone came across a similar issue before and it was
recommended that filters that use HDF5 calls should link with the
library
(https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015
-De cember/009100.html). I¹m not sure I understand why this is
necessary, since any code that is capable of opening an HDF5 file would
already link with libhdf5.
If this is normal, perhaps it¹s worth noting in the filter
documentation, especially since implementing a set_local callback would
almost certainly involve calling some HDF5 functions (we use it to
determine type size,
etc.)
Well, I am C++ naïve but this is also the conclusion I came to when I tried using custom filters with jhdf5 on Windows a while back.
I could compile the filter linked with the hd5f.lib import library to produce a filter plugin dll. This worked just fine as expected from a C++ test program that found the plugin on the HDF5_PLUGIN_PATH. When I tried to use the same plugin from a Java program using jhdf5.dll, the filter loaded but H5 API calls made by the plugin failed. I (naively) guessed this was because I had ended up with both statically and dynamically linked versions of hdf5. So I then tried to link the filter plugin using the jhdf5.lib import library instead, but that wouldn't link.
Dan
···
-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Miller, Mark C.
Sent: 18 May 2016 17:53
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Shared library problems with plugin filters
I may have a silly question because I am Java-naive but isn't the problem that the libhdf5 gets statically linked into the JNI? Is there any way to make libhdf5 dynamically linked into the JNI? If so, then I think a later dlopen of a libhdf5-dependent shared library (e.g. a compressor) would find and use the libhdf5 already loaded when the JNI was loaded.
On 5/18/16, 9:26 AM, "Hdf-forum on behalf of Derick Swanepoel"
<hdf-forum-bounces@lists.hdfgroup.org on behalf of dswanepoel@gmail.com>
wrote:
Good day,
We¹ve been developing a compression filter and have discovered that
tools such as h5dump and libraries such as h5py fail to load the
filter, while our own C++ tools that read/write datasets using the filter work fine.
However, explicitly linking the filter with libpthread allows h5dump to
load it, and additionally linking it with libhdf5 allows h5py to load it.
Our C++ tools are already linked with these libraries which would
explain why they have no trouble loading it.
I see that someone came across a similar issue before and it was
recommended that filters that use HDF5 calls should link with the
library
(https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015
-De cember/009100.html). I¹m not sure I understand why this is
necessary, since any code that is capable of opening an HDF5 file would
already link with libhdf5.
If this is normal, perhaps it¹s worth noting in the filter
documentation, especially since implementing a set_local callback would
almost certainly involve calling some HDF5 functions (we use it to
determine type size,
etc.)
The problem is in the H5PL__open() function in H5PL.c where it tries to load the plugin using the H5PL_OPEN_DLIB macro (dlopen() on Linux). This fails when our plugin is not linked with libhdf5, and adding a call to dlerror() to obtain the error message gives:
libhdf5 itself is loading this shared library and unable to resolve a symbol within itself, which is strange. Looking again at H5PL_OPEN_DLIB, it expands to dlopen(libname, RTLD_NOW). RTLD_NOW requires that all symbols used by the shared library be resolved at that point and produces the above error if they can’t. I don’t know enough about the dynamic linker to understand why that’s the case here, but changing it to RTLD_LAZY fixes the problem (RTLD_LAZY performs lazy binding).
Bottom line: I suggest that plugins be loaded using RTLD_LAZY so that plugins that use libhdf5 functions don’t need to link with it.
I’m not sure if this would solve the similar Java issues, or what the equivalent on Windows would be.
We’ve been developing a compression filter and have discovered that tools such as h5dump and libraries such as h5py fail to load the filter, while our own C++ tools that read/write datasets using the filter work fine. However, explicitly linking the filter with libpthread allows h5dump to load it, and additionally linking it with libhdf5 allows h5py to load it.
Our C++ tools are already linked with these libraries which would explain why they have no trouble loading it.
I see that someone came across a similar issue before and it was recommended that filters that use HDF5 calls should link with the library (https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015-December/009100.html). I’m not sure I understand why this is necessary, since any code that is capable of opening an HDF5 file would already link with libhdf5.
If this is normal, perhaps it’s worth noting in the filter documentation, especially since implementing a set_local callback would almost certainly involve calling some HDF5 functions (we use it to determine type size, etc.)
Thank you for your insights! We will consider the suggested solution.
Elena
···
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The problem is in the H5PL__open() function in H5PL.c where it tries to load the plugin using the H5PL_OPEN_DLIB macro (dlopen() on Linux). This fails when our plugin is not linked with libhdf5, and adding a call to dlerror() to obtain the error message gives:
libhdf5 itself is loading this shared library and unable to resolve a symbol within itself, which is strange. Looking again at H5PL_OPEN_DLIB, it expands to dlopen(libname, RTLD_NOW). RTLD_NOW requires that all symbols used by the shared library be resolved at that point and produces the above error if they can’t. I don’t know enough about the dynamic linker to understand why that’s the case here, but changing it to RTLD_LAZY fixes the problem (RTLD_LAZY performs lazy binding).
Bottom line: I suggest that plugins be loaded using RTLD_LAZY so that plugins that use libhdf5 functions don’t need to link with it.
I’m not sure if this would solve the similar Java issues, or what the equivalent on Windows would be.
We’ve been developing a compression filter and have discovered that tools such as h5dump and libraries such as h5py fail to load the filter, while our own C++ tools that read/write datasets using the filter work fine. However, explicitly linking the filter with libpthread allows h5dump to load it, and additionally linking it with libhdf5 allows h5py to load it.
Our C++ tools are already linked with these libraries which would explain why they have no trouble loading it.
I see that someone came across a similar issue before and it was recommended that filters that use HDF5 calls should link with the library (https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2015-December/009100.html). I’m not sure I understand why this is necessary, since any code that is capable of opening an HDF5 file would already link with libhdf5.
If this is normal, perhaps it’s worth noting in the filter documentation, especially since implementing a set_local callback would almost certainly involve calling some HDF5 functions (we use it to determine type size, etc.)
