I had always understood (perhaps incorrectly!) that mixing statically and dynamically linked versions of the same library in the same process was a bad idea. Don't you end up with two copies of whatever globals those libraries use? I had speculated that this was why I was having problems using the LZF plugin with JHDF5 on Windows. It seemed to be loading the plugin but then crashing, rather than failing to load it at all. The same plugin worked fine with a simple C++ test program.
Thanks for looking into it.
Dan.
···
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: 13 December 2015 03:30
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Problem with a filter as shared library in Java-HDF5 and HDF5View
Hi Ümit,
We investigated little-bit more. Here is a high-level description of what we found.
The lzf filter uses calls to the HDF5 library for the callback functions.
When HDFView tries to read compressed data, it calls JNI (shared) that calls HDF5 (statically linked with JNI); the HDF5 library then issues dlopen to find and load the lzf filter shared library. When the filter is linked with HDF5, dlopen pulls in the filter library and dependents (i.e., HDF5, etc.), and HDFView works. When the filter is not linked with HDF5, dlopen pulls in the filter library but cannot find the required HDF5 library since there is no dependency. The behavior makes sense.
What puzzles us is that we can build h5dump that is statically linked with HDF5, and the lzf filter (that is not linked with HDF5) still works. Apparently having intermediate JNI makes a difference. May be someone on the FORUM could explain such behavior?
We recommend to link with the HD5 library when a filter uses HDF5 calls. We will look into how to enhance HDF5 error reporting to incorporate the status of dlopen call. We will also add such filter to our internal testing.
Thank you!
Elena
On Nov 25, 2015, at 9:20 AM, Ümit Seren <uemit.seren@gmail.com<mailto:uemit.seren@gmail.com>> wrote:
Hi Elena,
Thanks for confirming.
I remeber that it used to work in one of the older Java-HDF5 versions. Unfortunately I can't remember which one.
On Wed, Nov 25, 2015 at 4:14 PM, Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>> wrote:
Hi Ümit,
Allen and I confirmed the problem you and Dan see with HDFView. For some reason HDFView cannot find the lzf plugin while it finds, for example, bzip2 plugin (see https://svn.hdfgroup.uiuc.edu/hdf5_plugins/trunk/BZIP2/). The only difference is that lzf plugin uses HDF5 calls in the callback functions. We are puzzled why the HDF5 library is able to find the lzf plugin for the command-line tools and fails for HDFView.
I opened the ticket (JAVA-1920) and we will investigate.
Thanks for reporting the problem!
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org<http://hdfgroup.org/>
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Nov 23, 2015, at 5:00 PM, Ümit Seren <uemit.seren@gmail.com<mailto:uemit.seren@gmail.com>> wrote:
Hi Elena,
Thanks for the response,
my problem is not that h5dump or other hdf5 command line utilities do not work the lzf plugin filter (they work just fine) but the Java HDF5 and HDFView can't find the plugin_filter.
I will check the symbols tomorrow but as far as I can tell the problem is somewhere with the java hdf5 library.
On Mon, Nov 23, 2015 at 11:41 PM, Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>> wrote:
Hello,
What kind of error are you getting? I was able to get the filter to work on Linux (CentOS 7) with h5dump from 1.8.16 distribution I built. Please notice that one has to specify HDF5 include files when building liblzf_filter.so.
Details:
Here is the output of the nm command and h5dump when include files are not specified:
[epourmal@platypus lzf]$ gcc -O2 -fPIC -shared lzf/*.c lzf_filter.c -o liblzf_filter.so
epourmal@platypus lzf]$ nm liblzf_filter.so
U H5E_CALLBACK_g
U H5E_CANTREGISTER_g
U H5E_PLINE_g
U H5Epush1
U H5Pget_chunk
U H5Pget_filter_by_id2
U H5Pmodify_filter
U H5Tget_size
U H5Zregister
U H5open
[epourmal@platypus lzf]$ hdf5/hdf5/bin/h5dump test_lzf.hdf5
HDF5 "test_lzf.hdf5" {
GROUP "/" {
DATASET "dset" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 100, 100, 100 ) / ( 100, 100, 100 ) }
DATA {h5dump error: unable to print data
}
}
}
}
[epourmal@platypus lzf]$ hdf5/hdf5/bin/h5dump -pH test_lzf.hdf5
HDF5 "test_lzf.hdf5" {
GROUP "/" {
DATASET "dset" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 100, 100, 100 ) / ( 100, 100, 100 ) }
STORAGE_LAYOUT {
CHUNKED ( 1, 100, 100 )
SIZE 174288 (22.951:1 COMPRESSION)
}
FILTERS {
PREPROCESSING SHUFFLE
USER_DEFINED_FILTER {
FILTER_ID 32000
COMMENT lzf
PARAMS { 4 261 40000 }
}
...
Now if I specify HDF5 include files, I will get (notice two symbols H5PLget_plugin_info/type in the nm output!)
[epourmal@platypus lzf]$ gcc -I//scr/epourmal/h5py-master/lzf/hdf5/hdf5/include -O2 -fPIC -shared lzf/*.c lzf_filter.c -o liblzf_filter.so
[epourmal@platypus lzf]$ nm liblzf_filter.so
U H5E_CALLBACK_g
U H5E_CANTREGISTER_g
U H5E_PLINE_g
U H5Epush1
0000000000001080 T H5PLget_plugin_info
0000000000001070 T H5PLget_plugin_type
U H5Pget_chunk
U H5Pget_filter_by_id2
U H5Pmodify_filter
U H5Tget_size
U H5Zregister
U H5open
And here we are:
[epourmal@platypus lzf]$ hdf5/hdf5/bin/h5dump test_lzf.hdf5 |more
HDF5 "test_lzf.hdf5" {
GROUP "/" {
DATASET "dset" {
DATATYPE H5T_IEEE_F32LE
DATASPACE SIMPLE { ( 100, 100, 100 ) / ( 100, 100, 100 ) }
DATA {
(0,0,0): 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
(0,0,18): 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32,
(0,0,33): 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
(0,0,48): 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62,
(0,0,63): 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77,
(0,0,78): 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92,
(0,0,93): 93, 94, 95, 96, 97, 98, 99,
.......
Hope it will help!
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org<http://hdfgroup.org/>
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Nov 23, 2015, at 8:50 AM, Daniel Tetlow <daniel.tetlow@optasense.com<mailto:daniel.tetlow@optasense.com>> wrote:
Hi,
I had a similar problem on Windows. I needed to link the LZF plugin against the hdf5 library (as expected) to get it to compile, and it worked fine with C++, but it wouldn't work with the Java H5 libraries. Attempting to link it against the jhdf5 2.11 library instead failed. I couldn't figure out what I was doing wrong, so in the end I rebuilt jhdf5 with the plugin sources built in.
Cheers,
Dan
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Ümit Seren
Sent: 23 November 2015 14:10
To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Problem with a filter as shared library in Java-HDF5 and HDF5View
I have been trying to compile the lzf compressor as a shared library (not linked against hdf5) to be used as a filter plugin for HDF5.
Basically I was following this: https://github.com/h5py/h5py/tree/master/lzf
I compiled lzf as a shared library using this compile command:
gcc -O2 -fPIC -shared lzf/*.c lzf_filter.c -o liblzf_filter.so
This will create a lzf filter plugin that I copied into the /usr/local/hdf5/lib/plugin folder.
The normal command line tools (h5ls, h5dump) work fine with lzf compressed dataset, so the plugin seems to work.
However when I try to read the same lzf compressed dataset using Java HDF5 or HDFView (2.11), I get the error that the lzf filter plugin can not be found.
The only workaround is to link it against the hdf5 library:
gcc -O2 -fPIC -shared lzf/*.c lzf_filter.c -lhdf5 -o liblzf_filter.so
Is this a known issue ?
This was on Ubuntu 14.04 and HDF5 1.8.15patch1
thanks in advance
cheers
Ümit
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
