HDF5 using my own libz (zlib)

bbrashers · February 17, 2020, 8:01pm

I am trying to use a newer version of zlib (which I compiled myself) with hdf5-1.8.20, using pgfortran. I’ve done like this (in tcsh):

setenv ZLIB /usr/local/src/zlib-1.2.11

setenv CPPFLAGS “-I$ZLIB/include”
setenv LDFLAGS “-L$ZLIB/lib”
setenv LD_LIBRARY_PATH $ZLIB/lib:$LD_LIBRARY_PATH # note: first in the list

./configure
–prefix=pwd
–enable-fortran
–with-zlib=/usr/local/zlib-1.2.11
>&! my.configure.out

I tried using LDFLAGS “-L$ZLIB/lib -lz” like I would have expected, but had to instead use sed after configure:

After compilation, it still appears that the executables will use the system’s (gcc) version of libz:

ldd bin/h5dump | grep libz

    libz.so.1 => /usr/lib/gcc/x86_64-redhat-linux/4.8.5/../../../../lib64/libz.so.1 (0x00007f41f6c9f000)

What am I doing wrong? /usr/local/src/zlib-1.2.11/lib is the first element in $LD_LIBRARY_PATH in my normal environment.

Bart

bbrashers · February 28, 2020, 4:32pm

I have just tried again with hdf5-1.10.6, still no luck. Here’s the only mentions of -lz in the Makefile:

FCLIBS = -L/usr/local/src/zlib-1.2.11/lib -lz -L/usr/local/pgi/linux86-64-nollvm/19.10/lib -L/usr/lib64 -L/usr/li
b/gcc/x86_64-redhat-linux/4.8.5 -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5/…/…/…/…/lib64 -lpgf90rtl -lpgf90 -lpg
f90_rpm1 -lpgf902 -lpgftnrtl -lpgmp -lnuma -lpthread -lpgmath -lnspgc -lpgc -lrt -lm

LDFLAGS = -L/usr/local/src/zlib-1.2.11/lib -lz

LIBS = -lz -ldl -lm

(Note that I edited the Makefile to move the -lz from after -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 to after -L/usr/local/src/zlib-1.2.11/lib.)

hdf5-1.10.6 already has a bin/ directory, so I could not use --prefix=pwd in the call to configure, I had to use --prefix=pwd/build.

After compilation, it still looks like it’s using the system’s version of zlib:

ldd build/bin/h5copy | eg libz
libz.so.1 => /usr/lib/gcc/x86_64-redhat-linux/4.8.5/…/…/…/…/lib64/libz.so.1 (0x00007f93b434f000)

Does anyone have any hints for me?

Bart

gheber · February 28, 2020, 6:26pm

The output of ldd depends on the shell environment. Assuming Bash, what if you export LD_LIBRARY_PATH=/usr/local/src/zlib-1.2.11/lib:$LD_LIBRARY_PATH and then run ldd? G.

daniel.kahn · February 28, 2020, 9:37pm

Hi Bart,

I see that in your original thread on the topic you already tried the LD_LIBRARY_PATH method (in tcsh) without success. In a case like this I would resort to strace. It can give you some hints as to the actual search which ld is performing when you run the code. For example:

strace -f /home/dkahn/HDF5LinkTest/bin/h5copy | & grep open | grep libz

In my test case it shows both the failures and success of the search (ld always find the library in the last place it looks:wink:):

    ahn: ~/src/hdf5-1.10.6 % setenv LD_LIBRARY_PATH /home/dkahn/lib
    ahn: ~/src/hdf5-1.10.6 % strace -f /home/dkahn/HDF5LinkTest/bin/h5copy | & grep open | grep libz
    open("/home/dkahn/HDF5LinkTest/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/home/dkahn/lib/tls/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/home/dkahn/lib/tls/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    open("/home/dkahn/lib/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    **open("/home/dkahn/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = 3**

If I change the search path to something non-nonsensical we’ll see that ld finds the system installation of the library:

ahn: ~/src/hdf5-1.10.6 % setenv LD_LIBRARY_PATH /home/dkahn/lutfisk
ahn: ~/src/hdf5-1.10.6 % strace -f /home/dkahn/HDF5LinkTest/bin/h5copy | & grep open | grep libz
open("/home/dkahn/HDF5LinkTest/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/tls/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/tls/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3

During compilation the linker (at least in the static linking case, so this might not be useful to you) is sensitive to the order in which paths and libraries appear on the command line. The variables names alone (LDFLAGS, LIBS, etc) don’t give the whole picture. You can build the code with:

make V=1

and that will print the full command line to the screen (in the case of HDF5 its a lot of output) so you can see the order in which the linker processing arguments. Sometimes this is useful for identifying library problems.

Good luck.

bbrashers · February 28, 2020, 7:37pm

Thanks, this would apply at runtime too, not just in the call to ldd. I believe my memory leak is coming from my program using /usr/lib64/libz.so.1 instead of /usr/local/src/zlib-1.2.11/lib.

I have $LD_LIBRARY_PATH set in /etc/profile.d/zlib.csh. Here’s the values (with a little tr magic to make it easier to read). You can see that my local self-compiled zlib is early in the list.

echo $LD_LIBRARY_PATH | tr : “\n”

/opt/ohpc/pub/compiler/gcc/7.3.0/lib64

/usr/local/src/zlib-1.2.11/lib

/usr/local/src/hdf5-1.8.20.pgi/lib

/usr/local/src/libpng-1.6.37/lib

/usr/local/src/netcdf-c-4.7.2.pgi/build/lib

/usr/local/pgi/linux86-64/2019/lib

/usr/local/pgi/linux86-64/2019/libso

/usr/local/src/szip-2.1.1/lib

I also have added it like so:

cat /etc/ld.so.conf

include ld.so.conf.d/*.conf

cat /etc/ld.so.conf.d/zlib.conf

/usr/local/src/zlib-1.2.11/lib

sudo ldconfig -v | grep -e libz -e zlib

/usr/local/src/zlib-1.2.11/lib:

libz.so.1 -> libz.so.1.2.11

libzip.so.2 -> libzip.so.2.1.0

libz.so.1 -> libz.so.1.2.7

I believe the fundamental problem is that /usr/lib64/libz.so.1 is the one it’s finding first, and using. /usr/lib64 is not in /etc/csh* or /etc/bash*. It’s not in /etc/ld.so.conf.d:

grep /usr/lib /etc/ld.so.conf.d/*

/etc/ld.so.conf.d/atlas-x86_64.conf:/usr/lib64/atlas

/etc/ld.so.conf.d/dyninst-x86_64.conf:/usr/lib64/dyninst

/etc/ld.so.conf.d/llvm5.0-x86_64.conf:/usr/lib64/llvm5.0/lib

/etc/ld.so.conf.d/llvm-x86_64.conf:/usr/lib64/llvm

/etc/ld.so.conf.d/mariadb-x86_64.conf:/usr/lib64/mysql

/etc/ld.so.conf.d/R-x86_64.conf:/usr/lib64/R/lib

/usr/lib64 is one of the default ‘trusted’ locations, according to the ldconfig man page. But I would think it sane to add user-specified paths before the system-trusted defaults. I can’t figure out how to change that.

Any of this making sense to you?

Bart

bbrashers · March 4, 2020, 1:23am

OK, I recompiled using make V=1 and re-directed to a file. Here’s my current compile script:

cat my.configure

#!/bin/csh -f

make distclean >& /dev/null

setenv FC pgfortran

setenv CC pgcc

setenv CXX pgc++

setenv FFLAGS “-tp=istanbul”

setenv FCFLAGS “-tp=istanbul”

setenv CFLAGS “-tp=istanbul”

setenv ZLIB /usr/local/src/zlib-1.2.11

setenv CPPFLAGS “-I$ZLIB/include”

setenv LDFLAGS “-L$ZLIB/lib”

if !($?LD_LIBRARY_PATH) then

setenv LD_LIBRARY_PATH $ZLIB/lib

else

if ( “$LD_LIBRARY_PATH” !~ $ZLIB/lib ) then

setenv LD_LIBRARY_PATH $ZLIB/lib:${LD_LIBRARY_PATH}

endif

echo $LD_LIBRARY_PATH | tr : “\n”

./configure \

–prefix=pwd/build \

–enable-fortran \

–with-zlib=$ZLIB \

&! my.configure.out

sed -i “s|-L/usr/local/src/zlib-1.2.11/lib|-L/usr/local/src/zlib-1.2.11/lib -lz|g” Makefile

sed -i “s|-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5 -lz|-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5|g” Makefile

make V=1 >&! make.out

make install >&! make.install.out

Yes, lots of output to make.out, a bit hard to parse, but I do see lots of -L/usr/local/src/zlib-1.2.11/lib -lz strings. Some redundancies, like

/bin/sh …/libtool --tag=CC --mode=link pgcc -c99 -Minform=inform -fast -s -tp=istanbul -L/usr/local/src/zlib-1.2.11/lib -L/usr/local/src/zlib-1.2.11/lib -o H5detect H5detect.o -lz -ldl -lm

Calls to ldd still show it using the /usr/lib64 version of zlib. As does strace:

strace -f build/bin/h5copy | & grep open | grep libz

open("/usr/local/src/hdf5-1.10.6.pgi/build/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/local/pgi/linux86-64-nollvm/19.10/lib/tls/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/local/pgi/linux86-64-nollvm/19.10/lib/tls/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/local/pgi/linux86-64-nollvm/19.10/lib/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/local/pgi/linux86-64-nollvm/19.10/lib/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/lib/gcc/x86_64-redhat-linux/4.8.5/…/…/…/…/lib64/tls/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/lib/gcc/x86_64-redhat-linux/4.8.5/…/…/…/…/lib64/tls/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/lib/gcc/x86_64-redhat-linux/4.8.5/…/…/…/…/lib64/x86_64/libz.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

open("/usr/lib/gcc/x86_64-redhat-linux/4.8.5/…/…/…/…/lib64/libz.so.1", O_RDONLY|O_CLOEXEC) = 3

Any further ideas? I really appreciate the help.

Bart

daniel.kahn · March 5, 2020, 1:49pm

The results of the strace indicate that the paths in LD_LIBRARY_PATH are being ignored. (There is small possibility that strace is truncating its output such that libz doesn’t appear. You can rerun strace adding the -v option to prevent the truncation.)

The only thing know that would cause this is having the setuid or setgid bits set on the executable, namely h5copy in your test case. By the way I’m assuming that the h5copy in your Build directory is truly the Linux binary–it seems that the HDF5 “make build” process produces an h5copy wrapped in a sh script, and it isn’t until you “make install” that a true binary shows up. You can check this with the file command.

Observe the following examples in which the dynamic linker searches for libc

~/src/hdf5-1.10.6 % echo $LD_LIBRARY_PATH
/home/dkahn/lutfisk

First a program without setuid bit set, we check the permissions for dc

~/src/hdf5-1.10.6 % ls -l /usr/bin/dc
-rwxr-xr-x. 1 root root 45392 Jun 9 2014 /usr/bin/dc

and run using strace, notice it includes the futile search of our LD_LIBRARY_PATH directory path.

~/src/hdf5-1.10.6 % strace -f /usr/bin/dc | & grep libc
open("/home/dkahn/lutfisk/tls/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/tls/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/x86_64/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/home/dkahn/lutfisk/libc.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

Now lets try /usr/bin/crontab, note the ‘s’ in permissions indicates setuid bit set:

~/src/hdf5-1.10.6 % ls -l /usr/bin/crontab
-rwsr-xr-x. 1 root root 57656 Aug 8 2019 /usr/bin/crontab

And the we see that LD_LIBRARY_PATH is not searched.

~/src/hdf5-1.10.6 % strace -f crontab -l | & grep libc
open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
open("/lib64/libcap-ng.so.0", O_RDONLY|O_CLOEXEC) = 3

This last behavior reminds me of what you reported.

My understanding of this is that the restriction kicks in if the setuid or setgid bits are set and you running as a user id or group id other than owner or group id of the file. This is a Unix security measure to prevent someone from substituting a system library when running a privileged code. If you inadvertently installed h5copy with these bits sets that could explain what you are seeing.

bbrashers · March 5, 2020, 3:48pm

Yes, the build/bin directory contains the executable, placed there by make install.

file h5copy

h5copy: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, stripped

It does not appear to me that the setuid or setgid bits are set:

\ls -lF h5copy

-rwxr-xr-x. 1 bbrashers users 219696 Mar 3 16:45 h5copy*

(cd …/ ; ls -dlF bin)

drwxrwxr-x. 2 bbrashers 4.0K Mar 3 16:45 bin/

I thought I was on to something when I remembered that CentOS7 assumes 0 < uid < 1000 are reserved for “special” accounts (it changed from 500 to 1000 with CentOS 7) and I had imported my account across many generations of CentOS machines as I upgraded. So I tested using another user’s account:

sudo su –

su – someuser

cd /usr/local/src/hdf5-1.10.6.pgi/build/bin/

echo $uid $gid

17887 100

echo $LD_LIBRARY_PATH

/usr/local/src/zlib-1.2.11/lib:/usr/local/src/szip-2.1.1/lib:/usr/local/pgi/linux86-64/2019/libso:/usr/local/pgi/linux86-64/2019/lib:/usr/local/src/netcdf-c-4.7.2.pgi/build/lib:/opt/ohpc/pub/compiler/gcc/7.3.0/lib64:/usr/local/src/libpng-1.6.37/lib:/usr/local/src/hdf5-1.10.6.pgi/lib

[wclyn-cluster1 build/bin]# strace -fv h5copy | & grep open | grep libz