Trouble Linking Parallel HDF5 Libraries with MPI

Hello,

I am trying to update a numerical model (RAMS) so that it has distributive memory capabilities when run in parallel. I would like to do this using PHDF5. Currently, RAMS uses MPI for the parallel computing, but it assembles the grids on a single node before writing to HDF5 in serial. I have re-built both MPI and HDF5 locally for this upgrade. Here is my problem:

While compiling RAMS with the new parallel HDF5 libraries, the loader stage complains that within the HDF5 libraries it has "undefined reference to" MANY MPI routines. If I look in the HDF5 .a libraries (using nm), I see the references. I have tried adding MPI libraries to the loader using -lxxx, but that does not seem to solve my problem. Because of this, I have tried many different configurations of the libraries. My latest configurations are:

MPI
./configure --prefix=/home/rseigel/mpich2-1.5-install --enable-romio --enable-mpe 2>&1 | tee c.txt

HDF5
./configure --prefix=/home/rseigel/hdf5-1.8.10 --enable-fortran --enable-parallel --enable-shared=no CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CC=/home/rseigel/mpich2-1.5-install/bin/mpicc

I have checked all of the paths in my environment and the makefile for RAMS numerous times. Everything is okay from that standpoint. I am not sure what to do next, as I do not know why there are undefined references.

Thanks!
Rob

Here are the loader errors:

/usr/local/pgi/linux86-64/8.0-4/bin/pgf90 -o rams-6.0-opt rammain.o -v -Wl,-zmuldefs -Wl, -ldl -lc -lgcc_eh -lpthread -lsz rams-6.0-opt.a \
        ./utils/libutils-2.4-opt.a -L./utils -lncarg-2.4 -L/home/rseigel/mpich2-1.5-install/lib -lmpich -lmpl -lmpichf90 -ltmpe -lampe -lfmpich -llmpe -lmpe -lmpe_collchk -lmpe_f2cmpi -lmpe_nompi -lmpe_nompi_null -lmpe_null -lmpichcxx -lopa -L/usr/lib64 -L/home/rseigel/hdf5-1.8.10/lib -lhdf5 -L/usr/lib64 -lsz -lz -lm

/usr/bin/ld /usr/lib64/crt1.o /usr/lib64/crti.o /usr/local/pgi/linux86-64/8.0-4/lib/trace_init.o /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtbegin.o /usr/local/pgi/linux86-64/8.0-4/lib/f90main.o -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/local/pgi/linux86-64/8.0-4/lib/pgi.ld -L./utils -L/home/rseigel/mpich2-1.5-install/lib -L/usr/lib64 -L/home/rseigel/hdf5-1.8.10/lib -L/usr/lib64 -L/usr/local/pgi/linux86-64/8.0-4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 rammain.o -zmuldefs -ldl -lc -lgcc_eh -lpthread -lsz rams-6.0-opt.a ./utils/libutils-2.4-opt.a -lncarg-2.4 -lmpich -lmpl -lmpichf90 -ltmpe -lampe -lfmpich -llmpe -lmpe -lmpe_collchk -lmpe_f2cmpi -lmpe_nompi -lmpe_nompi_null -lmpe_null -lmpichcxx -lopa -lhdf5 -lsz -lz -lm -rpath /usr/local/pgi/linux86-64/8.0-4/lib -o rams-6.0-opt -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtend.o /usr/lib64/crtn.o
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o): In function `H5AC_propagate_flushed_and_still_clean_entries_list':
H5AC.c:(.text+0xf03): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0xfdd): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0x10aa): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0x12ea): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o): In function `H5AC_rsp__p0_only__flush_to_min_clean':
H5AC.c:(.text+0x15a2): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o):H5AC.c:(.text+0x177c): more undefined references to `MPI_Error_string' follow
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Eint.o): In function `H5E_walk2_cb':
H5Eint.c:(.text+0xa52): undefined reference to `MPI_Initialized'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Eint.o): In function `H5E_walk1_cb':
H5Eint.c:(.text+0xcc7): undefined reference to `MPI_Initialized'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpi.o): In function `H5FD_mpi_comm_info_free':
H5FDmpi.c:(.text+0x273): undefined reference to `MPI_Info_free'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpi.o): In function `H5FD_mpi_comm_info_dup':
H5FDmpi.c:(.text+0x337): undefined reference to `MPI_Info_dup'
H5FDmpi.c:(.text+0x3c8): undefined reference to `MPI_Info_free'
H5FDmpi.c:(.text+0x3ed): undefined reference to `MPI_Error_string'
H5FDmpi.c:(.text+0x460): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_get_mpio_atomicity':
H5FDmpio.c:(.text+0xf2): undefined reference to `MPI_File_get_atomicity'
H5FDmpio.c:(.text+0x12b): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_set_mpio_atomicity':
H5FDmpio.c:(.text+0x1c3): undefined reference to `MPI_File_set_atomicity'
H5FDmpio.c:(.text+0x1e1): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_truncate':
H5FDmpio.c:(.text+0x29c): undefined reference to `MPI_File_set_size'
H5FDmpio.c:(.text+0x2d3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x35e): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_flush':
H5FDmpio.c:(.text+0x405): undefined reference to `MPI_File_sync'
H5FDmpio.c:(.text+0x41a): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_write':
H5FDmpio.c:(.text+0x541): undefined reference to `MPI_File_write_at'
H5FDmpio.c:(.text+0x55d): undefined reference to `MPI_Get_elements'
H5FDmpio.c:(.text+0x573): undefined reference to `MPI_Type_size'
H5FDmpio.c:(.text+0x63c): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0x681): undefined reference to `MPI_File_write_at_all'
H5FDmpio.c:(.text+0x6a8): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0x6c1): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x773): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x81c): undefined reference to `MPI_File_write_at'
H5FDmpio.c:(.text+0x835): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x8ad): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x920): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x9e7): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xaae): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_read':
H5FDmpio.c:(.text+0xbc1): undefined reference to `MPI_File_read_at'
H5FDmpio.c:(.text+0xbdd): undefined reference to `MPI_Get_elements'
H5FDmpio.c:(.text+0xbf3): undefined reference to `MPI_Type_size'
H5FDmpio.c:(.text+0xd1d): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0xd64): undefined reference to `MPI_File_read_at_all'
H5FDmpio.c:(.text+0xd8d): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0xda6): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xe47): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xeca): undefined reference to `MPI_File_read_at'
H5FDmpio.c:(.text+0xee3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xf5d): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xfd0): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x1097): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x115e): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_close':
H5FDmpio.c:(.text+0x11dd): undefined reference to `MPI_File_close'
H5FDmpio.c:(.text+0x120f): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_open':
H5FDmpio.c:(.text+0x1ecf): undefined reference to `MPI_File_open'
H5FDmpio.c:(.text+0x1ee8): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x1f82): undefined reference to `MPI_Info_free'
H5FDmpio.c:(.text+0x1fc3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x203d): undefined reference to `MPI_File_close'
H5FDmpio.c:(.text+0x216d): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2231): undefined reference to `MPI_File_get_size'
H5FDmpio.c:(.text+0x2253): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x22d9): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2347): undefined reference to `MPI_File_set_size'
H5FDmpio.c:(.text+0x2360): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2442): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5Pget_fapl_mpio':
H5FDmpio.c:(.text+0x2909): undefined reference to `MPI_Info_dup'
H5FDmpio.c:(.text+0x29aa): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2a1d): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_truncate':
H5FDmpiposix.c:(.text+0x1f8): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_fapl_get':
H5FDmpiposix.c:(.text+0x9c3): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_fapl_copy':
H5FDmpiposix.c:(.text+0xac2): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o):H5FDmpiposix.c:(.text+0xe29): more undefined references to `MPI_Error_string' follow
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o): In function `H5S_obtain_datatype':
H5Smpio.c:(.text+0x462): undefined reference to `MPI_Type_hindexed'
H5Smpio.c:(.text+0x47b): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x52a): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x59d): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x744): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x7b7): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o): In function `H5S_mpio_space_type':
H5Smpio.c:(.text+0xecd): undefined reference to `MPI_Type_extent'
H5Smpio.c:(.text+0x10a2): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1128): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1202): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1275): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1302): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o):H5Smpio.c:(.text+0x13a7): more undefined references to `MPI_Error_string' follow
pgf90-Fatal-linker completed with exit code 1

make: *** [rams-6.0-opt] Error 2

···

____________________________
Robert Seigel, Ph.D.
Colorado State University
Department of Atmospheric Science
1371 Campus Delivery
Fort Collins, CO 80523
(970) 491-8331
rseigel@atmos.colostate.edu

Hi,

I would suggest to check the order of the libraries on the linking line. HDF5 should probably be the first, then all MPI libraries, then all compression libraries. You may also try to use the h5pfc script (found under the bin directory of the HDF5 install directory) to compile and link your application.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Dec 11, 2012, at 11:41 AM, Robert Seigel wrote:

Hello,

I am trying to update a numerical model (RAMS) so that it has distributive memory capabilities when run in parallel. I would like to do this using PHDF5. Currently, RAMS uses MPI for the parallel computing, but it assembles the grids on a single node before writing to HDF5 in serial. I have re-built both MPI and HDF5 locally for this upgrade. Here is my problem:

While compiling RAMS with the new parallel HDF5 libraries, the loader stage complains that within the HDF5 libraries it has "undefined reference to" MANY MPI routines. If I look in the HDF5 .a libraries (using nm), I see the references. I have tried adding MPI libraries to the loader using -lxxx, but that does not seem to solve my problem. Because of this, I have tried many different configurations of the libraries. My latest configurations are:

MPI
./configure --prefix=/home/rseigel/mpich2-1.5-install --enable-romio --enable-mpe 2>&1 | tee c.txt

HDF5
./configure --prefix=/home/rseigel/hdf5-1.8.10 --enable-fortran --enable-parallel --enable-shared=no CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CC=/home/rseigel/mpich2-1.5-install/bin/mpicc

I have checked all of the paths in my environment and the makefile for RAMS numerous times. Everything is okay from that standpoint. I am not sure what to do next, as I do not know why there are undefined references.

Thanks!
Rob

Here are the loader errors:

/usr/local/pgi/linux86-64/8.0-4/bin/pgf90 -o rams-6.0-opt rammain.o -v -Wl,-zmuldefs -Wl, -ldl -lc -lgcc_eh -lpthread -lsz rams-6.0-opt.a \
        ./utils/libutils-2.4-opt.a -L./utils -lncarg-2.4 -L/home/rseigel/mpich2-1.5-install/lib -lmpich -lmpl -lmpichf90 -ltmpe -lampe -lfmpich -llmpe -lmpe -lmpe_collchk -lmpe_f2cmpi -lmpe_nompi -lmpe_nompi_null -lmpe_null -lmpichcxx -lopa -L/usr/lib64 -L/home/rseigel/hdf5-1.8.10/lib -lhdf5 -L/usr/lib64 -lsz -lz -lm

/usr/bin/ld /usr/lib64/crt1.o /usr/lib64/crti.o /usr/local/pgi/linux86-64/8.0-4/lib/trace_init.o /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtbegin.o /usr/local/pgi/linux86-64/8.0-4/lib/f90main.o -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/local/pgi/linux86-64/8.0-4/lib/pgi.ld -L./utils -L/home/rseigel/mpich2-1.5-install/lib -L/usr/lib64 -L/home/rseigel/hdf5-1.8.10/lib -L/usr/lib64 -L/usr/local/pgi/linux86-64/8.0-4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 rammain.o -zmuldefs -ldl -lc -lgcc_eh -lpthread -lsz rams-6.0-opt.a ./utils/libutils-2.4-opt.a -lncarg-2.4 -lmpich -lmpl -lmpichf90 -ltmpe -lampe -lfmpich -llmpe -lmpe -lmpe_collchk -lmpe_f2cmpi -lmpe_nompi -lmpe_nompi_null -lmpe_null -lmpichcxx -lopa -lhdf5 -lsz -lz -lm -rpath /usr/local/pgi/linux86-64/8.0-4/lib -o rams-6.0-opt -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtend.o /usr/lib64/crtn.o
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o): In function `H5AC_propagate_flushed_and_still_clean_entries_list':
H5AC.c:(.text+0xf03): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0xfdd): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0x10aa): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0x12ea): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o): In function `H5AC_rsp__p0_only__flush_to_min_clean':
H5AC.c:(.text+0x15a2): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o):H5AC.c:(.text+0x177c): more undefined references to `MPI_Error_string' follow
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Eint.o): In function `H5E_walk2_cb':
H5Eint.c:(.text+0xa52): undefined reference to `MPI_Initialized'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Eint.o): In function `H5E_walk1_cb':
H5Eint.c:(.text+0xcc7): undefined reference to `MPI_Initialized'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpi.o): In function `H5FD_mpi_comm_info_free':
H5FDmpi.c:(.text+0x273): undefined reference to `MPI_Info_free'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpi.o): In function `H5FD_mpi_comm_info_dup':
H5FDmpi.c:(.text+0x337): undefined reference to `MPI_Info_dup'
H5FDmpi.c:(.text+0x3c8): undefined reference to `MPI_Info_free'
H5FDmpi.c:(.text+0x3ed): undefined reference to `MPI_Error_string'
H5FDmpi.c:(.text+0x460): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_get_mpio_atomicity':
H5FDmpio.c:(.text+0xf2): undefined reference to `MPI_File_get_atomicity'
H5FDmpio.c:(.text+0x12b): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_set_mpio_atomicity':
H5FDmpio.c:(.text+0x1c3): undefined reference to `MPI_File_set_atomicity'
H5FDmpio.c:(.text+0x1e1): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_truncate':
H5FDmpio.c:(.text+0x29c): undefined reference to `MPI_File_set_size'
H5FDmpio.c:(.text+0x2d3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x35e): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_flush':
H5FDmpio.c:(.text+0x405): undefined reference to `MPI_File_sync'
H5FDmpio.c:(.text+0x41a): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_write':
H5FDmpio.c:(.text+0x541): undefined reference to `MPI_File_write_at'
H5FDmpio.c:(.text+0x55d): undefined reference to `MPI_Get_elements'
H5FDmpio.c:(.text+0x573): undefined reference to `MPI_Type_size'
H5FDmpio.c:(.text+0x63c): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0x681): undefined reference to `MPI_File_write_at_all'
H5FDmpio.c:(.text+0x6a8): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0x6c1): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x773): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x81c): undefined reference to `MPI_File_write_at'
H5FDmpio.c:(.text+0x835): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x8ad): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x920): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x9e7): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xaae): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_read':
H5FDmpio.c:(.text+0xbc1): undefined reference to `MPI_File_read_at'
H5FDmpio.c:(.text+0xbdd): undefined reference to `MPI_Get_elements'
H5FDmpio.c:(.text+0xbf3): undefined reference to `MPI_Type_size'
H5FDmpio.c:(.text+0xd1d): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0xd64): undefined reference to `MPI_File_read_at_all'
H5FDmpio.c:(.text+0xd8d): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0xda6): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xe47): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xeca): undefined reference to `MPI_File_read_at'
H5FDmpio.c:(.text+0xee3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xf5d): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xfd0): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x1097): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x115e): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_close':
H5FDmpio.c:(.text+0x11dd): undefined reference to `MPI_File_close'
H5FDmpio.c:(.text+0x120f): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_open':
H5FDmpio.c:(.text+0x1ecf): undefined reference to `MPI_File_open'
H5FDmpio.c:(.text+0x1ee8): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x1f82): undefined reference to `MPI_Info_free'
H5FDmpio.c:(.text+0x1fc3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x203d): undefined reference to `MPI_File_close'
H5FDmpio.c:(.text+0x216d): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2231): undefined reference to `MPI_File_get_size'
H5FDmpio.c:(.text+0x2253): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x22d9): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2347): undefined reference to `MPI_File_set_size'
H5FDmpio.c:(.text+0x2360): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2442): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5Pget_fapl_mpio':
H5FDmpio.c:(.text+0x2909): undefined reference to `MPI_Info_dup'
H5FDmpio.c:(.text+0x29aa): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2a1d): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_truncate':
H5FDmpiposix.c:(.text+0x1f8): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_fapl_get':
H5FDmpiposix.c:(.text+0x9c3): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_fapl_copy':
H5FDmpiposix.c:(.text+0xac2): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o):H5FDmpiposix.c:(.text+0xe29): more undefined references to `MPI_Error_string' follow
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o): In function `H5S_obtain_datatype':
H5Smpio.c:(.text+0x462): undefined reference to `MPI_Type_hindexed'
H5Smpio.c:(.text+0x47b): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x52a): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x59d): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x744): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x7b7): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o): In function `H5S_mpio_space_type':
H5Smpio.c:(.text+0xecd): undefined reference to `MPI_Type_extent'
H5Smpio.c:(.text+0x10a2): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1128): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1202): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1275): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1302): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o):H5Smpio.c:(.text+0x13a7): more undefined references to `MPI_Error_string' follow
pgf90-Fatal-linker completed with exit code 1

make: *** [rams-6.0-opt] Error 2

____________________________
Robert Seigel, Ph.D.
Colorado State University
Department of Atmospheric Science
1371 Campus Delivery
Fort Collins, CO 80523
(970) 491-8331
rseigel@atmos.colostate.edu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Thank you Elena - this worked!

Rob

···

On Dec 11, 2012, at 8:40 PM, Elena Pourmal wrote:

Hi,

I would suggest to check the order of the libraries on the linking line. HDF5 should probably be the first, then all MPI libraries, then all compression libraries. You may also try to use the h5pfc script (found under the bin directory of the HDF5 install directory) to compile and link your application.

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Dec 11, 2012, at 11:41 AM, Robert Seigel wrote:

Hello,

I am trying to update a numerical model (RAMS) so that it has distributive memory capabilities when run in parallel. I would like to do this using PHDF5. Currently, RAMS uses MPI for the parallel computing, but it assembles the grids on a single node before writing to HDF5 in serial. I have re-built both MPI and HDF5 locally for this upgrade. Here is my problem:

While compiling RAMS with the new parallel HDF5 libraries, the loader stage complains that within the HDF5 libraries it has "undefined reference to" MANY MPI routines. If I look in the HDF5 .a libraries (using nm), I see the references. I have tried adding MPI libraries to the loader using -lxxx, but that does not seem to solve my problem. Because of this, I have tried many different configurations of the libraries. My latest configurations are:

MPI
./configure --prefix=/home/rseigel/mpich2-1.5-install --enable-romio --enable-mpe 2>&1 | tee c.txt

HDF5
./configure --prefix=/home/rseigel/hdf5-1.8.10 --enable-fortran --enable-parallel --enable-shared=no CFLAGS="-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64" CC=/home/rseigel/mpich2-1.5-install/bin/mpicc

I have checked all of the paths in my environment and the makefile for RAMS numerous times. Everything is okay from that standpoint. I am not sure what to do next, as I do not know why there are undefined references.

Thanks!
Rob

Here are the loader errors:

/usr/local/pgi/linux86-64/8.0-4/bin/pgf90 -o rams-6.0-opt rammain.o -v -Wl,-zmuldefs -Wl, -ldl -lc -lgcc_eh -lpthread -lsz rams-6.0-opt.a \
        ./utils/libutils-2.4-opt.a -L./utils -lncarg-2.4 -L/home/rseigel/mpich2-1.5-install/lib -lmpich -lmpl -lmpichf90 -ltmpe -lampe -lfmpich -llmpe -lmpe -lmpe_collchk -lmpe_f2cmpi -lmpe_nompi -lmpe_nompi_null -lmpe_null -lmpichcxx -lopa -L/usr/lib64 -L/home/rseigel/hdf5-1.8.10/lib -lhdf5 -L/usr/lib64 -lsz -lz -lm

/usr/bin/ld /usr/lib64/crt1.o /usr/lib64/crti.o /usr/local/pgi/linux86-64/8.0-4/lib/trace_init.o /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtbegin.o /usr/local/pgi/linux86-64/8.0-4/lib/f90main.o -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/local/pgi/linux86-64/8.0-4/lib/pgi.ld -L./utils -L/home/rseigel/mpich2-1.5-install/lib -L/usr/lib64 -L/home/rseigel/hdf5-1.8.10/lib -L/usr/lib64 -L/usr/local/pgi/linux86-64/8.0-4/lib -L/usr/lib64 -L/usr/lib/gcc/x86_64-redhat-linux/4.1.2 rammain.o -zmuldefs -ldl -lc -lgcc_eh -lpthread -lsz rams-6.0-opt.a ./utils/libutils-2.4-opt.a -lncarg-2.4 -lmpich -lmpl -lmpichf90 -ltmpe -lampe -lfmpich -llmpe -lmpe -lmpe_collchk -lmpe_f2cmpi -lmpe_nompi -lmpe_nompi_null -lmpe_null -lmpichcxx -lopa -lhdf5 -lsz -lz -lm -rpath /usr/local/pgi/linux86-64/8.0-4/lib -o rams-6.0-opt -lpgf90 -lpgf90_rpm1 -lpgf902 -lpgf90rtl -lpgftnrtl -lnspgc -lpgc -lrt -lpthread -lm -lgcc -lc -lgcc /usr/lib/gcc/x86_64-redhat-linux/4.1.2/crtend.o /usr/lib64/crtn.o
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o): In function `H5AC_propagate_flushed_and_still_clean_entries_list':
H5AC.c:(.text+0xf03): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0xfdd): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0x10aa): undefined reference to `MPI_Error_string'
H5AC.c:(.text+0x12ea): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o): In function `H5AC_rsp__p0_only__flush_to_min_clean':
H5AC.c:(.text+0x15a2): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5AC.o):H5AC.c:(.text+0x177c): more undefined references to `MPI_Error_string' follow
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Eint.o): In function `H5E_walk2_cb':
H5Eint.c:(.text+0xa52): undefined reference to `MPI_Initialized'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Eint.o): In function `H5E_walk1_cb':
H5Eint.c:(.text+0xcc7): undefined reference to `MPI_Initialized'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpi.o): In function `H5FD_mpi_comm_info_free':
H5FDmpi.c:(.text+0x273): undefined reference to `MPI_Info_free'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpi.o): In function `H5FD_mpi_comm_info_dup':
H5FDmpi.c:(.text+0x337): undefined reference to `MPI_Info_dup'
H5FDmpi.c:(.text+0x3c8): undefined reference to `MPI_Info_free'
H5FDmpi.c:(.text+0x3ed): undefined reference to `MPI_Error_string'
H5FDmpi.c:(.text+0x460): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_get_mpio_atomicity':
H5FDmpio.c:(.text+0xf2): undefined reference to `MPI_File_get_atomicity'
H5FDmpio.c:(.text+0x12b): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_set_mpio_atomicity':
H5FDmpio.c:(.text+0x1c3): undefined reference to `MPI_File_set_atomicity'
H5FDmpio.c:(.text+0x1e1): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_truncate':
H5FDmpio.c:(.text+0x29c): undefined reference to `MPI_File_set_size'
H5FDmpio.c:(.text+0x2d3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x35e): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_flush':
H5FDmpio.c:(.text+0x405): undefined reference to `MPI_File_sync'
H5FDmpio.c:(.text+0x41a): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_write':
H5FDmpio.c:(.text+0x541): undefined reference to `MPI_File_write_at'
H5FDmpio.c:(.text+0x55d): undefined reference to `MPI_Get_elements'
H5FDmpio.c:(.text+0x573): undefined reference to `MPI_Type_size'
H5FDmpio.c:(.text+0x63c): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0x681): undefined reference to `MPI_File_write_at_all'
H5FDmpio.c:(.text+0x6a8): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0x6c1): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x773): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x81c): undefined reference to `MPI_File_write_at'
H5FDmpio.c:(.text+0x835): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x8ad): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x920): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x9e7): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xaae): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_read':
H5FDmpio.c:(.text+0xbc1): undefined reference to `MPI_File_read_at'
H5FDmpio.c:(.text+0xbdd): undefined reference to `MPI_Get_elements'
H5FDmpio.c:(.text+0xbf3): undefined reference to `MPI_Type_size'
H5FDmpio.c:(.text+0xd1d): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0xd64): undefined reference to `MPI_File_read_at_all'
H5FDmpio.c:(.text+0xd8d): undefined reference to `MPI_File_set_view'
H5FDmpio.c:(.text+0xda6): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xe47): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xeca): undefined reference to `MPI_File_read_at'
H5FDmpio.c:(.text+0xee3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xf5d): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0xfd0): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x1097): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x115e): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_close':
H5FDmpio.c:(.text+0x11dd): undefined reference to `MPI_File_close'
H5FDmpio.c:(.text+0x120f): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5FD_mpio_open':
H5FDmpio.c:(.text+0x1ecf): undefined reference to `MPI_File_open'
H5FDmpio.c:(.text+0x1ee8): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x1f82): undefined reference to `MPI_Info_free'
H5FDmpio.c:(.text+0x1fc3): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x203d): undefined reference to `MPI_File_close'
H5FDmpio.c:(.text+0x216d): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2231): undefined reference to `MPI_File_get_size'
H5FDmpio.c:(.text+0x2253): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x22d9): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2347): undefined reference to `MPI_File_set_size'
H5FDmpio.c:(.text+0x2360): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2442): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpio.o): In function `H5Pget_fapl_mpio':
H5FDmpio.c:(.text+0x2909): undefined reference to `MPI_Info_dup'
H5FDmpio.c:(.text+0x29aa): undefined reference to `MPI_Error_string'
H5FDmpio.c:(.text+0x2a1d): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_truncate':
H5FDmpiposix.c:(.text+0x1f8): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_fapl_get':
H5FDmpiposix.c:(.text+0x9c3): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o): In function `H5FD_mpiposix_fapl_copy':
H5FDmpiposix.c:(.text+0xac2): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5FDmpiposix.o):H5FDmpiposix.c:(.text+0xe29): more undefined references to `MPI_Error_string' follow
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o): In function `H5S_obtain_datatype':
H5Smpio.c:(.text+0x462): undefined reference to `MPI_Type_hindexed'
H5Smpio.c:(.text+0x47b): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x52a): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x59d): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x744): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x7b7): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o): In function `H5S_mpio_space_type':
H5Smpio.c:(.text+0xecd): undefined reference to `MPI_Type_extent'
H5Smpio.c:(.text+0x10a2): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1128): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1202): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1275): undefined reference to `MPI_Error_string'
H5Smpio.c:(.text+0x1302): undefined reference to `MPI_Error_string'
/home/rseigel/hdf5-1.8.10/lib/libhdf5.a(H5Smpio.o):H5Smpio.c:(.text+0x13a7): more undefined references to `MPI_Error_string' follow
pgf90-Fatal-linker completed with exit code 1

make: *** [rams-6.0-opt] Error 2

____________________________
Robert Seigel, Ph.D.
Colorado State University
Department of Atmospheric Science
1371 Campus Delivery
Fort Collins, CO 80523
(970) 491-8331
rseigel@atmos.colostate.edu

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org