Hello,
so I’m on a cluster and I want to get h5py working. I currently try to do so by loading the modules the cluster provides but in the end, I could always just compile everything myself but I would really like to not do so. Now I really think it should work but I’m simply doing something wrong.
The error I get is “‘h5py.h5p.PropFAID’ object has no attribute ‘set_fapl_mpio’”, the python code I run is the example provided here https://docs.h5py.org/en/stable/mpi.html
I’m aware of the issue of c++ and parallelism as described in https://portal.hdfgroup.org/pages/viewpage.action?pageId=48809672
Now currently I loaded the following LMOd (some module manager) modules:
- StdEnv 3) openmpi/4.0.2 5) openblas/0.2.20 7) libszip/2.1.1
- gcc/6.3.0 4) hdf5/1.10.1 6) python/3.7.4
Furthermore I can look up how hdf5/1.10.1 was compiled:
$ module show hdf5/1.10.1
------------------------------------------------------------------------------------------------------
/cluster/apps/lmodules/MPI/openmpi/4.0.2/gcc/6.3.0/hdf5/1.10.1.lua:
------------------------------------------------------------------------------------------------------
whatis("Name : hdf5")
whatis("Version : 1.10.1")
whatis("Short description : HDF5 is a data model, library, and file format for storing and managing data. It supports an unlimited variety of datatypes, and is designed for flexible and efficient I/O and for high volume and complex data. ")
whatis("Configure options : --enable-unsupported --disable-threadsafe --enable-cxx --enable-hl --enable-fortran --with-szlib=/cluster/apps/gcc-6.3.0/libszip-2.1.1-3vqkiqebuvl6dxxinxrx6l662u6zgvou --enable-build-mode=production --enable-shared CFLAGS=-fPIC CXXFLAGS=-fPIC FCFLAGS=-fPIC --enable-parallel CC=/cluster/apps/gcc-6.3.0/openmpi-4.0.2-4airvo32ypyuapzgi4fp2kjea5psqu3t/bin/mpicc CXX=/cluster/apps/gcc-6.3.0/openmpi-4.0.2-4airvo32ypyuapzgi4fp2kjea5psqu3t/bin/mpic++ FC=/cluster/apps/gcc-6.3.0/openmpi-4.0.2-4airvo32ypyuapzgi4fp2kjea5psqu3t/bin/mpif90 --with-zlib=/cluster/apps/gcc-6.3.0/zlib-1.2.11-bu2rglshnlxrwc24334r76jr34jm2fxy")
help([[HDF5 is a data model, library, and file format for storing and managing
data. It supports an unlimited variety of datatypes, and is designed for
flexible and efficient I/O and for high volume and complex data.]])
prepend_path("PATH","/cluster/apps/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u/bin")
prepend_path("LIBRARY_PATH","/cluster/apps/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u/lib")
prepend_path("LD_LIBRARY_PATH","/cluster/apps/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u/lib")
prepend_path("CPATH","/cluster/apps/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u/include")
prepend_path("CMAKE_PREFIX_PATH","/cluster/apps/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u/")
setenv("HDF5_ROOT","/cluster/apps/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u")
Note that the hdf5 module is a smart module, so if MPI is loaded it’ll load the module that is compiled with parallel support. h5py states that we need the --enable-parallel --enable-shared
flags, which we seem to use. So that’s fine, no?
Furthermore I have the following tools: h5pcc and h5c++ (no h5cc). No idea what the difference of h5pcc and h5cc is but using e.g. h5cc -showconfig I get
$ h5pcc -showconfig
SUMMARY OF THE HDF5 CONFIGURATION
=================================
General Information:
-------------------
HDF5 Version: 1.10.1
Configured on: Tue Feb 4 08:00:34 CET 2020
Configured by: spackapps@eu-c7-041-01
Host system: x86_64-unknown-linux-gnu
Uname information: Linux eu-c7-041-01 3.10.0-862.14.4.el7.x86_64 #1 SMP Wed Sep 26 15:12:11 UTC 2018 x86_64 GNU/Linux
Byte sex: little-endian
Installation point: /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/hdf5-1.10.1-crkd6anbl6gtrycr6cutnpmwcnqvkk5u
Compiling Options:
------------------
Build Mode: production
Debugging Symbols: no
Asserts: no
Profiling: no
Optimization Level: high
Linking Options:
----------------
Libraries: static, shared
Statically Linked Executables:
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS: -L/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/zlib-1.2.11-bu2rglshnlxrwc24334r76jr34jm2fxy/lib -L/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/libszip-2.1.1-3vqkiqebuvl6dxxinxrx6l662u6zgvou/lib
Extra libraries: -lsz -lz -ldl -lm
Archiver: ar
Ranlib: ranlib
Languages:
----------
C: yes
C Compiler: /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/openmpi-4.0.2-4airvo32ypyuapzgi4fp2kjea5psqu3t/bin/mpicc ( gcc (GCC) 6.3.0)
CPPFLAGS:
H5_CPPFLAGS: -D_GNU_SOURCE -D_POSIX_C_SOURCE=200112L -DNDEBUG -UH5_DEBUG_API
AM_CPPFLAGS: -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/zlib-1.2.11-bu2rglshnlxrwc24334r76jr34jm2fxy/include -I/cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/libszip-2.1.1-3vqkiqebuvl6dxxinxrx6l662u6zgvou/include
C Flags: -fPIC
H5 C Flags: -std=c99 -pedantic -Wall -Wextra -Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations -Wredundant-decls -Wnested-externs -Wfloat-equal -Wmissing-format-attribute -Wpacked -Wdisabled-optimization -Wformat=2 -Wunreachable-code -Wendif-labels -Wdeclaration-after-statement -Wold-style-definition -Winvalid-pch -Wvariadic-macros -Winit-self -Wmissing-include-dirs -Wswitch-default -Wswitch-enum -Wunused-macros -Wunsafe-loop-optimizations -Wc++-compat -Wnormalized -Wlogical-op -Wlarger-than=2048 -Wvla -Wsync-nand -Wframe-larger-than=16384 -Wpacked-bitfield-compat -Wstrict-overflow=5 -Wjump-misses-init -Wunsuffixed-float-constants -Wdouble-promotion -Wtrampolines -Wstack-usage=8192 -Wvector-operation-performance -Wdate-time -Warray-bounds=2 -Wc99-c11-compat -Wnull-dereference -Wunused-const-variable -Wduplicated-cond -Whsa -fstdarg-opt -s -Wno-inline -Wno-aggregate-return -Wno-missing-noreturn -Wno-suggest-attribute=const -Wno-suggest-attribute=pure -Wno-suggest-attribute=noreturn -Wno-suggest-attribute=format -O3
AM C Flags:
Shared C Library: yes
Static C Library: yes
Fortran: yes
Fortran Compiler: /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/openmpi-4.0.2-4airvo32ypyuapzgi4fp2kjea5psqu3t/bin/mpif90 ( built with gcc version 6.3.0 (GCC))
Fortran Flags: -fPIC
H5 Fortran Flags: -pedantic -Wall -Wextra -Wunderflow -Wimplicit-interface -Wsurprising -Wno-c-binding-type -s -O2
AM Fortran Flags:
Shared Fortran Library: yes
Static Fortran Library: yes
C++: yes
C++ Compiler: /cluster/spack/apps/linux-centos7-x86_64/gcc-6.3.0/openmpi-4.0.2-4airvo32ypyuapzgi4fp2kjea5psqu3t/bin/mpic++
C++ Flags: -fPIC
H5 C++ Flags: -pedantic -Wall -Wextra -Wundef -Wshadow -Wpointer-arith -Wcast-qual -Wcast-align -Wwrite-strings -Wconversion -Wredundant-decls -Winline -Wsign-promo -Woverloaded-virtual -Wold-style-cast -Weffc++ -Wreorder -Wnon-virtual-dtor -Wctor-dtor-privacy -Wabi -Wfloat-equal -Wmissing-format-attribute -Wmissing-noreturn -Wpacked -Wdisabled-optimization -Wformat=2 -Wendif-labels -Winvalid-pch -Wvariadic-macros -Winit-self -Wmissing-include-dirs -Wswitch-default -Wswitch-enum -Wunused-macros -Wunsafe-loop-optimizations -Wstrict-overflow -Wlogical-op -Wlarger-than=2048 -Wvla -Wsync-nand -Wframe-larger-than=16384 -Wpacked-bitfield-compat -Wstrict-overflow=5 -Wdouble-promotion -Wsuggest-attribute=const -Wtrampolines -Wstack-usage=8192 -Wvector-operation-performance -Wsuggest-attribute=pure -Wsuggest-attribute=noreturn -Wsuggest-attribute=format -Wdate-time -Wopenmp-simd -Warray-bounds=2 -fstdarg-opt -s -O3
AM C++ Flags:
Shared C++ Library: yes
Static C++ Library: yes
Java: no
Features:
---------
Parallel HDF5: yes
High-level library: yes
Threadsafety: no
Default API mapping: v110
With deprecated public symbols: yes
I/O filters (external): deflate(zlib),szip(encoder)
MPE:
Direct VFD: no
dmalloc: no
Packages w/ extra debug output: none
API tracing: no
Using memory checker: no
Memory allocation sanity checks: no
Metadata trace file: no
Function stack tracing: no
Strict file format checks: no
Optimization instrumentation: no
Now so I thought, let’s just compile a c example and see if everything works without the whole python stuff. So I compiled https://support.hdfgroup.org/ftp/HDF5/examples/parallel/coll_test.c using “h5pcc coll_test.c” and the nrun “mpiexec -n 4 ./a.out” which created a file and it seems to work just fine. To be sure I even unloaded MPI and tested it again and it didn’t work (cause no mpiexec).
So now, I also tried compiling it via h5c++ and it all worked fine althought the docs say, the c++ interface does not support parallelism? I guess I can still compile C files i.e. C API using the c++ compiler?
Now, h5py uses the C API. I can’t see the issue here.
Can someone please help me figuring out why h5py doesn’t work?