Hello,
As a dependency for another project I would like to install the old version of h5py==2.10.0 on a rolling-release linux system (manjaro) with pip 20.2.2 from /usr/lib/python3.9/site-packages/pip (python 3.9).
Running pip install h5py==2.10.0 leads to following error message:
Collecting h5py==2.10.0
Using cached h5py-2.10.0.tar.gz (301 kB)
Requirement already satisfied: numpy>=1.7 in /usr/lib/python3.9/site-packages (from h5py==2.10.0) (1.19.4)
Requirement already satisfied: six in /usr/lib/python3.9/site-packages (from h5py==2.10.0) (1.15.0)
Installing collected packages: h5py
Running setup.py install for h5py ... error
ERROR: Command errored out with exit status 1:
command: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mfjc2p9w/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mfjc2p9w/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-yoa5iho8/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/user/.local/include/python3.9/h5py
cwd: /tmp/pip-install-mfjc2p9w/h5py/
[...]
In file included from /usr/lib/python3.9/site-packages/numpy/core/include/numpy/ndarrayobject.h:21,
from /usr/lib/python3.9/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from /tmp/pip-install-mfjc2p9w/h5py/h5py/api_compat.h:26,
from /tmp/pip-install-mfjc2p9w/h5py/h5py/h5i.c:654:
At top level:
/usr/lib/python3.9/site-packages/numpy/core/include/numpy/__multiarray_api.h:1463:1: warning: â_import_arrayâ defined but not used [-Wunused-function]
1463 | _import_array(void)
| ^~~~~~~~~~~~~
error: command '/usr/bin/gcc' failed with exit code 1
----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-mfjc2p9w/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-mfjc2p9w/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-yoa5iho8/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/user/.local/include/python3.9/h5py Check the logs for full command output.
This happens both when running with and without administrator rights. Any suggestion?
I suspect youâll need to play with different versions of cython to get h5py 2.10 to build, I donât think we released python 3.9 wheels for 2.10.
1 Like
Thanks for your answer. Indeed installing with python 3.7.7 worked well.
Hello,
I have the need to use hdf5 v1.8.22 and so am trying to install a similar aged h5py==2.10.0 onto a python 3.9 distribution. First I install cython (it doesnât seem to matter what version, but here I use a similar vintage version):
pip install --no-cache-dir -I --no-binary=cython cython==0.29.22
then I install h5py (having set HDF5_DIR and HDF5_VERSION to point to my hdf5 build):
pip install --no-cache-dir -I --no-binary=h5py h5py==2.10.0
and this fails because it canât find Cython:
...
self.distribution.run_command(command)
File "/tmp/jay/pip-build-env-xx4797jf/overlay/lib/python3.9/site-packages/setuptools/dist.py", line 1233, in run_command
super().run_command(command)
File "/tmp/jay/pip-build-env-xx4797jf/overlay/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/tmp/jay/pip-install-lsdm4si0/h5py_605d1be1fdac465892ceac0d0878f29e/setup_build.py", line 161, in run
from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
[end of output]
It turns out that I can install h5py==3.8.0 and h5py==3.9.0 successfully (although their symbol tables donât match my hdf5 1.8.22 library, which is why I want to build an old h5py), but I canât build an older version.
My question is: is there a known break in support for older h5py (and hdf5) in python 3.9? And is there any chance that this can be made to work? Thanks.
Hello @salmonson1,
Have you tried installing h5py 3.9.0 with your libhdf5 version? I do not understand what you meant by âIt turns out that I can install h5py==3.8.0 and h5py==3.9.0 successfullyâ. Did you try something like:
% HDF5_DIR=... pip install --no-cache-dir --no-binary=h5py h5py
Are you using a virtual environment for this?
Aleksandar
Thank you for the rapid response. Yes, I did exactly as you say to install h5py==3.9.0; setting HDF5_DIR and HDF5_VERSION to point to my hdf5 library (v1.8.22). And yes, this is in a virtual environment. It installs ok (both 3.9.0 or 3.8.0), but I when I start python and âimport h5pyâ I get âundefined symbolsâ errors. My hypothesis is that this is because Iâm trying to link a modern h5py (3.9.0) to an old hdf5 library (1.8.22), thus I am trying to build an older h5py.
Thanks, Jay
Good. It is much better to re-try from a clean slate by deleting the old virtual environment.
Can you share the undefined symbols errors on h5py import?
As for trying the old h5py version, I suggest only running the pip install command for h5py and let it figure out which cython version to install as dependency.
Aleksandar
The error it reports is: undefined symbol: H5Rdereference. I will follow up with more undefined symbol names (as per running ânm hdf5_lib.soâ).
Jay
Ah yes⌠That function is available from libhdf5-1.10.0 but in the h5py source is not marked as such.
pip can build h5py from a git branch. If you supply a list of undefined symbols, I can prepare a special version of h5py that should hopefully build with your old version of libhdf5.
Aleksandar
That would be spectacular, Aleksandar! There are many undefined symbols. Here I installed current versions: cython 3.0.2 and h5py 3.9.0, compiled against HDF5 1.8.22. Then, running nm defs.cpython-39-x86_64-linux-gnu.so gives the following Undefined symbols (U):
U H5Aclose
U H5Acreate2
U H5Acreate_by_name
U H5Adelete
U H5Adelete_by_idx
U H5Adelete_by_name
U H5Aexists
U H5Aexists_by_name
U H5Aget_info
U H5Aget_info_by_idx
U H5Aget_info_by_name
U H5Aget_name
U H5Aget_num_attrs
U H5Aget_space
U H5Aget_storage_size
U H5Aget_type
U H5Aiterate2
U H5Aopen
U H5Aopen_by_idx
U H5Aopen_by_name
U H5Aopen_idx
U H5Aopen_name
U H5Aread
U H5Arename
U H5Arename_by_name
U H5Awrite
U H5check_version
U H5close
U H5Dclose
U H5Dcreate2
U H5Dcreate_anon
U H5Dextend
U H5Dfill
U H5Dget_access_plist
U H5Dget_create_plist
U H5Dget_offset
U H5Dget_space
U H5Dget_space_status
U H5Dget_storage_size
U H5Dget_type
U H5Diterate
U H5Dopen2
U H5DOwrite_chunk
U H5Dread
U H5DSattach_scale
U H5DSdetach_scale
U H5Dset_extent
U H5DSget_label
U H5DSget_num_scales
U H5DSget_scale_name
U H5DSis_attached
U H5DSis_scale
U H5DSiterate_scales
U H5DSset_label
U H5DSset_scale
U H5Dvlen_get_buf_size
U H5Dvlen_reclaim
U H5Dwrite
U H5Fclose
U H5Fcreate
U H5FDregister
U H5FDunregister
U H5Fflush
U H5Fget_access_plist
U H5Fget_create_plist
U H5Fget_file_image
U H5Fget_filesize
U H5Fget_freespace
U H5Fget_intent
U H5Fget_mdc_config
U H5Fget_mdc_hit_rate
U H5Fget_mdc_size
U H5Fget_name
U H5Fget_obj_count
U H5Fget_obj_ids
U H5Fget_vfd_handle
U H5Fis_hdf5
U H5Fmount
U H5Fopen
U H5free_memory
U H5Freopen
U H5Freset_mdc_hit_rate_stats
U H5Fset_mdc_config
U H5Funmount
U H5Gclose
U H5Gcreate2
U H5Gcreate_anon
U H5get_libversion
U H5Gget_comment
U H5Gget_create_plist
U H5Gget_info
U H5Gget_info_by_name
U H5Gget_linkval
U H5Gget_num_objs
U H5Gget_objinfo
U H5Gget_objname_by_idx
U H5Gget_objtype_by_idx
U H5Giterate
U H5Glink2
U H5Gmove2
U H5Gopen2
U H5Gset_comment
U H5Gunlink
U H5Idec_ref
U H5Iget_file_id
U H5Iget_name
U H5Iget_ref
U H5Iget_type
U H5Iinc_ref
U H5Iis_valid
U H5Lcopy
U H5Lcreate_external
U H5Lcreate_hard
U H5Lcreate_soft
U H5Ldelete
U H5Ldelete_by_idx
U H5Lexists
U H5Lget_info
U H5Lget_info_by_idx
U H5Lget_name_by_idx
U H5Lget_val
U H5Lget_val_by_idx
U H5Literate
U H5Literate_by_name
U H5Lmove
U H5LTopen_file_image
U H5Lunpack_elink_val
U H5Lvisit
U H5Lvisit_by_name
U H5Oclose
U H5Ocopy
U H5Odecr_refcount
U H5Oexists_by_name
U H5Oget_comment
U H5Oget_comment_by_name
U H5Oget_info
U H5Oget_info_by_idx
U H5Oget_info_by_name
U H5Oincr_refcount
U H5Olink
U H5Oopen
U H5Oopen_by_addr
U H5Oopen_by_idx
U H5open
U H5Oset_comment
U H5Oset_comment_by_name
U H5Ovisit
U H5Ovisit_by_name
U H5Pall_filters_avail
U H5Pclose
U H5Pclose_class
U H5Pcopy
U H5Pcreate
U H5Pequal
U H5Pfill_value_defined
U H5Pget_alignment
U H5Pget_alloc_time
U H5Pget_attr_creation_order
U H5Pget_attr_phase_change
U H5Pget_cache
U H5Pget_char_encoding
U H5Pget_chunk
U H5Pget_chunk_cache
U H5Pget_class
U H5Pget_copy_object
U H5Pget_create_intermediate_group
U H5Pget_driver
U H5Pget_driver_info
U H5Pget_edc_check
U H5Pget_efile_prefix
U H5Pget_elink_acc_flags
U H5Pget_elink_fapl
U H5Pget_elink_prefix
U H5Pget_est_link_info
U H5Pget_external
U H5Pget_external_count
U H5Pget_family_offset
U H5Pget_fapl_core
U H5Pget_fapl_family
U H5Pget_fclose_degree
U H5Pget_fill_time
U H5Pget_fill_value
U H5Pget_filter2
U H5Pget_filter_by_id2
U H5Pget_istore_k
U H5Pget_layout
U H5Pget_libver_bounds
U H5Pget_link_creation_order
U H5Pget_link_phase_change
U H5Pget_local_heap_size_hint
U H5Pget_mdc_config
U H5Pget_meta_block_size
U H5Pget_nfilters
U H5Pget_nlinks
U H5Pget_obj_track_times
U H5Pget_sieve_buf_size
U H5Pget_sizes
U H5Pget_sym_k
U H5Pget_userblock
U H5Pget_version
U H5Pmodify_filter
U H5Premove_filter
U H5Pset_alignment
U H5Pset_alloc_time
U H5Pset_attr_creation_order
U H5Pset_attr_phase_change
U H5Pset_cache
U H5Pset_char_encoding
U H5Pset_chunk
U H5Pset_chunk_cache
U H5Pset_copy_object
U H5Pset_create_intermediate_group
U H5Pset_deflate
U H5Pset_driver
U H5Pset_edc_check
U H5Pset_efile_prefix
U H5Pset_elink_acc_flags
U H5Pset_elink_fapl
U H5Pset_elink_prefix
U H5Pset_est_link_info
U H5Pset_external
U H5Pset_family_offset
U H5Pset_fapl_core
U H5Pset_fapl_family
U H5Pset_fapl_log
U H5Pset_fapl_multi
U H5Pset_fapl_sec2
U H5Pset_fapl_split
U H5Pset_fapl_stdio
U H5Pset_fclose_degree
U H5Pset_file_image
U H5Pset_fill_time
U H5Pset_fill_value
U H5Pset_filter
U H5Pset_fletcher32
U H5Pset_istore_k
U H5Pset_layout
U H5Pset_libver_bounds
U H5Pset_link_creation_order
U H5Pset_link_phase_change
U H5Pset_local_heap_size_hint
U H5Pset_mdc_config
U H5Pset_meta_block_size
U H5Pset_nlinks
U H5Pset_obj_track_times
U H5Pset_scaleoffset
U H5Pset_shuffle
U H5Pset_sieve_buf_size
U H5Pset_sizes
U H5Pset_sym_k
U H5Pset_szip
U H5Pset_userblock
U H5Rcreate
U H5Rdereference
U H5Rget_name
U H5Rget_obj_type2
U H5Rget_region
U H5Sclose
U H5Scopy
U H5Screate
U H5Screate_simple
U H5Sdecode
U H5Sencode
U H5Sextent_copy
U H5Sget_select_bounds
U H5Sget_select_elem_npoints
U H5Sget_select_elem_pointlist
U H5Sget_select_hyper_blocklist
U H5Sget_select_hyper_nblocks
U H5Sget_select_npoints
U H5Sget_select_type
U H5Sget_simple_extent_dims
U H5Sget_simple_extent_ndims
U H5Sget_simple_extent_npoints
U H5Sget_simple_extent_type
U H5Sis_simple
U H5Soffset_simple
U H5Sselect_all
U H5Sselect_elements
U H5Sselect_hyperslab
U H5Sselect_none
U H5Sselect_valid
U H5Sset_extent_none
U H5Sset_extent_simple
U H5Tarray_create2
U H5Tclose
U H5Tcommit2
U H5Tcommitted
U H5Tconvert
U H5Tcopy
U H5Tcreate
U H5Tdecode
U H5Tdetect_class
U H5Tencode
U H5Tenum_create
U H5Tenum_insert
U H5Tenum_nameof
U H5Tenum_valueof
U H5Tequal
U H5Tfind
U H5Tget_array_dims2
U H5Tget_array_ndims
U H5Tget_class
U H5Tget_create_plist
U H5Tget_cset
U H5Tget_ebias
U H5Tget_fields
U H5Tget_inpad
U H5Tget_member_class
U H5Tget_member_index
U H5Tget_member_name
U H5Tget_member_offset
U H5Tget_member_type
U H5Tget_member_value
U H5Tget_native_type
U H5Tget_nmembers
U H5Tget_norm
U H5Tget_offset
U H5Tget_order
U H5Tget_pad
U H5Tget_precision
U H5Tget_sign
U H5Tget_size
U H5Tget_strpad
U H5Tget_super
U H5Tget_tag
U H5Tinsert
U H5Tis_variable_str
U H5Tlock
U H5Topen2
U H5Tpack
U H5Tregister
U H5Tset_cset
U H5Tset_ebias
U H5Tset_fields
U H5Tset_inpad
U H5Tset_norm
U H5Tset_offset
U H5Tset_order
U H5Tset_pad
U H5Tset_precision
U H5Tset_sign
U H5Tset_size
U H5Tset_strpad
U H5Tset_tag
U H5Tunregister
U H5Tvlen_create
U H5Zfilter_avail
U H5Zget_filter_info
U H5Zregister
U H5Zunregister
and also some non-HDF5-specific symbols:
U PyBytes_FromStringAndSize
U PyCapsule_GetName
U PyCapsule_GetPointer
U PyCapsule_IsValid
U PyCapsule_New
U PyCode_NewEmpty
U _Py_Dealloc
U _PyDict_GetItem_KnownHash
U PyDict_GetItemString
U PyDict_New
U PyDict_SetItem
U PyDict_SetItemString
U Py_EnterRecursiveCall
U PyErr_Clear
U PyErr_ExceptionMatches
U PyErr_Format
U PyErr_Occurred
U PyErr_SetObject
U PyErr_SetString
U PyErr_WarnEx
U PyEval_RestoreThread
U PyEval_SaveThread
U PyExc_AttributeError
U PyExc_ImportError
U PyExc_NameError
U PyExc_RuntimeError
U PyExc_SystemError
U PyExc_TypeError
U PyExc_ValueError
U _Py_FalseStruct
U PyFrame_New
U Py_GetVersion
U PyImport_AddModule
U PyImport_GetModuleDict
U PyImport_ImportModule
000000000002fe10 T PyInit_defs
U PyInterpreterState_GetID
U Py_LeaveRecursiveCall
U PyMem_Malloc
U PyMem_Realloc
U PyModule_AddObject
U PyModuleDef_Init
U PyModule_GetDict
U PyModule_GetName
U PyModule_NewObject
U _Py_NoneStruct
U PyObject_Call
U PyObject_GetAttr
U PyObject_GetAttrString
U _PyObject_GetDictPtr
U PyObject_Hash
U PyObject_Not
U PyObject_SetAttr
U PyObject_SetAttrString
U PyOS_snprintf
U PyThreadState_Get
U _PyThreadState_UncheckedGet
U PyTraceBack_Here
U _Py_TrueStruct
U PyTuple_New
U PyTuple_Pack
U PyUnicode_AsUTF8
U PyUnicode_Decode
U PyUnicode_FromFormat
U PyUnicode_FromStringAndSize
U PyUnicode_InternFromString
Please advise if you need any more information.
Many thanks!
Jay
Hi @salmonson1,
Something is wrong⌠the list of undefined symbols is way too long. Many of the listed libhdf5 functions have existed since forever and therefore be correctly defined even in the h5py 2.10 code. Also, the Py... undefined symbols are for python and may indicate that the python development resources (packages) are missing or not found.
How did you install libhdf5-1.8.22? Can you run its tools, like h5dump or h5ls on an HDF5 file? How did you install Python 3.9? Whatâs the computer platform you are using?
Aleksandar
