Cross-compiling for Windows


#1

Hello, I’m trying to cross-compile HDF5 using BinaryBuilder, but it’s still not clear to me the setup to use. I’ve tried calling CMake in the following way:

cmake .. -DCMAKE_INSTALL_PREFIX=${prefix} \
    -DCMAKE_TOOLCHAIN_FILE=${CMAKE_TARGET_TOOLCHAIN} \
    -DHDF5_BUILD_CPP_LIB=OFF \
    -DONLY_SHARED_LIBS=ON \
    -DHDF5_BUILD_HL_LIB=ON \
    -DHDF5_ENABLE_Z_LIB_SUPPORT=ON \
    -DHDF5_ENABLE_SZIP_SUPPORT=OFF \
    -DHDF5_ENABLE_SZIP_ENCODING=OFF \
    -DBUILD_TESTING=OFF

but I get the following errors:

CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately:
   H5_LDOUBLE_TO_LONG_SPECIAL_RUN (advanced)
   H5_LDOUBLE_TO_LONG_SPECIAL_RUN__TRYRUN_OUTPUT (advanced)
For details see /workspace/srcdir/hdf5/build/TryRunResults.cmake
-- Checking IF your system converts long double to (unsigned) long values with special algorithm... no
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately:
   H5_LONG_TO_LDOUBLE_SPECIAL_RUN (advanced)
   H5_LONG_TO_LDOUBLE_SPECIAL_RUN__TRYRUN_OUTPUT (advanced)
For details see /workspace/srcdir/hdf5/build/TryRunResults.cmake
-- Checking IF your system can convert (unsigned) long to long double values with special algorithm... no
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately:
   H5_LDOUBLE_TO_LLONG_ACCURATE_RUN (advanced)
   H5_LDOUBLE_TO_LLONG_ACCURATE_RUN__TRYRUN_OUTPUT (advanced)
For details see /workspace/srcdir/hdf5/build/TryRunResults.cmake
-- Checking IF correctly converting long double to (unsigned) long long values... no
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately:
   H5_LLONG_TO_LDOUBLE_CORRECT_RUN (advanced)
   H5_LLONG_TO_LDOUBLE_CORRECT_RUN__TRYRUN_OUTPUT (advanced)
For details see /workspace/srcdir/hdf5/build/TryRunResults.cmake
-- Checking IF correctly converting (unsigned) long long to long double values... no
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately:
   H5_DISABLE_SOME_LDOUBLE_CONV_RUN (advanced)
   H5_DISABLE_SOME_LDOUBLE_CONV_RUN__TRYRUN_OUTPUT (advanced)
For details see /workspace/srcdir/hdf5/build/TryRunResults.cmake
-- Checking IF the cpu is power9 and cannot correctly converting long double values... no
CMake Error: TRY_RUN() invoked in cross-compiling mode, please set the following cache variables appropriately:
   H5_NO_ALIGNMENT_RESTRICTIONS_RUN (advanced)
   H5_NO_ALIGNMENT_RESTRICTIONS_RUN__TRYRUN_OUTPUT (advanced)
For details see /workspace/srcdir/hdf5/build/TryRunResults.cmake
-- Checking IF alignment restrictions are strictly enforced... no
-- Warnings Configuration:
-- Could NOT find ZLIB (missing: ZLIB_DIR)
-- Found ZLIB: /workspace/destdir/bin/libz.dll (found version "1.2.11") 
-- Filter ZLIB is ON
-- Found Perl: /usr/bin/perl (found version "5.26.3") 
-- Generating 'H5Epubgen.h'
Generating 'H5Einit.h'
Generating 'H5Eterm.h'
Generating 'H5Edefin.h'

-- Generating '/workspace/srcdir/hdf5/src/H5version.h'

-- Generating 'H5overflow.h'

-- Configuring incomplete, errors occurred!
See also "/workspace/srcdir/hdf5/build/CMakeFiles/CMakeOutput.log".
See also "/workspace/srcdir/hdf5/build/CMakeFiles/CMakeError.log".

My understanding was that HDF5 had support for cross-compilation with the CMake build system, but even if I’m passing the CMake toolchain file for the cross-compiler it’s trying to run foreign programs. What should I do?


#2

According to CMake, in the cross-compile toolchain you pre-set those variables to the results of the try run. Yes you need to know what those values are. I would start with the info from the CMake sites and go from there.

What version of hdf5 are you building? The most recent versions have improved support for cross-compiling and support a cross-compile executable. I.E. on ubuntu using mingw, I set the program to use wine.

Allen


#3

Is there a documentation of what these variables represent? I mean, apart from reading the CMake file and guessing what those variable should be set to by reverse engineering the commands that would need to be run. I thought that the “improved support for cross-compiling” was something less convoluted.

b49b22d6882d97b1ec01d482822955bd8e923203


#4

It would be great if someone could help Mosè along a little bit here. He’s doing this to provide the Julia scientific programming community with consistent builds using BinaryBuilder. Having a proper HDF5 build would help a great deal for the many Julia packages that rely on the HDF5 library, both directly and indirectly. Thank you!


#5

This is Steven,
Would it be possible to have a conference call? I am available from 15:00EDT today if that works for you?
Also I need the details of the stack: versionininfo().
best: steve


#6

Cool, thanks, yes, I’m available at that time.

The output of versioninfo() in my Julia REPL is

julia> versioninfo()
Julia Version 1.3.1
Commit 2d5741174c (2019-12-30 21:36 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-4870HQ CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, haswell)

Is this what you were asking?


#7

yes. 14:00EDT is good. skype: vargaconsulting.ca google video: steven.varga@gmail.com
best:
steve


#8

Mose,
It was great meeting you! Can we use this download link https://github.com/steven-varga/hdf5.git instead of the official ?

sources = [
     GitSource("https://bitbucket.hdfgroup.org/scm/hdffv/hdf5.git",
              "b49b22d6882d97b1ec01d482822955bd8e923203"),
]

This is to set up a custom pipeline for us to fiddle with, as I have to make some modifications to the HDF5 system. The process is:

  1. modify cmake + HDF5 on my laptop
  2. upload back to my github HDF5 repository:
  3. send pull request against your Yggrsdrasil/HDF5
  4. check for errors
  5. start again or celebrate

Once you OK-d this, we can do a test round - to oil up the cogs, so to speak.

what do you think?
steve


#9

Yes, no problem with using it!


#10

Hi guys,

Have you made progress w/ the cross-compilation issues? I’m currently trying something similar, attempting to compile for a PowerPC target using an x86 host and the IBM Advanced Toolchain. Following the snippets above compilation fails quickly with

.../build/bin/H5detect: cannot execute binary file

which I guess is to be expected. I would be grateful for any advice, and am happy to help out if I can.

Mike.


#11

Hi Mike,

according to Allen Byrn the cmake system needs a set of variables pre-defined: here is the source code to the test programs and a makefile to produce the macro definitions

Once you compiled and ran the test programs on the target architecture, please collect the results and send them to me. (or better: create a ticket on my github page). This is how far I got – other than examined alternative approaches.
When enough results are collected, it is easy to observe the stationary variables, and the ones which are actually function of the architecture.
Since I am to do this in my own time, and resources help is always welcome – in any format. Please anyone who reads this, and capable of running the test suite on unusual architecture please help make this happen by submitting the results.

best: steve


#12

Hi Steven,

Thanks for the response. We’ve decided in the meantime to compile directly on a cloud instance of a PowerPC machine, primarily for testing purposes. We’re still evaluating this as a final solution though – I’ll ping you if / when we continue down the ppc cross-compilation path.

Cheers.


#13

Can you please point me to the direction how that works? I work on H5CPP, an HDF5 based persistence for modern C++, and would love to have access rental (IAAS) computers of less common architecture for testing purpose.
best: steve


#14

Hi Steven,

Sure. Our company got limited access to a system for porting via the IBM Power Development Cloud.

Best,
Mike


#15

We additionally also been looking a the OpenPOWER Foundation, and will re-evaluate once we get beyond the deployment stage.


#16

Thanks for the links! I’ve applied for both. If you folks do C++ development you may be interested in my project: http://isc19.h5cpp.org

best:steve


#17

Hi all, has there been any progress on this front wrt cross-compilation support for Windows and other platforms from cmake?

There is also this half-finished PR https://github.com/stevengj/hdf5/pull/1


#18

Hi, thanks for the interest. I am aware of Steven’s work. The progress is slow, as I need the output of this program executed on various platforms.

best: steve


#19

Which program exactly and which platforms should we post the output for? I have a Windows machine I can post the output of and I’m sure I can ask for other platforms, so we can help support this process.


#20

The more the better, as far as I recall a conversation with Allen Byrne cmake version is the more promising way to go. There are a set of cmake variables need to be set, downloading this source then executing make the list is printed with the correct values for your system, which is the target.

You need to have these values for all target platforms. In my understanding the HDF5 cmake and configure executes two programs: ConversionTests.c, HDFTests.c with various -Dxyz macros defined, and statistics collected. The provided make file mimics this behaviour and collects the statistics for you.

In a different phase, using the collected statistics and a modified/updated cmake script one should be able to cross compile to given target.

Analysing the statistics one could possibly simplify the procedure, without dire consequences, but that is only a hunch. Also note as months go by, all that readings I did half year ago is slowly evaporating, so I could be slightly or entirely wrong and your new compiler tool chain will be choking on zeros and ones.

best wishes:
steven
vargaconsulting