HDF5 cross-compile error: configure:23533: error: cannot run test program while cross compiling

Hdf-forum archives
1

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

···

=======================================
configure:23533: error: cannot run test program while cross compiling

Details below in config.log snippet:

This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by HDF5 configure 1.8.14, which was
generated by GNU Autoconf 2.69. Invocation command line was

   $ ./configure --target=x86_64-buildroot-linux-uclibc --host=x86_64-buildroot-linux-uclibc --build=x8
6_64-unknown-linux-gnu --prefix=/usr --exec-prefix=/usr --sysconfdir=/etc --localstatedir=/var --progr
am-prefix= --disable-gtk-doc --disable-doc --disable-docs --disable-documentation --with-xmlto=no --wi
th-fop=no --disable-dependency-tracking --disable-nls --disable-ipv6 --enable-debug --enable-static --
enable-shared --enable-threadsafe --with-szlib

## --------- ##
## Platform. ##
## --------- ##

hostname = localhost.localdomain
uname -m = x86_64
uname -r = 3.17.4-301.fc21.x86_64
uname -s = Linux
uname -v = #1 SMP Thu Nov 27 19:09:10 UTC 2014

/usr/bin/uname -p = x86_64
/bin/uname -X = unknown
/bin/arch = x86_64
/usr/bin/arch -k = unknown
/usr/convex/getsysinfo = unknown
/usr/bin/hostinfo = unknown
/bin/machine = unknown
/usr/bin/oslevel = unknown
/bin/universe = unknown

PATH: /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/bin
PATH: /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/sbin
PATH: /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/usr/bin
PATH: /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/usr/sbin
PATH: /usr/lib64/qt-3.3/bin
PATH: /usr/lib/qtchooser
PATH: /usr/local/bin
PATH: /usr/local/sbin
PATH: /usr/bin
PATH: /usr/sbin
PATH: /bin
PATH: /sbin
PATH: /home/ernesto/.local/bin
PATH: /home/ernesto/bin

## ----------- ##
## Core tests. ##
## ----------- ##
configure:3254: checking for a BSD-compatible install
configure:3322: result: /usr/bin/install -c
configure:3333: checking whether build environment is sane
configure:3388: result: yes
configure:3447: checking for x86_64-buildroot-linux-uclibc-strip
configure:3474: result: /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux
-x86_64/usr/bin/x86_64-buildroot-linux-uclibc-strip
configure:3539: checking for a thread-safe mkdir -p
configure:3578: result: /usr/bin/mkdir -p
configure:3585: checking for gawk
configure:3601: found /usr/bin/gawk
configure:3612: result: gawk
configure:3623: checking whether make sets $(MAKE)
configure:3645: result: yes
configure:3674: checking whether make supports nested variables
configure:3691: result: yes
configure:3827: checking whether make supports nested variables
configure:3844: result: yes
configure:3867: checking whether to enable maintainer-specific portions of Makefiles
configure:3876: result: no
configure:3920: checking build system type
configure:3934: result: x86_64-unknown-linux-gnu
configure:3954: checking host system type
configure:3967: result: x86_64-buildroot-linux-uclibc
configure:4046: checking shell variables initial values
ACLOCAL='${SHELL} /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/output/build/hdf5-1
.8.14/bin/missing aclocal-1.14'
AMTAR='$${TAR-tar}'
AM_BACKSLASH='\'
AM_CFLAGS=
AM_CPPFLAGS=
AM_CXXFLAGS=
AM_DEFAULT_V='$(AM_DEFAULT_VERBOSITY)'
AM_DEFAULT_VERBOSITY=0
AM_FCFLAGS=
AM_LDFLAGS=' -L/usr/lib'
AM_V='$(V)'
AR=/home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/usr/bin/x86_6
4-buildroot-linux-uclibc-ar
AR_FOR_BUILD=/usr/bin/ar
AS=/home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/usr/bin/x86_6
4-buildroot-linux-uclibc-as
AS_FOR_BUILD=/usr/bin/as
AUTOCONF='${SHELL} /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/output/build/hdf5-
1.8.14/bin/missing autoconf'
AUTOHEADER='${SHELL} /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/output/build/hdf
5-1.8.14/bin/missing autoheader'

*

configure:23155: result: yes
configure:23226: checking if libtool needs -no-undefined flag to build shared libraries
configure:23237: result: no
configure:23246: checking if configure should try to set up large file support
configure:23276: result: yes
configure:23281: checking for special C compiler options needed for large files
configure:23334: result: no
configure:23348: checking for _FILE_OFFSET_BITS value needed for large files
configure:23381: /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64
/usr/bin/ccache /home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/../host/linux-x86_64/
usr/bin/x86_64-buildroot-linux-uclibc-gcc -c -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -pipe -O2 -g2
  -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE conftest.c >&5
configure:23381: $? = 0
configure:23421: result: no
configure:23525: checking if large (64-bit) files are supported on this system.
configure:23531: error: in `/home/ernesto/packages/buildroot-2014.11/buildroot-uclibc-x86_64/output/bu
ild/hdf5-1.8.14':
configure:23533: error: cannot run test program while cross compiling
See `config.log' for more details

## ---------------- ##
## Cache variables. ##
## ---------------- ##

=====================================

Cheers,
Ernesto

2

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

bgq-hdf5build18.sh (2.72 KB)

···

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

3

Hi Rob,

···

________________________________________
From: Hdf-forum [hdf-forum-bounces@lists.hdfgroup.org] on behalf of Rob Latham [robl@mcs.anl.gov]
Sent: Monday, January 5, 2015 7:10 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] HDF5 cross-compile error: configure:23533: error: cannot run test program while cross compiling

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a
gigantic pain.

I have managed to work-around this in my "buildroot" environment.
I am using the buildroot tools to build an embedded linux target and tool-chain.
The buildroot developers gave me some tricks to handle the "AC_TRY_RUN"

Workaround: set a pile of environment variables. Most of the tests are
obvious: checking for example if you are not running on an HP-UX machine
with a buggy compiler from 1998 (I think I have exaggerated that
slightly... but only slightly)

Fortunately, they do not seem to be adding any *new* instances of
AC_TRY_RUN, so once you set up all the needed environment variables,
you'll be set for the next several release cycles.

I hope they will start supporting cross-builds. autotools definitely has the infrastructure. :slight_smile:

To get you started, here's a shell script (attached) I use with lots of
the necessary environment variables set. Richard Hedges got me most of
the way a few years back (in the 1.6 days), and so I pay it forward to
you. I did this for Blue Gene, so usual caveats apply: I have no idea
what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode,
then re-configure everything for cross compiling.

Now, here is the most non portable part:
The automatic generation of "C" files with a machine dependent executable. :frowning:
================================== ERROR =======================================
/bin/sh: ./H5make_libsettings: /lib/ld64-uClibc.so.0: bad ELF interpreter: No such file or directory

These "C" files should be generated via autotools "configure" and then some portable shell script such as PERL ?
In any, case how important are the following files for using the API only:

(1) H5lib_settings.c
(2) H5Tinit.c

In my case, since my embedded target is "x86_64-buildroot-linux-uclibc" which is very similar to my build host --> ":x86_64-unknown-linux-gnu"
I just copied those files into the buildroot environment but this would not work for such as an ARM target :frowning:

Thanks,
Ernesto

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

4

Hi Rob,

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn’t run on the right platform (architecture).

Elena

···

On Jan 5, 2015, at 9:10 AM, Rob Latham <robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

5

Hi Rob,

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn�t run on the right platform (architecture).

The other problem is "H5lib_settings.c"

If these files are really needed, they could be generated with autotools and PERL.

Cheers,
Ernest

···

On 01/05/2015 10:29 AM, Elena Pourmal wrote:

On Jan 5, 2015, at 9:10 AM, Rob Latham <robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Elena

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

6

Hi Ernest,

Let me see if I understand your need correctly. You would like to:
   configure and compile the HDF5 library in machine A but the binary generated does not run in machine A but runs in platform B.
Is that right?

Do you have a means to launch binary generated in machine A to run in platform B? E.g.,
machine A $ Bcc program.c -o a.out
machine A $ Blaunch ./a.out
and Blaunch passes back the exit code of ./a.out run in platform B.

If so, we already have a means to do that for HDF5 library, using $RUNSERIAL and yodconfigure.
This is created to handle the case of building and running HDF5 software in the compute clusters
environments in which binary are generated in a frontend machine and executed in the backend
compute nodes. You can see details of it in release_docs/INSTALL_parallel, section
2.4.1 Building serial HDF5 for Red Storm

Try it out to see if it meets your need.

-Albert Cheng
THG staff

···

On Jan 5, 2015, at 12:41 PM, Ernest L. Williams Jr. <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/05/2015 10:29 AM, Elena Pourmal wrote:

Hi Rob,

On Jan 5, 2015, at 9:10 AM, Rob Latham <robl@mcs.anl.gov><mailto:robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn’t run on the right platform (architecture).

The other problem is "H5lib_settings.c"

If these files are really needed, they could be generated with autotools and PERL.

Cheers,
Ernest

Elena

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

7

Hi Ernest,

Let me see if I understand your need correctly. You would like to:
   configure and compile the HDF5 library in machine A but the binary generated does not run in machine A but runs in platform B.
Is that right?

That is correct but there is another caveat.
Machine A and machine B are different architectures.
If we were talking about java code then I would not have an issue.

However, we are talking C-code.

Machine A for example is INTEL x86
Machine B is PowerPC

I need to perform a cross compile.

I don't have a problem cross compiling the SZIP package.
However, I do have the problem compiling the HDF5 package.

Since, your group use the GNU autotools for building packages and support for cross compiling should be achievable. If the hdf5 code has been written with portability in mind, this would make supporting cross compiling easier as well.

Do you have a means to launch binary generated in machine A to run in platform B? E.g.,
machine A $ Bcc program.c -o a.out
machine A $ Blaunch ./a.out
and Blaunch passes back the exit code of ./a.out run in platform B.

If so, we already have a means to do that for HDF5 library, using $RUNSERIAL and yodconfigure.
This is created to handle the case of building and running HDF5 software in the compute clusters
environments in which binary are generated in a frontend machine and executed in the backend
compute nodes. You can see details of it in release_docs/INSTALL_parallel, section
2.4.1 Building serial HDF5 for Red Storm

For my particular application; I am not using compute clusters.

Cheers,
Ernest

···

On 01/06/2015 09:28 AM, Albert Cheng wrote:

Try it out to see if it meets your need.

-Albert Cheng
THG staff

On Jan 5, 2015, at 12:41 PM, Ernest L. Williams Jr. > <ernesto@slac.stanford.edu <mailto:ernesto@slac.stanford.edu>> wrote:

On 01/05/2015 10:29 AM, Elena Pourmal wrote:

Hi Rob,

On Jan 5, 2015, at 9:10 AM, Rob Latham<robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn�t run on the right platform (architecture).

The other problem is "H5lib_settings.c"

If these files are really needed, they could be generated with autotools and PERL.

Cheers,
Ernest

Elena

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter:https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

8

Hi Ernest,

Let us say you do the compiling in machine A.
Can you “remote execute” the generated a.out in machine B?
If so, let us say it is called “Blaunch”.
You can try these steps in a fresh hdf5 source directory.

1. cp configure Bconfigure # just to keep configure original in case
2. env RUNSERIAL=“Blaunch” bin/yodconfigure Bconfigure
3. env RUNSERIAL=“Blaunch” Bconfigure
4. env RUNSERIAL=“Blaunch” gmake # you can use make if it is gmake compatible

Let me know if they work for you.

-Albert Cheng
THG staff

···

On Jan 6, 2015, at 12:41 PM, Ernest L Williams <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/06/2015 09:28 AM, Albert Cheng wrote:
Hi Ernest,

Let me see if I understand your need correctly. You would like to:
   configure and compile the HDF5 library in machine A but the binary generated does not run in machine A but runs in platform B.
Is that right?
That is correct but there is another caveat.
Machine A and machine B are different architectures.
If we were talking about java code then I would not have an issue.

However, we are talking C-code.

Machine A for example is INTEL x86
Machine B is PowerPC

I need to perform a cross compile.

I don't have a problem cross compiling the SZIP package.
However, I do have the problem compiling the HDF5 package.

Since, your group use the GNU autotools for building packages and support for cross compiling should be achievable. If the hdf5 code has been written with portability in mind, this would make supporting cross compiling easier as well.

Do you have a means to launch binary generated in machine A to run in platform B? E.g.,
machine A $ Bcc program.c -o a.out
machine A $ Blaunch ./a.out
and Blaunch passes back the exit code of ./a.out run in platform B.

If so, we already have a means to do that for HDF5 library, using $RUNSERIAL and yodconfigure.
This is created to handle the case of building and running HDF5 software in the compute clusters
environments in which binary are generated in a frontend machine and executed in the backend
compute nodes. You can see details of it in release_docs/INSTALL_parallel, section
2.4.1 Building serial HDF5 for Red Storm

For my particular application; I am not using compute clusters.

Cheers,
Ernest

Try it out to see if it meets your need.

-Albert Cheng
THG staff

On Jan 5, 2015, at 12:41 PM, Ernest L. Williams Jr. <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/05/2015 10:29 AM, Elena Pourmal wrote:

Hi Rob,

On Jan 5, 2015, at 9:10 AM, Rob Latham <robl@mcs.anl.gov><mailto:robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn’t run on the right platform (architecture).

The other problem is "H5lib_settings.c"

If these files are really needed, they could be generated with autotools and PERL.

Cheers,
Ernest

Elena

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

9
10

Let us say you do the compiling in machine A.
Can you �remote execute� the generated a.out in machine B?

========= ELW ===============
The answer is no.
This is because machine B is a powerPC and machine A is an INTEL x86 CPU.
Again, I need a cross compiler.

Right, you are cross compiling on the host (or you would simply build directly on the target). How do you get exectuables from host to target? One way someone could do such a thing (maybe not your way?) is to ssh:

--8<------
!/bin/sh
scp a.out target:
ssh target a.out
--8<------

What Albert suggested was you call this script above 'blaunch', and it's the way some folks like to configure on Blue Gene or one of the old ASCI machines (I forget which color...)

now, if executing on the target means "i have to burn an image to flash and boot the board", well, it gets more difficult.

==rob

···

On 01/06/2015 02:47 PM, Williams Jr., Ernest L. wrote:

If so, let us say it is called �Blaunch�.
You can try these steps in a fresh hdf5 source directory.

1. cp configure Bconfigure # just to keep configure original in case
2. env RUNSERIAL=�Blaunch� bin/yodconfigure Bconfigure
3. env RUNSERIAL=�Blaunch� Bconfigure
4. env RUNSERIAL=�Blaunch� gmake # you can use make if it is gmake compatible

Let me know if they work for you.

-Albert Cheng
THG staff

On Jan 6, 2015, at 12:41 PM, Ernest L Williams <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/06/2015 09:28 AM, Albert Cheng wrote:
Hi Ernest,

Let me see if I understand your need correctly. You would like to:
    configure and compile the HDF5 library in machine A but the binary generated does not run in machine A but runs in platform B.
Is that right?
That is correct but there is another caveat.
Machine A and machine B are different architectures.
If we were talking about java code then I would not have an issue.

However, we are talking C-code.

Machine A for example is INTEL x86
Machine B is PowerPC

I need to perform a cross compile.

I don't have a problem cross compiling the SZIP package.
However, I do have the problem compiling the HDF5 package.

Since, your group use the GNU autotools for building packages and support for cross compiling should be achievable. If the hdf5 code has been written with portability in mind, this would make supporting cross compiling easier as well.

Do you have a means to launch binary generated in machine A to run in platform B? E.g.,
machine A $ Bcc program.c -o a.out
machine A $ Blaunch ./a.out
and Blaunch passes back the exit code of ./a.out run in platform B.

If so, we already have a means to do that for HDF5 library, using $RUNSERIAL and yodconfigure.
This is created to handle the case of building and running HDF5 software in the compute clusters
environments in which binary are generated in a frontend machine and executed in the backend
compute nodes. You can see details of it in release_docs/INSTALL_parallel, section
2.4.1 Building serial HDF5 for Red Storm

For my particular application; I am not using compute clusters.

Cheers,
Ernest

Try it out to see if it meets your need.

-Albert Cheng
THG staff

On Jan 5, 2015, at 12:41 PM, Ernest L. Williams Jr. <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/05/2015 10:29 AM, Elena Pourmal wrote:

Hi Rob,

On Jan 5, 2015, at 9:10 AM, Rob Latham <robl@mcs.anl.gov><mailto:robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn�t run on the right platform (architecture).

The other problem is "H5lib_settings.c"

If these files are really needed, they could be generated with autotools and PERL.

Cheers,
Ernest

Elena

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

11

Hi Rob,

Yes, you could SSH

However, we use TFTP to load the OS then NFS for the remaining files.

Cheers,
Ernest

···

________________________________________
From: Hdf-forum [hdf-forum-bounces@lists.hdfgroup.org] on behalf of Rob Latham [robl@mcs.anl.gov]
Sent: Wednesday, January 7, 2015 6:27 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] HDF5 cross-compile error: configure:23533: error: cannot run test program while cross compiling

On 01/06/2015 02:47 PM, Williams Jr., Ernest L. wrote:

Let us say you do the compiling in machine A.
Can you “remote execute” the generated a.out in machine B?

========= ELW ===============
The answer is no.
This is because machine B is a powerPC and machine A is an INTEL x86 CPU.
Again, I need a cross compiler.

Right, you are cross compiling on the host (or you would simply build
directly on the target). How do you get exectuables from host to
target? One way someone could do such a thing (maybe not your way?) is
to ssh:

--8<------
!/bin/sh
scp a.out target:
ssh target a.out
--8<------

What Albert suggested was you call this script above 'blaunch', and it's
the way some folks like to configure on Blue Gene or one of the old ASCI
machines (I forget which color...)

now, if executing on the target means "i have to burn an image to flash
and boot the board", well, it gets more difficult.

==rob

If so, let us say it is called “Blaunch”.
You can try these steps in a fresh hdf5 source directory.

1. cp configure Bconfigure # just to keep configure original in case
2. env RUNSERIAL=“Blaunch” bin/yodconfigure Bconfigure
3. env RUNSERIAL=“Blaunch” Bconfigure
4. env RUNSERIAL=“Blaunch” gmake # you can use make if it is gmake compatible

Let me know if they work for you.

-Albert Cheng
THG staff

On Jan 6, 2015, at 12:41 PM, Ernest L Williams <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/06/2015 09:28 AM, Albert Cheng wrote:
Hi Ernest,

Let me see if I understand your need correctly. You would like to:
    configure and compile the HDF5 library in machine A but the binary generated does not run in machine A but runs in platform B.
Is that right?
That is correct but there is another caveat.
Machine A and machine B are different architectures.
If we were talking about java code then I would not have an issue.

However, we are talking C-code.

Machine A for example is INTEL x86
Machine B is PowerPC

I need to perform a cross compile.

I don't have a problem cross compiling the SZIP package.
However, I do have the problem compiling the HDF5 package.

Since, your group use the GNU autotools for building packages and support for cross compiling should be achievable. If the hdf5 code has been written with portability in mind, this would make supporting cross compiling easier as well.

Do you have a means to launch binary generated in machine A to run in platform B? E.g.,
machine A $ Bcc program.c -o a.out
machine A $ Blaunch ./a.out
and Blaunch passes back the exit code of ./a.out run in platform B.

If so, we already have a means to do that for HDF5 library, using $RUNSERIAL and yodconfigure.
This is created to handle the case of building and running HDF5 software in the compute clusters
environments in which binary are generated in a frontend machine and executed in the backend
compute nodes. You can see details of it in release_docs/INSTALL_parallel, section
2.4.1 Building serial HDF5 for Red Storm

For my particular application; I am not using compute clusters.

Cheers,
Ernest

Try it out to see if it meets your need.

-Albert Cheng
THG staff

On Jan 5, 2015, at 12:41 PM, Ernest L. Williams Jr. <ernesto@slac.stanford.edu<mailto:ernesto@slac.stanford.edu>> wrote:

On 01/05/2015 10:29 AM, Elena Pourmal wrote:

Hi Rob,

On Jan 5, 2015, at 9:10 AM, Rob Latham <robl@mcs.anl.gov><mailto:robl@mcs.anl.gov> wrote:

On 01/01/2015 10:52 PM, Ernest L Williams wrote:

Hi

I am trying to cross-compile hdf5 for a different architecture:
I get the following error in the configuration stage:

configure:23533: error: cannot run test program while cross compiling

HDF5 uses AC_TRY_RUN in way too many places. It makes cross compiling a gigantic pain.

Workaround: set a pile of environment variables. Most of the tests are obvious: checking for example if you are not running on an HP-UX machine with a buggy compiler from 1998 (I think I have exaggerated that slightly... but only slightly)

You are not exaggerating! Our configure needs a major cleanup!

Fortunately, they do not seem to be adding any *new* instances of AC_TRY_RUN, so once you set up all the needed environment variables, you'll be set for the next several release cycles.

I hope other HDF folks will chime in if I am wrong, but the major problem with cross-compiling is h5detect. Configure runs h5detect to generate H5Tinit.c which is platform specific. You may get the wrong H5Tinit.c if h5detect doesn’t run on the right platform (architecture).

The other problem is "H5lib_settings.c"

If these files are really needed, they could be generated with autotools and PERL.

Cheers,
Ernest

Elena

To get you started, here's a shell script (attached) I use with lots of the necessary environment variables set. Richard Hedges got me most of the way a few years back (in the 1.6 days), and so I pay it forward to you. I did this for Blue Gene, so usual caveats apply: I have no idea what the right values should be for your environment.

Oh, note the one odd step: you have to make one c file in normal mode, then re-configure everything for cross compiling.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
<bgq-hdf5build18.sh>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5