It’s been about 10 years since the last time I brought up this topic, but here we go again. Could The HDFGroup give some guidance on their support for the new Apple Silicon (M1, ARM64) architecture? The way I see it, based on the 1.10.x release, is that we will need to compile on each machine separately due to the use of executables during the build process to gather information about the system.
Hi Mike! Hope things are well with you.
Any news on this issue? I just cloned from github master and see this is still there:
message (FATAL_ERROR "Building Universal Binaries on OS X is NOT supported by the HDF5 project.
I also looked through the tickets on github and don’t see any about supporting Universal Binaries. I can create one I guess, but thought I’d ping here first…
Cheers,
Sean
I don’t think you can create universal binaries the “easy” way. I think you are going to have to do 2 separate compiles in 2 separate directories. One for ARM and one for x86-64. Then you “lipo” them together. The interesting part of this is that this can ONLY be achieved on an M1 Mac since the x86 macs can not emulate the ARM macs but the ARM macs can emulate the x86_64 macs. The same was true with the PPC/Intel transition. This just brings back memories…
I was successful in compiling HDF5 1.10.7 on our M1 mac in the office. I didn’t run the full test suite as I was interested in just getting our own software up and running on the M1 Mac.
If the HDFGroup is interested in a build bot or a sanity check I am certainly open to helping that out.
Two separate compiles on two different computers makes automated nightly builds very complicated.
The code has this:
if (APPLE)
list (LENGTH CMAKE_OSX_ARCHITECTURES ARCH_LENGTH)
if (ARCH_LENGTH GREATER 1)
set (CMAKE_OSX_ARCHITECTURES "" CACHE STRING "" FORCE)
message (FATAL_ERROR "Building Universal Binaries on OS X is NOT supported by the HDF5 project. This is"
"due to technical reasons. The best approach would be build each architecture in separate directories"
"and use the 'lipo' tool to combine them into a single executable or library. The 'CMAKE_OSX_ARCHITECTURES'"
"variable has been set to a blank value which will build the default architecture for this system.")
endif ()
set (${HDF_PREFIX}_AC_APPLE_UNIVERSAL_BUILD 0)
endif ()
Any idea what the ‘technical reasons’ are? Presumably the building machine is introspected for things like endianness and pointer size?
Unlike previously, with ppc+ppc64+i386+x86_64 universal binaries, the situation is actually much simpler today between x86_64 and arm64:
- endianness is the same
- pointer and int size is the same
If I simply remove the if (ARCH_LENGTH GREATER 1) check above, I’m able to build as universal (on intel host) and all tests pass (on intel). But I don’t have an ARM Mac to test the result with.
Mike, I recall you received an ARM Mac? Could you try building with CMAKE_OSX_ARCHITECTURES=x86_64? Since that’s only one architecture, it will pass that GREATER 1 test. I’d be curious to see if all tests pass under Rosetta.
My ARM Mac is due to finally arrive in 4 days, despite being ordered months ago!
Sean
Hi Mike and Sean,
Sorry for my slow response. We just got our ARM system and are setting it up for testing. As I recall from my previous attempt to create universal binaries, I had to use two different builds and then ran a command to combine them.
We need to be careful and make sure that the installed header files are the same. We will look into the problem when we have our system up and running.
Thank you!
Elena
Sean,
Pretty sure that is how I got it to build by setting the CMAKE_OSX_ARCHITECTURES to x86_64 (on intel) or by ARM64 on the M1 mac. Those build and pass. As Elena alluded to earlier a complete review of the generated headers would be needed to ensure that they are effectively the same. Does the build system for HDF5 still create an executable to figure out certain aspects of a system? That would still need to be gotten rid of otherwise how would I build for ARM64 on an Intel based Mac?
Elena, do you recall why you “had” to build twice and then combine them?
As it happens, we got our ARM Mac yesterday. I’ve built HDF5 develop branch with default settings as x84_64 only, arm64 only, and universal. The only header file with any differences at all is H5Epubgen.h, and it’s different in all 3 cases. It looks like contents just moved around. It’s basically all just #define statements. If I sort the files, then all 3 are identical. So I think all is well with headers.
As for the tests: 3 fail under Rosetta and a different one fails running as native arm. Didn’t investigate deeply…
Sean
Sean,
Header files were different, but I don’t remember now. We provide examples with HDF5 installations, have you tested them?
Elena
Perhaps previous header differences were due to different pointer size and endianness, something that’s not a problem this time between x86_64 and arm64.
I tried the unit tests, but not other examples. I built with HDF5_BUILD_EXAMPLES=ON, is that the examples you mean?
Yes, I believe it was 32vs64 builds and yes to your question about examples build.
I tried to compile HDF5 1.8.22 on my M1 mac. but failed. I try to upload the log files but not allowed as a new comer. basically, it says a lots of files are missing in the build/bin folders, including: h52gif-shared/h5diff-shared/filter_plugin/h5importtest/…
I also found a PreinstallOutput.log file in the build/_CPack_Packages/Darwin/TGZ/ folder, says:
Run command: /opt/homebrew/Cellar/cmake/3.21.3/bin/cmake --build . --config “RelWithDebInfo” --target “preinstall”
Directory: /Users/twister/temp/IfcOpenShell/build/Darwin/arm64/build/CMake-hdf5-1.8.22/build
Output:
[ 0%] Built target SZIP
[ 0%] Performing build step for ‘ZLIB’
Consolidate compiler generated dependencies of target zlib-static
[ 6%] Building C object CMakeFiles/zlib-static.dir/gzlib.c.o
/Users/twister/temp/IfcOpenShell/build/Darwin/arm64/build/CMake-hdf5-1.8.22/build/ZLIB-prefix/src/ZLIB/gzlib.c:214:15: error: implicitly declaring library function ‘snprintf’ with type ‘int (char *, unsigned long, const char *, …)’ [-Werror,-Wimplicit-function-declaration]
(void)snprintf(state->path, len + 1, “%s”, (const char *)path);
^
/Users/twister/temp/IfcOpenShell/build/Darwin/arm64/build/CMake-hdf5-1.8.22/build/ZLIB-prefix/src/ZLIB/gzlib.c:214:15: note: include the header <stdio.h> or explicitly provide a declaration for ‘snprintf’
1 error generated.
make[5]: *** [CMakeFiles/zlib-static.dir/gzlib.c.o] Error 1
make[4]: *** [CMakeFiles/zlib-static.dir/all] Error 2
make[3]: *** [all] Error 2
make[2]: *** [ZLIB-prefix/src/ZLIB-stamp/ZLIB-build] Error 2
make[1]: *** [CMakeFiles/ZLIB.dir/all] Error 2
make: *** [all] Error 2
I don’t know if this is the root of failures. anyone can help? thanks in advance.
m1 support has just recently been added to the codebase. I believe it made it into the 1.8 branch, so you might try that first. Do you need the szip/zlib libraries built? We are currently working on some updates to address those libs (Still in process). If they are already built and installed on your machine - use the alternative CMake option to point the build to them.
Allen
Neo,
I’ve built for arm64 Mac and not seen those problems using the tip of the 1.8 branch.
One issue still remaining is compile-time introspection that breaks Universal Binary builds, see:
https://github.com/HDFGroup/hdf5/issues/566
Sean
Thank you, Sean. I will try the 1.8 branch later. thanks.