Make the Cmake Windows build static please !


#1

Hi Allen, Ward

I have a request regarding your new CMake Windows build system, could you add an option to make the build static regarding the Microsoft libraries (MSVCR100D.dll) ?

Starting with version 4.3.1, NCO

http://nco.sourceforge.net/

uses the HDF5 and netCDF Windows libraries made with your CMake system, and this is causing problems for NCO users, as explained here

https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151

and here

https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103

This is just a matter of changing the compiler flag to /MT(d)

http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx

Using a dynamic build is just a bad idea, because of these DLL issues.

I have some Windows executables from code I did in the early 90's , that unfortunately I cannot run today,
just because I linked them with DLLs, with the DLLs from the Visual Studio from that time, that do not exist anymore (it seems every new version they change the Visual Studio Dlls).

Because of this I do not use Dlls, I know that eventually something will go wrong :slight_smile:

Pedro

···

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/


#2

Funny, I actually _prefer_ the DLL builds because I distribute the runtime C/C++ libraries as allowed by MS. Depending on how one is using the HDF5 executables/libraries having each library linked statically against their own C/C++ libraries can also lead to problems because of how memory is allocated/deallocated in each library version. There are 2 evils here and the idea is to pick the least of them. If anything I would petition the HDFGroup to provide BOTH a dynamically linked AND a statically linked runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project precisely because of issues like this.

···

___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:

Hi Allen, Ward

I have a request regarding your new CMake Windows build system, could you add an option to make the build static regarding the Microsoft libraries (MSVCR100D.dll) ?

Starting with version 4.3.1, NCO

http://nco.sourceforge.net/

uses the HDF5 and netCDF Windows libraries made with your CMake system, and this is causing problems for NCO users, as explained here

https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151

and here

https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103

This is just a matter of changing the compiler flag to /MT(d)

http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx

Using a dynamic build is just a bad idea, because of these DLL issues.

I have some Windows executables from code I did in the early 90's , that unfortunately I cannot run today,
just because I linked them with DLLs, with the DLLs from the Visual Studio from that time, that do not exist anymore (it seems every new version they change the Visual Studio Dlls).

Because of this I do not use Dlls, I know that eventually something will go wrong :slight_smile:

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#3

Hi Ward, Allen

Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration.

Ward, this seems not to be working with the current release candidate netcdf-4.3.1-rc2 (it was working for netcdf-4.3.0)

I tried

cd build

cmake -C ../cmake/ConfigUser.cmake -D"BUILD_SHARED_LIBS=OFF" -D"ENABLE_DLL=OFF" -D"HDF5_INCLUDE_DIR=I:\hdf5-1.8.11\src" -D"HDF5_LIB=I:\hdf5-1.8.11\build\bin\Debug\hdf5_D.lib" -D"HDF5_HL_LIB=I:\hdf5-1.8.11\build\bin\Debug\hdf5_hl_D.lib" -D"ZLIB_INCLUDE_DIR:PATH=I:\zlib-1.2.5" -D"ZLIB_LIBRARY:FILEPATH=I:\zlib-1.2.5\build\bin\Debug\libzlib_D.lib" -D"SZIP_INCLUDE_DIR:PATH=I:\szip-2.1\src" -D"SZIP_LIBRARY:FILEPATH=I:\szip-2.1\build\bin\Debug\libszip_D.lib" -D"ENABLE_DAP=ON" -D"CURL_LIBRARY=J:\curl-7.29.0\builds\libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl\lib\libcurl_a_debug.lib" -D"CURL_INCLUDE_DIR=J:\curl-7.29.0\builds\libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl\include" ..

and the generated projects were all runtime library Multi-threaded Debug DLL (/MDd)

Allen,
on the other hand, I managed to make the generated projects by HDF5 CMake all runtime library Multi-threaded Debug (/MTd)

by using this post

http://www.cmake.org/pipermail/cmake/2006-July/010121.html

and using the code in CMakeLists.txt

I tried the same code in CMakeLists.txt of netCDF, but it did not make the substitution from /MD to /MT

Is the next HDF5 version going to include the option -D"BUILD_SHARED_LIBS=OFF", like netCDF does now ?

Pedro

···

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

  ----- Original Message -----
  From: Ward Fisher
  To: Pedro Vicente
  Cc: netcdfgroup@unidata.ucar.edu ; Allen D Byrne
  Sent: Wednesday, June 05, 2013 9:36 AM
  Subject: Re: [netcdfgroup] Make the Cmake Windows build static please !

  Good morning Pedro,

  Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration. This will build the netcdf libraries and utilities statically, avoiding the direct dependency on MSVCR100.dll. They will, however, still inherit any downstream .dll dependencies from the curl, hdf and zlib libraries. You might be able to work around this by downloading (or building) static versions of these libraries. However, when I first investigated this process some months ago, it became obvious that this would require more time and effort on our end than we can reasonably expend (particularly for 64-bit versions of the libraries). Hence, we only provide shared libraries pre-built :).

  On a tangental note, I see from the NCO discussion you linked that your user was able to resolve the issue by removing the MSVCR100D.dll from the c:\nco\ directory. On Windows, it is preferable to use the Release version of the netcdf libraries (dependent on MSVCR100.dll) for the time being, due to cross-dll memory management situations which arise between the netcdf and hdf libraries. This is a known issue which can be followed in our JIRA system at https://bugtracking.unidata.ucar.edu/browse/NCF-220 . It is possible to build and use the debug libraries, but running the unit tests will result in a handful of errors.

  I hope this clarifies the difficulty faced in providing purely static libraries for netcdf-4/DAP enabled builds; as always, I am open to suggestions and work-arounds!

  Have a great day,

  -Ward

  On 6/5/13 12:24 AM, Pedro Vicente wrote:

    Hi Allen, Ward

    I have a request regarding your new CMake Windows build system, could you add an option to make the build static regarding the Microsoft libraries (MSVCR100D.dll) ?

    Starting with version 4.3.1, NCO

    http://nco.sourceforge.net/

    uses the HDF5 and netCDF Windows libraries made with your CMake system, and this is causing problems for NCO users, as explained here

    https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151

    and here

    https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103

    This is just a matter of changing the compiler flag to /MT(d)

    http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx

    Using a dynamic build is just a bad idea, because of these DLL issues.

    I have some Windows executables from code I did in the early 90's , that unfortunately I cannot run today,
    just because I linked them with DLLs, with the DLLs from the Visual Studio from that time, that do not exist anymore (it seems every new version they change the Visual Studio Dlls).
     
    Because of this I do not use Dlls, I know that eventually something will go wrong :slight_smile:

    Pedro

    ------
    Pedro Vicente, Earth System Science
    University of California, Irvine
    http://www.ess.uci.edu/

------------------------------------------------------------------------------

  _______________________________________________
  netcdfgroup mailing list
  netcdfgroup@unidata.ucar.edu
  For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/


#4

Our binaries redistribute the MS CRT dlls that are used to build the binaries.
Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

Hopefully, as more people use CMake and build a common knowledge of using
CMake, those wishing to build alternative versions of HDF will share their
code changes.

Allen

···

On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:

Funny, I actually _prefer_ the DLL builds because I distribute the runtime
C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
executables/libraries having each library linked statically against their
own C/C++ libraries can also lead to problems because of how memory is
allocated/deallocated in each library version. There are 2 evils here and
the idea is to pick the least of them. If anything I would petition the
HDFGroup to provide BOTH a dynamically linked AND a statically linked
runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project precisely
because of issues like this.
___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
> Hi Allen, Ward
>
> I have a request regarding your new CMake Windows build system, could you
> add an option to make the build static regarding the Microsoft libraries
> (MSVCR100D.dll) ?
>
> Starting with version 4.3.1, NCO
>
> http://nco.sourceforge.net/
>
> uses the HDF5 and netCDF Windows libraries made with your CMake system,
> and this is causing problems for NCO users, as explained here
>
> https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>
> and here
>
> https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>
>
> This is just a matter of changing the compiler flag to /MT(d)
>
> http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>
> Using a dynamic build is just a bad idea, because of these DLL issues.
>
> I have some Windows executables from code I did in the early 90's , that
> unfortunately I cannot run today, just because I linked them with DLLs,
> with the DLLs from the Visual Studio from that time, that do not exist
> anymore (it seems every new version they change the Visual Studio Dlls).
>
> Because of this I do not use Dlls, I know that eventually something will
> go wrong :slight_smile:
>
> Pedro
>
> ------
> Pedro Vicente, Earth System Science
> University of California, Irvine
> http://www.ess.uci.edu/
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#5

Hey , Allen & Ward, next time I send an email to *both* of your group lists, can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the answers.

Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration.
This will build the netcdf libraries and utilities statically, avoiding the direct dependency on MSVCR100.dll.
They will, however, still inherit any downstream .dll dependencies from the curl, hdf and zlib libraries.

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

And, yes, the solution is to build *all* downstream code in the *same* way, either all static or all dynamic;
mixing both will get you linking errors.

You might be able to work around this by downloading (or building) static versions of these libraries.

Yes, Ward, this can be done by editing the HDF Cmake generated VS projects, and changing all of them to static, which I just did.

In the curl case, you can do this by editing the file MakefileBuild.vc and change all /MD (dynamic) to /MT (static)

Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd

Our binaries redistribute the MS CRT dlls that are used to build the binaries.

Hey, Allen, I was not asking for the *binaries* !

I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them *already* "static ready" if they wish...

So that, next time I need to generate everything from scratch I don't have to go each to each one of them (50 + projects ) and click the static button 50 times ...

Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

read this

quote

" i personally prefer statically linked. i hate scrambling around looking for the right redist/dll/etc."

that's what I do, I set to all static , and then move on to do more interesting things, like writing software, than dealing with DLL idiosyncracies.

quote

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up with multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT would get its own heap, "

Was this the problem you meant ?

This might be true if you are distributing *binaries* with both the EXE and DLL. Or if you are linking your code against a 3rd party library *without* the code,
someone gave you a LIB and a DLL only.

In the HDF, netCDF and NCO worlds none of this is true: all sources are available , no secrets here :slight_smile:

And you are lucky, Allen , because you only have the downstream ZLIB, and netCDF only has curl

Here's a list of all NCO dependencies

zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :slight_smile:
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )

I have Visual Studio projects for all these (except UDUnits and Regular Expressions)
, because I need to build the *source* , as you can see many projects to change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

Pedro

···

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: "Allen Byrne" <byrn@hdfgroup.org>
To: <hdf-forum@lists.hdfgroup.org>
Sent: Wednesday, June 05, 2013 6:18 AM
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

Our binaries redistribute the MS CRT dlls that are used to build the binaries.
Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

Hopefully, as more people use CMake and build a common knowledge of using
CMake, those wishing to build alternative versions of HDF will share their
code changes.

Allen

On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:

Funny, I actually _prefer_ the DLL builds because I distribute the runtime
C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
executables/libraries having each library linked statically against their
own C/C++ libraries can also lead to problems because of how memory is
allocated/deallocated in each library version. There are 2 evils here and
the idea is to pick the least of them. If anything I would petition the
HDFGroup to provide BOTH a dynamically linked AND a statically linked
runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project precisely
because of issues like this.
___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
> Hi Allen, Ward
>
> I have a request regarding your new CMake Windows build system, could you
> add an option to make the build static regarding the Microsoft libraries
> (MSVCR100D.dll) ?
>
> Starting with version 4.3.1, NCO
>
> http://nco.sourceforge.net/
>
> uses the HDF5 and netCDF Windows libraries made with your CMake system,
> and this is causing problems for NCO users, as explained here
>
> https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>
> and here
>
> https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>
>
> This is just a matter of changing the compiler flag to /MT(d)
>
> http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>
> Using a dynamic build is just a bad idea, because of these DLL issues.
>
> I have some Windows executables from code I did in the early 90's , that
> unfortunately I cannot run today, just because I linked them with DLLs,
> with the DLLs from the Visual Studio from that time, that do not exist
> anymore (it seems every new version they change the Visual Studio Dlls).
>
> Because of this I do not use Dlls, I know that eventually something will
> go wrong :slight_smile:
>
> Pedro
>
> ------
> Pedro Vicente, Earth System Science
> University of California, Irvine
> http://www.ess.uci.edu/
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#6

Hi Ward

Can you be a little more specific about how this does not work?
Is the issue that the generated projects are configured to use the the dynamic instead of static CRT?

Yes, I meant that the the generated projects were all runtime library Multi-threaded Debug DLL (/MDd).

I was under the impression that if I do -D"BUILD_SHARED_LIBS=OFF", then the projects would be static CRT ?
Or is this option, just relative to the netCDF library itself, netcdf.lib ? In this case, my bad, sorry for the confusion.

I see how this could be set as a user-configurable option when invoking cmake. I will try to add this in and, assuming there aren't any problems caused by using the static CRT, it will be an option available in the >>upcoming netCDF-4.3.1-RC3.

OK, great, thanks

Allen, maybe the HD5 Cmake could use the same ?

Pedro

···

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

J:\netcdf-c-4.3.1-rc2>cd build

J:\netcdf-c-4.3.1-rc2\build>cmake -C ../cmake/ConfigUser.cmake -D"BUILD_SHARED_LIBS=OFF" -D"ENABLE_DLL=OFF" -D"HDF5_INCLUDE_DIR=I:\hdf5-1.8.11\src" -D"HDF5_LIB=I:\hdf5-1.8.11\build
\bin\Debug\hdf5_D.lib" -D"HDF5_HL_LIB=I:\hdf5-1.8.11\build\bin\Debug\hdf5_hl_D.lib" -D"ZLIB_INCLUDE_DIR:PATH=I:\zlib-1.2.5" -D"ZLIB_LIBRARY:FILEPATH=I:\zlib-1.2.5\build\bin\Debug\l
ibzlib_D.lib" -D"SZIP_INCLUDE_DIR:PATH=I:\szip-2.1\src" -D"SZIP_LIBRARY:FILEPATH=I:\szip-2.1\build\bin\Debug\libszip_D.lib" -D"ENABLE_DAP=ON" -D"CURL_LIBRARY=J:\curl-7.29.0\builds
\libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl\lib\libcurl_a_debug.lib" -D"CURL_INCLUDE_DIR=J:\curl-7.29.0\builds\libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl\include" ..loading initial cache file ../cmake/ConfigUser.cmake-- Building for: Visual Studio 10-- The C compiler identification is MSVC 16.0.30319.1-- Check for working C compiler using: Visual Studio 10-- Check for working C compiler using: Visual Studio 10 -- works-- Detecting C compiler ABI info-- Detecting C compiler ABI info - done-- Found ZLIB: I:/zlib-1.2.5/build/bin/Debug/libzlib_D.lib (found version"1.2.5")-- Found CURL:J:/curl-7.29.0/builds/libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl/lib/libcurl_a_debug.lib (found version "7.29.0")-- Looking for math.h-- Looking for math.h - found-- Looking for unistd.h-- Looking for unistd.h - not found-- Looking for alloca.h-- Looking for alloca.h - not found-- Looking for malloc.h-- Looking for malloc.h - found-- Looking for ctype.h-- Looking for ctype.h - found-- Looking for dirent.h-- Looking for dirent.h - not found-- !
Looking for dlfcn.h-- Looking for dlfcn.h - not found-- Looking for errno.h-- Looking for errno.h - found-- Looking for fcntl.h-- Looking for fcntl.h - found-- Looking for getopt.h-- Looking for getopt.h - not found-- Looking for stdbool.h-- Looking for stdbool.h - not found-- Looking for locale.h-- Looking for locale.h - found-- Looking for stdint.h-- Looking for stdint.h - found-- Looking for stdio.h-- Looking for stdio.h - found-- Looking for stdlib.h-- Looking for stdlib.h - found-- Looking for stdarg.h-- Looking for stdarg.h - found-- Looking for strings.h-- Looking for strings.h - not found-- Looking for signal.h-- Looking for signal.h - found-- Looking for sys/dir.h-- Looking for sys/dir.h - not found-- Looking for sys/ndir.h-- Looking for sys/ndir.h - not found-- Looking for sys/param.h-- Looking for sys/param.h - not found-- Looking for sys/stat.h-- Looking for sys/stat.h - found-- Looking for sys/time.h-- Looking for sys/time.h - not found-- Looking for sys/types.!
h-- Looking for sys/types.h - found-- Looking for sys/wait.h-- Looking
for sys/wait.h - not found-- Looking for sys/resource.h-- Looking for sys/resource.h - not found-- Looking for inttypes.h-- Looking for inttypes.h - not found-- Looking for pstdint.h-- Looking for pstdint.h - found-- Looking for endian.h-- Looking for endian.h - not found-- Looking for BaseTsd.h-- Looking for BaseTsd.h - found-- Looking for stddef.h-- Looking for stddef.h - found-- Check size of double-- Check size of double - done-- Check size of float-- Check size of float - done-- Check size of int-- Check size of int - done-- Check size of long-- Check size of long - done-- Check size of long long-- Check size of long long - done-- Check size of off_t-- Check size of off_t - done-- Check size of off64_t-- Check size of off64_t - failed-- Check size of short-- Check size of short - done-- Check size of size_t-- Check size of size_t - done-- Check size of ssize_t-- Check size of ssize_t - failed-- Check size of uchar-- Check size of uchar - failed-- Check size of int64_t-!
- Check size of int64_t - done-- Check size of uint64_t-- Check size of uint64_t - done-- Looking for fsync-- Looking for fsync - not found-- Looking for strlcat-- Looking for strlcat - not found-- Looking for strerror-- Looking for strerror - found-- Looking for snprintf-- Looking for snprintf - not found-- Looking for strchr-- Looking for strchr - found-- Looking for strrchr-- Looking for strrchr - found-- Looking for strcat-- Looking for strcat - found-- Looking for strcpy-- Looking for strcpy - found-- Looking for strdup-- Looking for strdup - found-- Looking for strcasecmp-- Looking for strcasecmp - not found-- Looking for strtod-- Looking for strtod - found-- Looking for strtoll-- Looking for strtoll - not found-- Looking for strtoull-- Looking for strtoull - not found-- Looking for strstr-- Looking for strstr - found-- Looking for mkstemp-- Looking for mkstemp - not found-- Looking for rand-- Looking for rand - found-- Looking for gettimeofday-- Looking for gettimeof!
day - not found-- Looking for MPI_Comm_f2C-- Looking for MPI_Comm_f2C - not found-- Looking for memmove-- Looking for memmove - found-- Looking for getpagesize-- Looking for getpagesize - not found-- Looking for sysconf-- Looking for sysconf - not found-- Looking for mremap-- Looking for mremap - not found-- Looking for getrlimit-- Looking for getrlimit - not found-- CMake Summary:-- Build Type: DEBUG-- Building Shared Libraries: OFF-- Building netCDF-4: ON-- Building DAP Support: ON-- Building Utilities: ON-- Using pnetcdf: OFF-- Using Parallel IO: OFF-- Linking against:I:\hdf5-1.8.11\build\bin\Debug\hdf5_D.lib;I:\hdf5-1.8.11\build\bin\Debug\hdf5_hl_D.lib;I:/zlib-1.2.5/build/bin/Debug/libzlib_D.lib;I:/szip-2.1/build/bin/Debug/libszip_D.lib;J:/curl-7.29.0/builds/libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl/lib/libcurl_a_debug.lib-- Configuring done-- Generating doneCMake Warning: Manually-specified variabl!
es were not used by the project: SZIP_INCLUDE_DIR----- Original Message -----From: Ward FisherTo: Pedro VicenteCc: Allen D Byrne ; netcdfgroup@unidata.ucar.eduSent: Wednesday, September 18, 2013 11:46 AMSubject: Re: [netcdfgroup] Make the Cmake Windows build static please !Hello Pedro,On 9/17/13 10:10 PM, Pedro Vicente wrote:Hi Ward, Allen>>> Building NetCDF statically is already an option, bypassing -DBUILD_SHARED_LIBS=OFF during configuration.Ward, this seems not to be working with the current release candidatenetcdf-4.3.1-rc2 (it was working for netcdf-4.3.0)I triedcd buildcmake -C../cmake/ConfigUser.cmake -D"BUILD_SHARED_LIBS=OFF" -D"ENABLE_DLL=OFF" -D"HDF5_INCLUDE_DIR=I:\hdf5-1.8.11\src" -D"HDF5_LIB=I:\hdf5-1.8.11\build\bin\Debug\hdf5_D.lib" -D"HDF5_HL_LIB=I:\hdf5-1.8.11\build\bin\Debug\hdf5_hl_D.lib" -D"ZLIB_INCLUDE_DIR:PATH=I:\zlib-1.2.5" -D"ZLIB_LIBRARY:FILEPATH=I:\zlib-1.2.5\build\bin\Debug\libzlib_D.lib" -D"SZIP_INCLUDE_DIR:PATH=I:\szip-2.1\src" -D"SZIP_LIBRARY!
:FILEPATH=I:\szip-2.1\build\bin\Debug\libszip_D.lib" -D"ENABLE_DAP=ON"
  -D"CURL_LIBRARY=J:\curl-7.29.0\builds\libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl\lib\libcurl_a_debug.lib" -D"CURL_INCLUDE_DIR=J:\curl-7.29.0\builds\libcurl-vc10-x86-debug-static-ipv6-sspi-spnego-winssl\include" ..and the generated projects were all runtime library Multi-threaded Debug DLL(/MDd)Can you be a little more specific about how this does not work? I was ableto build the netCDF-C libraries statically using the latest developersnapshot and Visual Studio 10, with the resultant netcdf.lib file being aproper 'ar' archive. All tests ran properly.Is the issue that the generated projects are configured to use the thedynamic instead of static CRT? This is as it was in the netcdf-4.3.0libraries, so if that was working for you I suspect this will as well.However, looking at the relevant CMake documentation,http://www.cmake.org/Wiki/CMake_FAQ#How_can_I_build_my_MSVC_application_with_a_static_runtime.3FI see how this could be set as a user-configurable option whe!
n invokingcmake. I will try to add this in and, assuming there aren't any problemscaused by using the static CRT, it will be an option available in theupcoming netCDF-4.3.1-RC3. In the meantime, you can replace '/MD' with'/MT' in the cache editor.If I have misunderstood the issue, or if you are seeing an outrighterror/unexpected behavior please let me know. Have a great day,-WardAllen,on the other hand, I managed to make the generated projects by HDF5 CMakeall runtime library Multi-threaded Debug (/MTd)by using this posthttp://www.cmake.org/pipermail/cmake/2006-July/010121.htmland using the code in CMakeLists.txtI tried the same code in CMakeLists.txt of netCDF, but it did not make thesubstitution from /MD to /MTIs the next HDF5 version going to include theption -D"BUILD_SHARED_LIBS=OFF", like netCDF does now ?Pedro------------pvicente@uci.eduDepartment of Earth System Science3200 Croul HallUniversity of California, IrvineIrvine, CA 92697-3100----- Original Message ----!
-From: Ward FisherTo: Pedro VicenteCc: netcdfgroup@unidata.ucar.edu ; Allen D ByrneSent: Wednesday, June 05, 2013 9:36 AMSubject: Re: [netcdfgroup] Make the Cmake Windows build static please !Good morning Pedro,Building NetCDF statically is already an option, bypassing -DBUILD_SHARED_LIBS=OFF during configuration. This will build thenetcdf libraries and utilities statically, avoiding the direct dependency onMSVCR100.dll. They will, however, still inherit any downstream .dlldependencies from the curl, hdf and zlib libraries. You might be able towork around this by downloading (or building) static versions of theselibraries. However, when I first investigated this process some months ago,it became obvious that this would require more time and effort on our endthan we can reasonably expend (particularly for 64-bit versions of thelibraries). Hence, we only provide shared libraries pre-built :).On a tangental note, I see from the NCO discussion you linked that your userwas able to resolve the issue by removing the MSVCR100D.dll from the c:\nco!
\directory. On Windows, it is preferable to use the Release version of thenetcdf libraries (dependent on MSVCR100.dll) for the time being, due tocross-dll memory management situations which arise between the netcdf andhdf libraries. This is a known issue which can be followed in our JIRAsystem at https://bugtracking.unidata.ucar.edu/browse/NCF-220 . It ispossible to build and use the debug libraries, but running the unit testswill result in a handful of errors.I hope this clarifies the difficulty faced in providing purely staticlibraries for netcdf-4/DAP enabled builds; as always, I am open tosuggestions and work-arounds!Have a great day,-WardOn 6/5/13 12:24 AM, Pedro Vicente wrote:Hi Allen, WardI have a request regarding your new CMake Windows build system, could youadd an option to make the build static regarding the Microsoft libraries(MSVCR100D.dll) ?Starting with version 4.3.1, NCOhttp://nco.sourceforge.net/uses the HDF5 and netCDF Windows libraries made with your C!
Make system, andthis is causing problems for NCO users, as explained h
erehttps://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151and herehttps://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103This is just a matter of changing the compiler flag to /MT(d)http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspxUsing a dynamic build is just a bad idea, because of these DLL issues.I have some Windows executables from code I did in the early 90's , thatunfortunately I cannot run today,just because I linked them with DLLs, with the DLLs from the Visual Studiofrom that time, that do not exist anymore (it seems every new version theychange the Visual Studio Dlls).Because of this I do not use Dlls, I know that eventually something will gowrong :-)Pedro------Pedro Vicente, Earth System ScienceUniversity of California, Irvinehttp://www.ess.uci.edu/_______________________________________________netcdfgroup mailing listnetcdfgroup@unidata.ucar.eduFor list information or to unsubscribe, visit:http://www.unidata.ucar.edu/mailing_lists/


#7

+1 for " i personally prefer statically linked. i hate scrambling
around looking for the right redist/dll/etc."

I'd call it hell [1] for novice users like me. :slight_smile:

[1] http://en.wikipedia.org/wiki/DLL_Hell

···

--
HDF: Software that Powers Science

On Thu, Jun 6, 2013 at 3:42 AM, Pedro Vicente <pvicente@uci.edu> wrote:

Hey , Allen & Ward, next time I send an email to *both* of your group lists,
can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the
answers.

Building NetCDF statically is already an option, by passing
-DBUILD_SHARED_LIBS=OFF during configuration.
This will build the netcdf libraries and utilities statically, avoiding
the direct dependency on MSVCR100.dll.
They will, however, still inherit any downstream .dll dependencies from
the curl, hdf and zlib libraries.

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

And, yes, the solution is to build *all* downstream code in the *same* way,
either all static or all dynamic;
mixing both will get you linking errors.

You might be able to work around this by downloading (or building)
static versions of these libraries.

Yes, Ward, this can be done by editing the HDF Cmake generated VS projects,
and changing all of them to static, which I just did.

In the curl case, you can do this by editing the file MakefileBuild.vc and
change all /MD (dynamic) to /MT (static)

Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd

Our binaries redistribute the MS CRT dlls that are used to build the
binaries.

Hey, Allen, I was not asking for the *binaries* !

I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them *already*
"static ready" if they wish...

So that, next time I need to generate everything from scratch I don't have
to go each to each one of them (50 + projects ) and click the static button
50 times ...

Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

read this

http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking

quote

" i personally prefer statically linked. i hate scrambling around looking
for the right redist/dll/etc."

that's what I do, I set to all static , and then move on to do more
interesting things, like writing software, than dealing with DLL
idiosyncracies.

quote

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up with
multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT
would get its own heap, "

Was this the problem you meant ?

This might be true if you are distributing *binaries* with both the EXE and
DLL. Or if you are linking your code against a 3rd party library *without*
the code,
someone gave you a LIB and a DLL only.

In the HDF, netCDF and NCO worlds none of this is true: all sources are
available , no secrets here :slight_smile:

And you are lucky, Allen , because you only have the downstream ZLIB, and
netCDF only has curl

Here's a list of all NCO dependencies

zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :slight_smile:
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )

I have Visual Studio projects for all these (except UDUnits and Regular
Expressions)
, because I need to build the *source* , as you can see many projects to
change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: "Allen Byrne" <byrn@hdfgroup.org>
To: <hdf-forum@lists.hdfgroup.org>
Sent: Wednesday, June 05, 2013 6:18 AM
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

Our binaries redistribute the MS CRT dlls that are used to build the
binaries.
Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

Hopefully, as more people use CMake and build a common knowledge of using
CMake, those wishing to build alternative versions of HDF will share their
code changes.

Allen

On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:

Funny, I actually _prefer_ the DLL builds because I distribute the
runtime
C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
executables/libraries having each library linked statically against their
own C/C++ libraries can also lead to problems because of how memory is
allocated/deallocated in each library version. There are 2 evils here and
the idea is to pick the least of them. If anything I would petition the
HDFGroup to provide BOTH a dynamically linked AND a statically linked
runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project
precisely
because of issues like this.
___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
> Hi Allen, Ward
>
> I have a request regarding your new CMake Windows build system, could
> you
> add an option to make the build static regarding the Microsoft
> libraries
> (MSVCR100D.dll) ?
>
> Starting with version 4.3.1, NCO
>
> http://nco.sourceforge.net/
>
> uses the HDF5 and netCDF Windows libraries made with your CMake system,
> and this is causing problems for NCO users, as explained here
>
> https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>
> and here
>
> https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>
>
> This is just a matter of changing the compiler flag to /MT(d)
>
> http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>
> Using a dynamic build is just a bad idea, because of these DLL issues.
>
> I have some Windows executables from code I did in the early 90's ,
> that
> unfortunately I cannot run today, just because I linked them with DLLs,
> with the DLLs from the Visual Studio from that time, that do not exist
> anymore (it seems every new version they change the Visual Studio
> Dlls).
>
> Because of this I do not use Dlls, I know that eventually something
> will
> go wrong :slight_smile:
>
> Pedro
>
> ------
> Pedro Vicente, Earth System Science
> University of California, Irvine
> http://www.ess.uci.edu/
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
netcdfgroup mailing list
netcdfgroup@unidata.ucar.edu
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/


#8

All,

Support for /MT(d) was removed from CMake because we do not have resources to support it with the HDF software (including HDF5).

If this is a desired feature, we will need someone to test it on a regular basis (daily with development and release HDF5 branches). If there are volunteers, please contact Allen. We will go from there.

Thank you!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820


(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:

Hey , Allen & Ward, next time I send an email to *both* of your group lists, can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the answers.

>>Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration.
>>This will build the netcdf libraries and utilities statically, avoiding the direct dependency on MSVCR100.dll.
>>They will, however, still inherit any downstream .dll dependencies from the curl, hdf and zlib libraries.

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

And, yes, the solution is to build *all* downstream code in the *same* way, either all static or all dynamic;
mixing both will get you linking errors.

>>> You might be able to work around this by downloading (or building) static versions of these libraries.

Yes, Ward, this can be done by editing the HDF Cmake generated VS projects, and changing all of them to static, which I just did.

In the curl case, you can do this by editing the file MakefileBuild.vc and change all /MD (dynamic) to /MT (static)

Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd

>>Our binaries redistribute the MS CRT dlls that are used to build the binaries.

Hey, Allen, I was not asking for the *binaries* !

I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them *already* "static ready" if they wish...

So that, next time I need to generate everything from scratch I don't have to go each to each one of them (50 + projects ) and click the static button 50 times ...

>>Because of the danger of using different CRTs (HDF lib uses one and user
>>program uses different one) and the possible memory corruption with
>>allocations, we build with /MD and provide the CRT with our binary.

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

read this

http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking

quote

" i personally prefer statically linked. i hate scrambling around looking for the right redist/dll/etc."

that's what I do, I set to all static , and then move on to do more interesting things, like writing software, than dealing with DLL idiosyncracies.

quote

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up with multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT would get its own heap, "

Was this the problem you meant ?

This might be true if you are distributing *binaries* with both the EXE and DLL. Or if you are linking your code against a 3rd party library *without* the code,
someone gave you a LIB and a DLL only.

In the HDF, netCDF and NCO worlds none of this is true: all sources are available , no secrets here :slight_smile:

And you are lucky, Allen , because you only have the downstream ZLIB, and netCDF only has curl

Here's a list of all NCO dependencies

zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :slight_smile:
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )

I have Visual Studio projects for all these (except UDUnits and Regular Expressions)
, because I need to build the *source* , as you can see many projects to change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: "Allen Byrne" <byrn@hdfgroup.org>
To: <hdf-forum@lists.hdfgroup.org>
Sent: Wednesday, June 05, 2013 6:18 AM
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

> Our binaries redistribute the MS CRT dlls that are used to build the binaries.
> Because of the danger of using different CRTs (HDF lib uses one and user
> program uses different one) and the possible memory corruption with
> allocations, we build with /MD and provide the CRT with our binary.
>
> Hopefully, as more people use CMake and build a common knowledge of using
> CMake, those wishing to build alternative versions of HDF will share their
> code changes.
>
> Allen
>
> On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:
>> Funny, I actually _prefer_ the DLL builds because I distribute the runtime
>> C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
>> executables/libraries having each library linked statically against their
>> own C/C++ libraries can also lead to problems because of how memory is
>> allocated/deallocated in each library version. There are 2 evils here and
>> the idea is to pick the least of them. If anything I would petition the
>> HDFGroup to provide BOTH a dynamically linked AND a statically linked
>> runtime version.
>>
>> Just my 2 cents. Then again, I build my own HDF5 for our project precisely
>> because of issues like this.
>> ___________________________________________________________
>> Mike Jackson Principal Software Engineer
>> BlueQuartz Software Dayton, Ohio
>> mike.jackson@bluequartz.net www.bluequartz.net
>>
>> On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
>> > Hi Allen, Ward
>> >
>> > I have a request regarding your new CMake Windows build system, could you
>> > add an option to make the build static regarding the Microsoft libraries
>> > (MSVCR100D.dll) ?
>> >
>> > Starting with version 4.3.1, NCO
>> >
>> > http://nco.sourceforge.net/
>> >
>> > uses the HDF5 and netCDF Windows libraries made with your CMake system,
>> > and this is causing problems for NCO users, as explained here
>> >
>> > https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>> >
>> > and here
>> >
>> > https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>> >
>> >
>> > This is just a matter of changing the compiler flag to /MT(d)
>> >
>> > http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>> >
>> > Using a dynamic build is just a bad idea, because of these DLL issues.
>> >
>> > I have some Windows executables from code I did in the early 90's , that
>> > unfortunately I cannot run today, just because I linked them with DLLs,
>> > with the DLLs from the Visual Studio from that time, that do not exist
>> > anymore (it seems every new version they change the Visual Studio Dlls).
>> >
>> > Because of this I do not use Dlls, I know that eventually something will
>> > go wrong :slight_smile:
>> >
>> > Pedro
>> >
>> > ------
>> > Pedro Vicente, Earth System Science
>> > University of California, Irvine
>> > http://www.ess.uci.edu/
>> >
>> >
>> > _______________________________________________
>> > Hdf-forum is for HDF software users discussion.
>> > Hdf-forum@lists.hdfgroup.org
>> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@lists.hdfgroup.org
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
> _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#9

Hi Ward

Can you be a little more specific about how this does not work?
Is the issue that the generated projects are configured to use the the dynamic instead of static CRT?

Yes, I meant that the the generated projects were all runtime library Multi-threaded Debug DLL (/MDd).

I was under the impression that if I do -D"BUILD_SHARED_LIBS=OFF", then the projects would be static CRT ?
Or is this option, just relative to the netCDF library itself, netcdf.lib ? In this case, my bad, sorry for the confusion.

I'm not sure what you mean by 'the projects would be static CRT'. The file netcdf.lib would be a static library, but not a CRT. The Microsoft C Run-Time Libraries (CRT) are a specific set of system libraries, enumerated here:

http://msdn.microsoft.com/en-us/library/abx4dbyh(v=vs.100).aspx

It is possible to compile the netCDF library as a static object (a standard 'ar' archive) that still has dependencies on shared libraries, such as the hdf5 or libcurl dll's.

In the current system up to netCDF-C 4.3.1-rc2, netCDF was being built statically, with a dependency on the shared CRT libraries. With the changes previously described, it will be possible to (easily) build the netCDF library and utilities statically, with a dependency on the static CRT libraries.

In either case, the netCDF library and utilities are static; the difference is in which library (CRT) was being linked against.

I hope this clarifies things; the new option (NC_USE_STATIC_CRT) is being added and should be in the upcoming 4.3.1-rc3 release.

  -Ward

···

On 9/18/13 1:36 PM, Pedro Vicente wrote:

I see how this could be set as a user-configurable option when invoking cmake. I will try to add this in and, assuming there aren't any problems caused by using the static CRT, it will be an option available in the >>upcoming netCDF-4.3.1-RC3.

OK, great, thanks

Allen, maybe the HD5 Cmake could use the same ?

Pedro

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100


#10

I'd just like to second Elena's comment below. As one of the people responsible for the cmake implementation of hdf5, I had a really hard time with this static/dynamic CRT link issue.
I had set up 4 nightly builds of hdf5, static and dynamic, with and without fortran, and unfortunately, all goes wrong when you start using ifort on windows because of the way the supplied fortran libs are linked (once you link the intel fortran libraries your link errors multiply and you have no control of those libraries so having an option to change static/dynamic is no help!)
So there was not a simple combination that was easy for all setups and we basically opted to remove most of the options to get a combination that worked - It took up so much of my time that I didn't 'finish the job cleanly' and allow the user to decide for all cases.

I would concur that

1) Adding a version stamp to the dll/lib names is desirable (and easy to do)

2) Adding a simple static/dynamic run time link option (and easy to do)

I'm happy to test patches if people create them, but all my stuff works fine, so I'm not volunteering!

JB

[I would also personally favour dynamic linking in 99% of cases. When you have plugin based architectures where objects are created and destroyed in different modules, static linking is a no go. I do not suffer from DLL hell and actually find it much much much easier to distribute binaries to people on windows than linux]

···

--
John Biddiscombe, email:biddisco @.at.@ cscs.ch


CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07
Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
Sent: 06 June 2013 17:01
To: HDF Users Discussion List
Cc: help@hdfgroup.org; netcdfgroup@unidata.ucar.edu
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

All,

Support for /MT(d) was removed from CMake because we do not have resources to support it with the HDF software (including HDF5).

If this is a desired feature, we will need someone to test it on a regular basis (daily with development and release HDF5 branches). If there are volunteers, please contact Allen. We will go from there.

Thank you!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820
www.hdfgroup.org<http://www.hdfgroup.org>
(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:

Hey , Allen & Ward, next time I send an email to *both* of your group lists, can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the answers.

Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration.
This will build the netcdf libraries and utilities statically, avoiding the direct dependency on MSVCR100.dll.
They will, however, still inherit any downstream .dll dependencies from the curl, hdf and zlib libraries.

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

And, yes, the solution is to build *all* downstream code in the *same* way, either all static or all dynamic;
mixing both will get you linking errors.

You might be able to work around this by downloading (or building) static versions of these libraries.

Yes, Ward, this can be done by editing the HDF Cmake generated VS projects, and changing all of them to static, which I just did.

In the curl case, you can do this by editing the file MakefileBuild.vc and change all /MD (dynamic) to /MT (static)

Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd

Our binaries redistribute the MS CRT dlls that are used to build the binaries.

Hey, Allen, I was not asking for the *binaries* !

I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them *already* "static ready" if they wish...

So that, next time I need to generate everything from scratch I don't have to go each to each one of them (50 + projects ) and click the static button 50 times ...

Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

read this

quote

" i personally prefer statically linked. i hate scrambling around looking for the right redist/dll/etc."

that's what I do, I set to all static , and then move on to do more interesting things, like writing software, than dealing with DLL idiosyncracies.

quote

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up with multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT would get its own heap, "

Was this the problem you meant ?

This might be true if you are distributing *binaries* with both the EXE and DLL. Or if you are linking your code against a 3rd party library *without* the code,
someone gave you a LIB and a DLL only.

In the HDF, netCDF and NCO worlds none of this is true: all sources are available , no secrets here :slight_smile:

And you are lucky, Allen , because you only have the downstream ZLIB, and netCDF only has curl

Here's a list of all NCO dependencies

zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :slight_smile:
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )

I have Visual Studio projects for all these (except UDUnits and Regular Expressions)
, because I need to build the *source* , as you can see many projects to change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: "Allen Byrne" <byrn@hdfgroup.org<mailto:byrn@hdfgroup.org>>
To: <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Sent: Wednesday, June 05, 2013 6:18 AM
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

Our binaries redistribute the MS CRT dlls that are used to build the binaries.
Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

Hopefully, as more people use CMake and build a common knowledge of using
CMake, those wishing to build alternative versions of HDF will share their
code changes.

Allen

On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:

Funny, I actually _prefer_ the DLL builds because I distribute the runtime
C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
executables/libraries having each library linked statically against their
own C/C++ libraries can also lead to problems because of how memory is
allocated/deallocated in each library version. There are 2 evils here and
the idea is to pick the least of them. If anything I would petition the
HDFGroup to provide BOTH a dynamically linked AND a statically linked
runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project precisely
because of issues like this.
___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net<mailto:mike.jackson@bluequartz.net> www.bluequartz.net<http://www.bluequartz.net>

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu<mailto:pvicente@uci.edu>> wrote:
> Hi Allen, Ward
>
> I have a request regarding your new CMake Windows build system, could you
> add an option to make the build static regarding the Microsoft libraries
> (MSVCR100D.dll) ?
>
> Starting with version 4.3.1, NCO
>
> http://nco.sourceforge.net/
>
> uses the HDF5 and netCDF Windows libraries made with your CMake system,
> and this is causing problems for NCO users, as explained here
>
> https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>
> and here
>
> https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>
>
> This is just a matter of changing the compiler flag to /MT(d)
>
> http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>
> Using a dynamic build is just a bad idea, because of these DLL issues.
>
> I have some Windows executables from code I did in the early 90's , that
> unfortunately I cannot run today, just because I linked them with DLLs,
> with the DLLs from the Visual Studio from that time, that do not exist
> anymore (it seems every new version they change the Visual Studio Dlls).
>
> Because of this I do not use Dlls, I know that eventually something will
> go wrong :slight_smile:
>
> Pedro
>
> ------
> Pedro Vicente, Earth System Science
> University of California, Irvine
> http://www.ess.uci.edu/
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#11

We have implemeted this with our "UserOptions" cmake hooks. Feel free to use
the same technique.

"BUILD_SHARED_LIBS=OFF" should only control the library under build, not how
it is linked to CRT libraries. Which is Ward's understanding as well.

Allen

···

On Wednesday, September 18, 2013 02:05:01 PM Ward Fisher wrote:

On 9/18/13 1:36 PM, Pedro Vicente wrote:
> Hi Ward
>
>>> Can you be a little more specific about how this does not work?
>>> Is the issue that the generated projects are configured to use the
>>> the dynamic instead of static CRT?
>
> Yes, I meant that the the generated projects were all runtime library
> Multi-threaded Debug DLL (/MDd).
>
> I was under the impression that if I do -D"BUILD_SHARED_LIBS=OFF",
> then the projects would be static CRT ?
> Or is this option, just relative to the netCDF library itself,
> netcdf.lib ? In this case, my bad, sorry for the confusion.

I'm not sure what you mean by 'the projects would be static CRT'. The
file netcdf.lib would be a static library, but not a CRT. The Microsoft
C Run-Time Libraries (CRT) are a specific set of system libraries,
enumerated here:

http://msdn.microsoft.com/en-us/library/abx4dbyh(v=vs.100).aspx

It is possible to compile the netCDF library as a static object (a
standard 'ar' archive) that still has dependencies on shared libraries,
such as the hdf5 or libcurl dll's.

In the current system up to netCDF-C 4.3.1-rc2, netCDF was being built
statically, with a dependency on the shared CRT libraries. With the
changes previously described, it will be possible to (easily) build the
netCDF library and utilities statically, with a dependency on the static
CRT libraries.

In either case, the netCDF library and utilities are static; the
difference is in which library (CRT) was being linked against.

I hope this clarifies things; the new option (NC_USE_STATIC_CRT) is
being added and should be in the upcoming 4.3.1-rc3 release.

  -Ward

>>> I see how this could be set as a user-configurable option when
>>> invoking cmake. I will try to add this in and, assuming there
>>> aren't any problems caused by using the static CRT, it will be an
>>> option available in the >>upcoming netCDF-4.3.1-RC3.
>
> OK, great, thanks
>
>
> Allen, maybe the HD5 Cmake could use the same ?
>
> Pedro
>
> ------------
> pvicente@uci.edu
> Department of Earth System Science
> 3200 Croul Hall
> University of California, Irvine
> Irvine, CA 92697-3100


#12

Re: Version Stamp on library names. Please, do NOT do this. Boost does
this and trying to get a build system to find those libraries via
wildcard searching is fraught with danger and error prone. My
FindHDF5.cmake file is crazy enough as it is. Adding in the
complication of looking for hdf5_1_8_10 OR blah blah blah is going to
make those Find modules even harder to write.

I do vote for the other idea though (static/dynamic runtime linking
option). I am with John on this, once you "crack the code" on how to
deploy your application with DLL libraries it is not that hard to do
properly with the dynamic runtime libraries.

The original question though was more about the clashing of the
runtime library versions which is why most of the developers that I
meet we all compile our own versions rather than relying on the
vendor's version. There are just too many variables for a vendor to
try to cover to make it realistic.

Just my 2 cents.
Mike Jackson

···

_________________________________________________________
Mike Jackson mike.jackson@bluequartz.net
BlueQuartz Software www.bluequartz.net
Principal Software Engineer Dayton, Ohio

On Fri, Jun 7, 2013 at 10:05 AM, Biddiscombe, John A. <biddisco@cscs.ch> wrote:

I’d just like to second Elena’s comment below. As one of the people
responsible for the cmake implementation of hdf5, I had a really hard time
with this static/dynamic CRT link issue.

I had set up 4 nightly builds of hdf5, static and dynamic, with and without
fortran, and unfortunately, all goes wrong when you start using ifort on
windows because of the way the supplied fortran libs are linked (once you
link the intel fortran libraries your link errors multiply and you have no
control of those libraries so having an option to change static/dynamic is
no help!)

So there was not a simple combination that was easy for all setups and we
basically opted to remove most of the options to get a combination that
worked - It took up so much of my time that I didn’t ‘finish the job
cleanly’ and allow the user to decide for all cases.

I would concur that

1) Adding a version stamp to the dll/lib names is desirable (and easy
to do)

2) Adding a simple static/dynamic run time link option (and easy to do)

I’m happy to test patches if people create them, but all my stuff works
fine, so I’m not volunteering!

JB

[I would also personally favour dynamic linking in 99% of cases. When you
have plugin based architectures where objects are created and destroyed in
different modules, static linking is a no go. I do not suffer from DLL hell
and actually find it much much much easier to distribute binaries to people
on windows than linux]

--

John Biddiscombe, email:biddisco @.at.@ cscs.ch

http://www.cscs.ch/

CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07

Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of
Elena Pourmal
Sent: 06 June 2013 17:01
To: HDF Users Discussion List
Cc: help@hdfgroup.org; netcdfgroup@unidata.ucar.edu

Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

All,

Support for /MT(d) was removed from CMake because we do not have resources
to support it with the HDF software (including HDF5).

If this is a desired feature, we will need someone to test it on a regular
basis (daily with development and release HDF5 branches). If there are
volunteers, please contact Allen. We will go from there.

Thank you!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820
www.hdfgroup.org
(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:

Hey , Allen & Ward, next time I send an email to *both* of your group lists,
can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the
answers.

Building NetCDF statically is already an option, by passing
-DBUILD_SHARED_LIBS=OFF during configuration.
This will build the netcdf libraries and utilities statically, avoiding
the direct dependency on MSVCR100.dll.
They will, however, still inherit any downstream .dll dependencies from
the curl, hdf and zlib libraries.

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

And, yes, the solution is to build *all* downstream code in the *same* way,
either all static or all dynamic;
mixing both will get you linking errors.

You might be able to work around this by downloading (or building)
static versions of these libraries.

Yes, Ward, this can be done by editing the HDF Cmake generated VS projects,
and changing all of them to static, which I just did.

In the curl case, you can do this by editing the file MakefileBuild.vc and
change all /MD (dynamic) to /MT (static)

Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd

Our binaries redistribute the MS CRT dlls that are used to build the
binaries.

Hey, Allen, I was not asking for the *binaries* !

I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them *already*
"static ready" if they wish...

So that, next time I need to generate everything from scratch I don't have
to go each to each one of them (50 + projects ) and click the static button
50 times ...

Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

read this

http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking

quote

" i personally prefer statically linked. i hate scrambling around looking
for the right redist/dll/etc."

that's what I do, I set to all static , and then move on to do more
interesting things, like writing software, than dealing with DLL
idiosyncracies.

quote

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up with
multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT
would get its own heap, "

Was this the problem you meant ?

This might be true if you are distributing *binaries* with both the EXE and
DLL. Or if you are linking your code against a 3rd party library *without*
the code,
someone gave you a LIB and a DLL only.

In the HDF, netCDF and NCO worlds none of this is true: all sources are
available , no secrets here :slight_smile:

And you are lucky, Allen , because you only have the downstream ZLIB, and
netCDF only has curl

Here's a list of all NCO dependencies

zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :slight_smile:
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )

I have Visual Studio projects for all these (except UDUnits and Regular
Expressions)
, because I need to build the *source* , as you can see many projects to
change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----

From: "Allen Byrne" <byrn@hdfgroup.org>

To: <hdf-forum@lists.hdfgroup.org>

Sent: Wednesday, June 05, 2013 6:18 AM

Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

Our binaries redistribute the MS CRT dlls that are used to build the
binaries.
Because of the danger of using different CRTs (HDF lib uses one and user
program uses different one) and the possible memory corruption with
allocations, we build with /MD and provide the CRT with our binary.

Hopefully, as more people use CMake and build a common knowledge of using
CMake, those wishing to build alternative versions of HDF will share their
code changes.

Allen

On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:

Funny, I actually _prefer_ the DLL builds because I distribute the
runtime
C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
executables/libraries having each library linked statically against their
own C/C++ libraries can also lead to problems because of how memory is
allocated/deallocated in each library version. There are 2 evils here and
the idea is to pick the least of them. If anything I would petition the
HDFGroup to provide BOTH a dynamically linked AND a statically linked
runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project
precisely
because of issues like this.
___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
> Hi Allen, Ward
>
> I have a request regarding your new CMake Windows build system, could
> you
> add an option to make the build static regarding the Microsoft
> libraries
> (MSVCR100D.dll) ?
>
> Starting with version 4.3.1, NCO
>
> http://nco.sourceforge.net/
>
> uses the HDF5 and netCDF Windows libraries made with your CMake system,
> and this is causing problems for NCO users, as explained here
>
> https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>
> and here
>
> https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>
>
> This is just a matter of changing the compiler flag to /MT(d)
>
> http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>
> Using a dynamic build is just a bad idea, because of these DLL issues.
>
> I have some Windows executables from code I did in the early 90's ,
> that
> unfortunately I cannot run today, just because I linked them with DLLs,
> with the DLLs from the Visual Studio from that time, that do not exist
> anymore (it seems every new version they change the Visual Studio
> Dlls).
>
> Because of this I do not use Dlls, I know that eventually something
> will
> go wrong :slight_smile:
>
> Pedro
>
> ------
> Pedro Vicente, Earth System Science
> University of California, Irvine
> http://www.ess.uci.edu/
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#13

Hi Elena, John, Ward, Allen, Michael

Support for /MT(d) was removed from CMake because we do not have resources to support it with the HDF software (including HDF5).
If this is a desired feature, we will need someone to test it on a regular basis (daily with development and release HDF5 branches).

I had set up 4 nightly builds of hdf5, static and dynamic, with and
without fortran, and unfortunately, all goes wrong when you start
using ifort on windows because of the way the supplied fortran libs
are linked (once you link the intel fortran libraries your link errors
multiply and you have no control of those libraries so having an
option to change static/dynamic is no help!)
I would concur that
1) Adding a version stamp to the dll/lib names is desirable (and easy to do)
2) Adding a simple static/dynamic run time link option (and easy to do)

OK, so, why not just add now the simplest configurations that are already working, and that Ward has already done for netCDF?

The only minimal configuration that I think is certainly needed for everyone is item 2), the static/dynamic run time link option, the equivalent of GNU automake --enable-static/shared

One of the things that I am interested in is to build a Cmake build system for NCO also, so what do you all think of a common CMake system for HDF, netCDF and NCO?

I mean, things like

1) common syntax options like netCDF's

-DBUILD_SHARED_LIBS=OFF

2) use of environment variables, with the same name for libraries and header files names;
for example, I use on my projects for the location of the curl library, the name

LIB_CURL

and netCDF uses

CURL_LIBRARY

this would make things simpler for anyone that needs to build HDF, netCDF and NCO.

Elena,
if you need to discuss in detail, call me (same number still), or we can set Skype video/audio joint conference calls if needed.

Pedro

···

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

  ----- Original Message -----
  From: Biddiscombe, John A.
  To: HDF Users Discussion List
  Cc: help@hdfgroup.org ; netcdfgroup@unidata.ucar.edu
  Sent: Friday, June 07, 2013 1:05 AM
  Subject: Re: [netcdfgroup] [Hdf-forum] Make the Cmake Windows build staticplease !

  I'd just like to second Elena's comment below. As one of the people responsible for the cmake implementation of hdf5, I had a really hard time with this static/dynamic CRT link issue.

  I had set up 4 nightly builds of hdf5, static and dynamic, with and without fortran, and unfortunately, all goes wrong when you start using ifort on windows because of the way the supplied fortran libs are linked (once you link the intel fortran libraries your link errors multiply and you have no control of those libraries so having an option to change static/dynamic is no help!)

  So there was not a simple combination that was easy for all setups and we basically opted to remove most of the options to get a combination that worked - It took up so much of my time that I didn't 'finish the job cleanly' and allow the user to decide for all cases.

  I would concur that

  1) Adding a version stamp to the dll/lib names is desirable (and easy to do)

  2) Adding a simple static/dynamic run time link option (and easy to do)

  I'm happy to test patches if people create them, but all my stuff works fine, so I'm not volunteering!

  JB

  [I would also personally favour dynamic linking in 99% of cases. When you have plugin based architectures where objects are created and destroyed in different modules, static linking is a no go. I do not suffer from DLL hell and actually find it much much much easier to distribute binaries to people on windows than linux]

  --

  John Biddiscombe, email:biddisco @.at.@ cscs.ch

  http://www.cscs.ch/

  CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07

  Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82

  From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
  Sent: 06 June 2013 17:01
  To: HDF Users Discussion List
  Cc: help@hdfgroup.org; netcdfgroup@unidata.ucar.edu
  Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

  All,

  Support for /MT(d) was removed from CMake because we do not have resources to support it with the HDF software (including HDF5).

  If this is a desired feature, we will need someone to test it on a regular basis (daily with development and release HDF5 branches). If there are volunteers, please contact Allen. We will go from there.

  Thank you!

  Elena

  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Elena Pourmal
  Director of Technical Services and Operations
  The HDF Group
  1800 So. Oak St., Suite 203,
  Champaign, IL 61820
  www.hdfgroup.org
  (217)531-6112 (office)
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:

  Hey , Allen & Ward, next time I send an email to *both* of your group lists, can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the answers.

  >>Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration.
  >>This will build the netcdf libraries and utilities statically, avoiding the direct dependency on MSVCR100.dll.
  >>They will, however, still inherit any downstream .dll dependencies from the curl, hdf and zlib libraries.

  Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

  And, yes, the solution is to build *all* downstream code in the *same* way, either all static or all dynamic;
  mixing both will get you linking errors.

  >>> You might be able to work around this by downloading (or building) static versions of these libraries.

  Yes, Ward, this can be done by editing the HDF Cmake generated VS projects, and changing all of them to static, which I just did.

  In the curl case, you can do this by editing the file MakefileBuild.vc and change all /MD (dynamic) to /MT (static)

   Runtime library configuration
  !IF "$(RTLIBCFG)"=="static"
  #RTLIB = /MT
  RTLIB = /MTd
  RTLIB_DEBUG = /MTd
  !ELSE
  #RTLIB = /MD
  #RTLIB_DEBUG = /MDd

  >>Our binaries redistribute the MS CRT dlls that are used to build the binaries.

  Hey, Allen, I was not asking for the *binaries* !

  I was asking for an *option* in your CMake system that allows
  anyone who wants to generate the Visual Studio projects have them *already* "static ready" if they wish...

  So that, next time I need to generate everything from scratch I don't have to go each to each one of them (50 + projects ) and click the static button 50 times ...

  >>Because of the danger of using different CRTs (HDF lib uses one and user
  >>program uses different one) and the possible memory corruption with
  >>allocations, we build with /MD and provide the CRT with our binary.

  I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

  read this

  http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking

  quote

  " i personally prefer statically linked. i hate scrambling around looking for the right redist/dll/etc."

  that's what I do, I set to all static , and then move on to do more interesting things, like writing software, than dealing with DLL idiosyncracies.

  quote

  "Using /MT is risky if you create DLLs as well as an EXE. You'll end up with multiple copies of the CRT in your program.
  This was especially a problem with earlier versions of VS where each CRT would get its own heap, "

  Was this the problem you meant ?

  This might be true if you are distributing *binaries* with both the EXE and DLL. Or if you are linking your code against a 3rd party library *without* the code,
  someone gave you a LIB and a DLL only.

  In the HDF, netCDF and NCO worlds none of this is true: all sources are available , no secrets here :slight_smile:

  And you are lucky, Allen , because you only have the downstream ZLIB, and netCDF only has curl

  Here's a list of all NCO dependencies

  zlib ,
  HDF5,
  curl,
  netCDF, including OpenDAP, thank you for that :slight_smile:
  ANTLR, a parser generator for ncap2
  GSL, the GNU scientific library
  UDUnits, Unidata units (Not yet available for Windows)
  Regular Expressions (Not yet available for Windows, tough one this one )

  I have Visual Studio projects for all these (except UDUnits and Regular Expressions)
  , because I need to build the *source* , as you can see many projects to change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

  Pedro

  ------
  Pedro Vicente, Earth System Science
  University of California, Irvine
  http://www.ess.uci.edu/

  ----- Original Message -----

  From: "Allen Byrne" <byrn@hdfgroup.org>

  To: <hdf-forum@lists.hdfgroup.org>

  Sent: Wednesday, June 05, 2013 6:18 AM

  Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

  > Our binaries redistribute the MS CRT dlls that are used to build the binaries.
  > Because of the danger of using different CRTs (HDF lib uses one and user
  > program uses different one) and the possible memory corruption with
  > allocations, we build with /MD and provide the CRT with our binary.
  >
  > Hopefully, as more people use CMake and build a common knowledge of using
  > CMake, those wishing to build alternative versions of HDF will share their
  > code changes.
  >
  > Allen
  >
  > On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:
  >> Funny, I actually _prefer_ the DLL builds because I distribute the runtime
  >> C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
  >> executables/libraries having each library linked statically against their
  >> own C/C++ libraries can also lead to problems because of how memory is
  >> allocated/deallocated in each library version. There are 2 evils here and
  >> the idea is to pick the least of them. If anything I would petition the
  >> HDFGroup to provide BOTH a dynamically linked AND a statically linked
  >> runtime version.
  >>
  >> Just my 2 cents. Then again, I build my own HDF5 for our project precisely
  >> because of issues like this.
  >> ___________________________________________________________
  >> Mike Jackson Principal Software Engineer
  >> BlueQuartz Software Dayton, Ohio
  >> mike.jackson@bluequartz.net www.bluequartz.net
  >>
  >> On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
  >> > Hi Allen, Ward
  >> >
  >> > I have a request regarding your new CMake Windows build system, could you
  >> > add an option to make the build static regarding the Microsoft libraries
  >> > (MSVCR100D.dll) ?
  >> >
  >> > Starting with version 4.3.1, NCO
  >> >
  >> > http://nco.sourceforge.net/
  >> >
  >> > uses the HDF5 and netCDF Windows libraries made with your CMake system,
  >> > and this is causing problems for NCO users, as explained here
  >> >
  >> > https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
  >> >
  >> > and here
  >> >
  >> > https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
  >> >
  >> >
  >> > This is just a matter of changing the compiler flag to /MT(d)
  >> >
  >> > http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
  >> >
  >> > Using a dynamic build is just a bad idea, because of these DLL issues.
  >> >
  >> > I have some Windows executables from code I did in the early 90's , that
  >> > unfortunately I cannot run today, just because I linked them with DLLs,
  >> > with the DLLs from the Visual Studio from that time, that do not exist
  >> > anymore (it seems every new version they change the Visual Studio Dlls).
  >> >
  >> > Because of this I do not use Dlls, I know that eventually something will
  >> > go wrong :slight_smile:
  >> >
  >> > Pedro
  >> >
  >> > ------
  >> > Pedro Vicente, Earth System Science
  >> > University of California, Irvine
  >> > http://www.ess.uci.edu/
  >> >
  >> >
  >> > _______________________________________________
  >> > Hdf-forum is for HDF software users discussion.
  >> > Hdf-forum@lists.hdfgroup.org
  >> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
  >>
  >> _______________________________________________
  >> Hdf-forum is for HDF software users discussion.
  >> Hdf-forum@lists.hdfgroup.org
  >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
  >
  > _______________________________________________
  > Hdf-forum is for HDF software users discussion.
  > Hdf-forum@lists.hdfgroup.org
  > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
  >
  > _______________________________________________
  Hdf-forum is for HDF software users discussion.
  Hdf-forum@lists.hdfgroup.org
  http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

------------------------------------------------------------------------------

  _______________________________________________
  netcdfgroup mailing list
  netcdfgroup@unidata.ucar.edu
  For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/


#14

Hi Allen

"BUILD_SHARED_LIBS=OFF" should only control the library under build, not how
it is linked to CRT libraries. Which is Ward's understanding as well.

yes, my mistake

We have implemeted this with our "UserOptions" cmake hooks. Feel free to use
the same technique.

where can I find these "UserOptions" documentation?

I am using hdf5-1.8.11.
I guess you mean this was added to the svn repo after hdf5-1.8.11?
how to get the svn repo ?

Pedro

···

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

----- Original Message ----- From: "Allen Byrne" <byrn@hdfgroup.org>
To: "Ward Fisher" <wfisher@unidata.ucar.edu>
Cc: "Pedro Vicente" <pvicente@uci.edu>; "HDF Users Discussion List" <hdf-forum@lists.hdfgroup.org>; <netcdfgroup@unidata.ucar.edu>
Sent: Wednesday, September 18, 2013 3:37 PM
Subject: Re: [netcdfgroup] Make the Cmake Windows build static please !

We have implemeted this with our "UserOptions" cmake hooks. Feel free to use
the same technique.

"BUILD_SHARED_LIBS=OFF" should only control the library under build, not how
it is linked to CRT libraries. Which is Ward's understanding as well.

Allen

On Wednesday, September 18, 2013 02:05:01 PM Ward Fisher wrote:

On 9/18/13 1:36 PM, Pedro Vicente wrote:
> Hi Ward
>
>>> Can you be a little more specific about how this does not work?
>>> Is the issue that the generated projects are configured to use the
>>> the dynamic instead of static CRT?
>
> Yes, I meant that the the generated projects were all runtime library
> Multi-threaded Debug DLL (/MDd).
>
> I was under the impression that if I do -D"BUILD_SHARED_LIBS=OFF",
> then the projects would be static CRT ?
> Or is this option, just relative to the netCDF library itself,
> netcdf.lib ? In this case, my bad, sorry for the confusion.

I'm not sure what you mean by 'the projects would be static CRT'. The
file netcdf.lib would be a static library, but not a CRT. The Microsoft
C Run-Time Libraries (CRT) are a specific set of system libraries,
enumerated here:

http://msdn.microsoft.com/en-us/library/abx4dbyh(v=vs.100).aspx

It is possible to compile the netCDF library as a static object (a
standard 'ar' archive) that still has dependencies on shared libraries,
such as the hdf5 or libcurl dll's.

In the current system up to netCDF-C 4.3.1-rc2, netCDF was being built
statically, with a dependency on the shared CRT libraries. With the
changes previously described, it will be possible to (easily) build the
netCDF library and utilities statically, with a dependency on the static
CRT libraries.

In either case, the netCDF library and utilities are static; the
difference is in which library (CRT) was being linked against.

I hope this clarifies things; the new option (NC_USE_STATIC_CRT) is
being added and should be in the upcoming 4.3.1-rc3 release.

  -Ward

>>> I see how this could be set as a user-configurable option when
>>> invoking cmake. I will try to add this in and, assuming there
>>> aren't any problems caused by using the static CRT, it will be an
>>> option available in the >>upcoming netCDF-4.3.1-RC3.
>
> OK, great, thanks
>
> Allen, maybe the HD5 Cmake could use the same ?
>
> Pedro
>
> ------------
> pvicente@uci.edu
> Department of Earth System Science
> 3200 Croul Hall
> University of California, Irvine
> Irvine, CA 92697-3100


#15

I wonder if the message I sent out about the MT solution never actually got
delivered? Yes it has been in the SVN repository since the long discussion
about this issue.

I will need to update the documentation in the files (will be done today)
however, just copy the code in the config/cmake/UserMacros/Windows_MT.cmake file
to the root UserMacros.cmake file and enble the option during configure.

Allen

···

On Wednesday, September 18, 2013 03:59:37 PM Pedro Vicente wrote:

Hi Allen

> "BUILD_SHARED_LIBS=OFF" should only control the library under build, not
> how
> it is linked to CRT libraries. Which is Ward's understanding as well.

yes, my mistake

> We have implemeted this with our "UserOptions" cmake hooks. Feel free to
> use
> the same technique.

where can I find these "UserOptions" documentation?

I am using hdf5-1.8.11.
I guess you mean this was added to the svn repo after hdf5-1.8.11?
how to get the svn repo ?

Pedro

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

----- Original Message -----
From: "Allen Byrne" <byrn@hdfgroup.org>
To: "Ward Fisher" <wfisher@unidata.ucar.edu>
Cc: "Pedro Vicente" <pvicente@uci.edu>; "HDF Users Discussion List"
<hdf-forum@lists.hdfgroup.org>; <netcdfgroup@unidata.ucar.edu>
Sent: Wednesday, September 18, 2013 3:37 PM
Subject: Re: [netcdfgroup] Make the Cmake Windows build static please !

> We have implemeted this with our "UserOptions" cmake hooks. Feel free to
> use
> the same technique.
>
> "BUILD_SHARED_LIBS=OFF" should only control the library under build, not
> how
> it is linked to CRT libraries. Which is Ward's understanding as well.
>
> Allen
>
> On Wednesday, September 18, 2013 02:05:01 PM Ward Fisher wrote:
>> On 9/18/13 1:36 PM, Pedro Vicente wrote:
>> > Hi Ward
>> >


#16

Michael

You do not need findHdf5 any more.

All you do is find_package(HDF5 NO_MODULE) and point hdf5 dir to where cmake installed the hdf5-config.cmake (or whatever the file name is)

CMake defines all the targets as import/export definitions and everything is done for you. There is simply no need to use find hdf5 if you have a cmake install of it.

In your project you just say target_link_libraries(mytarget, hdf5 hdf5_hl etc etc)

Cmake defines these targets for you when it imports the config.

You can install dozens of versions each with different names like hdf5-1.8.11-static or anything and the name will be correctly picked up (although we don't actually generate them yet so technically what I'm saying is hypothetical!)

Throw away your findhdf5 file and move on with your life!

JB

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Michael Jackson
Sent: 07 June 2013 14:39
To: HDF Users Discussion List
Cc: help@hdfgroup.org; netcdfgroup@unidata.ucar.edu
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

Re: Version Stamp on library names. Please, do NOT do this. Boost does this and trying to get a build system to find those libraries via wildcard searching is fraught with danger and error prone. My FindHDF5.cmake file is crazy enough as it is. Adding in the complication of looking for hdf5_1_8_10 OR blah blah blah is going to make those Find modules even harder to write.

I do vote for the other idea though (static/dynamic runtime linking option). I am with John on this, once you "crack the code" on how to deploy your application with DLL libraries it is not that hard to do properly with the dynamic runtime libraries.

The original question though was more about the clashing of the runtime library versions which is why most of the developers that I meet we all compile our own versions rather than relying on the vendor's version. There are just too many variables for a vendor to try to cover to make it realistic.

Just my 2 cents.

Mike Jackson

_________________________________________________________

Mike Jackson mike.jackson@bluequartz.net<mailto:mike.jackson@bluequartz.net>

BlueQuartz Software www.bluequartz.net<http://www.bluequartz.net>

Principal Software Engineer Dayton, Ohio

On Fri, Jun 7, 2013 at 10:05 AM, Biddiscombe, John A. <biddisco@cscs.ch<mailto:biddisco@cscs.ch>> wrote:

I'd just like to second Elena's comment below. As one of the people

responsible for the cmake implementation of hdf5, I had a really hard

time with this static/dynamic CRT link issue.

I had set up 4 nightly builds of hdf5, static and dynamic, with and

without fortran, and unfortunately, all goes wrong when you start

using ifort on windows because of the way the supplied fortran libs

are linked (once you link the intel fortran libraries your link errors

multiply and you have no control of those libraries so having an

option to change static/dynamic is no help!)

So there was not a simple combination that was easy for all setups and

we basically opted to remove most of the options to get a combination

that worked - It took up so much of my time that I didn't 'finish the

job cleanly' and allow the user to decide for all cases.

I would concur that

1) Adding a version stamp to the dll/lib names is desirable (and easy

to do)

2) Adding a simple static/dynamic run time link option (and easy to do)

I'm happy to test patches if people create them, but all my stuff

works fine, so I'm not volunteering!

JB

[I would also personally favour dynamic linking in 99% of cases. When

you have plugin based architectures where objects are created and

destroyed in different modules, static linking is a no go. I do not

suffer from DLL hell and actually find it much much much easier to

distribute binaries to people on windows than linux]

--

John Biddiscombe, email:biddisco @.at.@ cscs.ch

http://www.cscs.ch/

CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07

Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On

Behalf Of Elena Pourmal

Sent: 06 June 2013 17:01

To: HDF Users Discussion List

Cc: help@hdfgroup.org<mailto:help@hdfgroup.org>; netcdfgroup@unidata.ucar.edu<mailto:netcdfgroup@unidata.ucar.edu>

Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

All,

Support for /MT(d) was removed from CMake because we do not have

resources to support it with the HDF software (including HDF5).

If this is a desired feature, we will need someone to test it on a

regular basis (daily with development and release HDF5 branches). If

there are volunteers, please contact Allen. We will go from there.

Thank you!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Elena Pourmal

Director of Technical Services and Operations The HDF Group

1800 So. Oak St., Suite 203,

Champaign, IL 61820

www.hdfgroup.org<http://www.hdfgroup.org>

(217)531-6112 (office)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:

Hey , Allen & Ward, next time I send an email to *both* of your group lists,

can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the

answers.

Building NetCDF statically is already an option, by passing

-DBUILD_SHARED_LIBS=OFF during configuration.

This will build the netcdf libraries and utilities statically, avoiding

the direct dependency on MSVCR100.dll.

They will, however, still inherit any downstream .dll dependencies from

the curl, hdf and zlib libraries.

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

And, yes, the solution is to build *all* downstream code in the *same* way,

either all static or all dynamic;

mixing both will get you linking errors.

You might be able to work around this by downloading (or building)

static versions of these libraries.

Yes, Ward, this can be done by editing the HDF Cmake generated VS projects,

and changing all of them to static, which I just did.

In the curl case, you can do this by editing the file MakefileBuild.vc and

change all /MD (dynamic) to /MT (static)

Runtime library configuration

!IF "$(RTLIBCFG)"=="static"

#RTLIB = /MT

RTLIB = /MTd

RTLIB_DEBUG = /MTd

!ELSE

#RTLIB = /MD

#RTLIB_DEBUG = /MDd

Our binaries redistribute the MS CRT dlls that are used to build the

binaries.

Hey, Allen, I was not asking for the *binaries* !

I was asking for an *option* in your CMake system that allows

anyone who wants to generate the Visual Studio projects have them *already*

"static ready" if they wish...

So that, next time I need to generate everything from scratch I don't have

to go each to each one of them (50 + projects ) and click the static button

50 times ...

Because of the danger of using different CRTs (HDF lib uses one and user

program uses different one) and the possible memory corruption with

allocations, we build with /MD and provide the CRT with our binary.

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

read this

http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking

quote

" i personally prefer statically linked. i hate scrambling around looking

for the right redist/dll/etc."

that's what I do, I set to all static , and then move on to do more

interesting things, like writing software, than dealing with DLL

idiosyncracies.

quote

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up with

multiple copies of the CRT in your program.

This was especially a problem with earlier versions of VS where each CRT

would get its own heap, "

Was this the problem you meant ?

This might be true if you are distributing *binaries* with both the EXE and

DLL. Or if you are linking your code against a 3rd party library *without*

the code,

someone gave you a LIB and a DLL only.

In the HDF, netCDF and NCO worlds none of this is true: all sources are

available , no secrets here :slight_smile:

And you are lucky, Allen , because you only have the downstream ZLIB, and

netCDF only has curl

Here's a list of all NCO dependencies

zlib ,

HDF5,

curl,

netCDF, including OpenDAP, thank you for that :slight_smile:

ANTLR, a parser generator for ncap2

GSL, the GNU scientific library

UDUnits, Unidata units (Not yet available for Windows)

Regular Expressions (Not yet available for Windows, tough one this one )

I have Visual Studio projects for all these (except UDUnits and Regular

Expressions)

, because I need to build the *source* , as you can see many projects to

change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

Pedro

------

Pedro Vicente, Earth System Science

University of California, Irvine

http://www.ess.uci.edu/

----- Original Message -----

From: "Allen Byrne" <byrn@hdfgroup.org<mailto:byrn@hdfgroup.org>>

To: <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>

Sent: Wednesday, June 05, 2013 6:18 AM

Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

Our binaries redistribute the MS CRT dlls that are used to build the

binaries.

Because of the danger of using different CRTs (HDF lib uses one and user

program uses different one) and the possible memory corruption with

allocations, we build with /MD and provide the CRT with our binary.

Hopefully, as more people use CMake and build a common knowledge of using

CMake, those wishing to build alternative versions of HDF will share their

code changes.

Allen

On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:

Funny, I actually _prefer_ the DLL builds because I distribute the

runtime

C/C++ libraries as allowed by MS. Depending on how one is using the HDF5

executables/libraries having each library linked statically against their

own C/C++ libraries can also lead to problems because of how memory is

allocated/deallocated in each library version. There are 2 evils here and

the idea is to pick the least of them. If anything I would petition the

HDFGroup to provide BOTH a dynamically linked AND a statically linked

runtime version.

Just my 2 cents. Then again, I build my own HDF5 for our project

precisely

because of issues like this.

___________________________________________________________

Mike Jackson Principal Software Engineer

BlueQuartz Software Dayton, Ohio

mike.jackson@bluequartz.net<mailto:mike.jackson@bluequartz.net> www.bluequartz.net<http://www.bluequartz.net>

On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu<mailto:pvicente@uci.edu>> wrote:

> Hi Allen, Ward

>

> I have a request regarding your new CMake Windows build system, could

> you

> add an option to make the build static regarding the Microsoft

> libraries

> (MSVCR100D.dll) ?

>

> Starting with version 4.3.1, NCO

>

> http://nco.sourceforge.net/

>

> uses the HDF5 and netCDF Windows libraries made with your CMake system,

> and this is causing problems for NCO users, as explained here

>

> https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151

>

> and here

>

> https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103

>

>

> This is just a matter of changing the compiler flag to /MT(d)

>

> http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx

>

> Using a dynamic build is just a bad idea, because of these DLL issues.

>

> I have some Windows executables from code I did in the early 90's ,

> that

> unfortunately I cannot run today, just because I linked them with DLLs,

> with the DLLs from the Visual Studio from that time, that do not exist

> anymore (it seems every new version they change the Visual Studio

> Dlls).

>

> Because of this I do not use Dlls, I know that eventually something

> will

> go wrong :slight_smile:

>

> Pedro

>

> ------

> Pedro Vicente, Earth System Science

> University of California, Irvine

> http://www.ess.uci.edu/

>

>

> _______________________________________________

> Hdf-forum is for HDF software users discussion.

> Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>

http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________

Hdf-forum is for HDF software users discussion.

Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>


#17

Hi all,

Just thought I'd toss in my two cents on this:

Statically linking to other libraries when you are building your own
library is not considered a very good idea on Windows. The reason for this
is that Microsoft implements the C runtime (CRT) functionality on a
per-library basis. i.e., each version of the CRT (debug vs. release, 10.0
vs. 9.0, etc.) has its own private version of the heap, file handles, and
other CRT objects. This state is absolutely not shared among the dlls.
This means that you have to be VERY careful to ensure that CRT objects are
not manipulated by the wrong library (e.g., calling 'free' on a buffer
allocated in a different version of the CRT). Not paying careful attention
to this will result in subtle, difficult to find bugs down the road. When
you statically link to a version of the CRT, you aren't fixing the problem,
you are actually multiplying it. Each static link gives you a separate CRT
state, just as if you mixed dll versions.

In general, we try to be very good about this in the HDF5 library and it's
not really an issue in normal use, though I'm pretty sure our help desk
still fields questions that are resolved by ensuring all libraries and the
executable are using the same CRT.

Even if it's not that big of a problem for us, the official word from
Microsoft is:

"Because a DLL built by linking to a static CRT will have its own CRT
state, it is not recommended to link statically to the CRT in a DLL unless
the consequences of this are specifically desired and understood."

http://msdn.microsoft.com/en-us/library/abx4dbyh(VS.80).aspx

Yeah, the static link makes it super easy to build your software, but at
the risk of corrupt applications. Personally, I'd rather solve the "where
are my dlls" problem instead of the "why is my software mysteriously
crashing or corrupting data" problem. The first one is a LOT more
tractable.

If anyone is sitting on a pile of money they need to spend, this might
actually be a fixable problem. The Win32 libraries do not have this
linkage issue, so if we extended our platform-independence layer so that
our C calls mapped to Win32 calls, the problem should go away, with almost
no change to the bulk of the source code.

Personally, I'm not keen on making it easy for people to statically link to
the CRT since it's a documented bad idea. If we're going to spend any
money on Windows library issues, I'd rather see us properly distribute
debug libraries.

Cheers,

Dana

···

On Sat, Jun 8, 2013 at 3:48 PM, Pedro Vicente <pvicente@uci.edu> wrote:

**

Hi Elena, John, Ward, Allen, Michael

> Support for /MT(d) was removed from CMake because we do not have
resources to support it with the HDF software (including HDF5).
> If this is a desired feature, we will need someone to test it on a
regular basis (daily with development and release HDF5 branches).

> I had set up 4 nightly builds of hdf5, static and dynamic, with and
> without fortran, and unfortunately, all goes wrong when you start
> using ifort on windows because of the way the supplied fortran libs
> are linked (once you link the intel fortran libraries your link errors
> multiply and you have no control of those libraries so having an
> option to change static/dynamic is no help!)
> I would concur that
> 1) Adding a version stamp to the dll/lib names is desirable (and
easy to do)
> 2) Adding a simple static/dynamic run time link option (and easy to
do)

OK, so, why not just add now the simplest configurations that are already
working, and that Ward has already done for netCDF?

The only minimal configuration that I think is certainly needed for
everyone is item 2), the static/dynamic run time link option,
the equivalent of GNU automake --enable-static/shared

One of the things that I am interested in is to build a Cmake build system
for NCO also, so what do you all think of a common CMake system for HDF,
netCDF and NCO?

I mean, things like

1) common syntax options like netCDF's

-DBUILD_SHARED_LIBS=OFF

2) use of environment variables, with the same name for libraries and
header files names;
for example, I use on my projects for the location of the curl library,
the name

LIB_CURL

and netCDF uses

CURL_LIBRARY

this would make things simpler for anyone that needs to build HDF, netCDF
and NCO.

Elena,
if you need to discuss in detail, call me (same number still), or we can
set Skype video/audio joint conference calls if needed.

Pedro
------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
*From:* Biddiscombe, John A. <biddisco@cscs.ch>
*To:* HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Cc:* help@hdfgroup.org ; netcdfgroup@unidata.ucar.edu
*Sent:* Friday, June 07, 2013 1:05 AM
*Subject:* Re: [netcdfgroup] [Hdf-forum] Make the Cmake Windows build
staticplease !

I’d just like to second Elena’s comment below. As one of the people
responsible for the cmake implementation of hdf5, I had a really hard time
with this static/dynamic CRT link issue.****

I had set up 4 nightly builds of hdf5, static and dynamic, with and
without fortran, and unfortunately, all goes wrong when you start using
ifort on windows because of the way the supplied fortran libs are linked
(once you link the intel fortran libraries your link errors multiply and
you have no control of those libraries so having an option to change
static/dynamic is no help!)****

So there was not a simple combination that was easy for all setups and we
basically opted to remove most of the options to get a combination that
worked - It took up so much of my time that I didn’t ‘finish the job
cleanly’ and allow the user to decide for all cases.****

** **

I would concur that****

**1) **Adding a version stamp to the dll/lib names is desirable (and
easy to do)****

**2) **Adding a simple static/dynamic run time link option (and easy
to do)****

** **

I’m happy to test patches if people create them, but all my stuff works
fine, so I’m not volunteering!****

** **

JB****

** **

[I would also personally favour dynamic linking in 99% of cases. When you
have plugin based architectures where objects are created and destroyed in
different modules, static linking is a no go. I do not suffer from DLL hell
and actually find it much much much easier to distribute binaries to people
on windows than linux]****

** **

** **

-- ****

John Biddiscombe, email:biddisco @.at.@ cscs.ch****

http://www.cscs.ch/****

CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07****

Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82****

** **

** **

** **

** **

** **

** **

** **

** **

** **

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Elena Pourmal
*Sent:* 06 June 2013 17:01
*To:* HDF Users Discussion List
*Cc:* help@hdfgroup.org; netcdfgroup@unidata.ucar.edu
*Subject:* Re: [Hdf-forum] Make the Cmake Windows build static please !***
*

** **

All,****

** **

Support for /MT(d) was removed from CMake because we do not have resources
to support it with the HDF software (including HDF5).****

** **

If this is a desired feature, we will need someone to test it on a regular
basis (daily with development and release HDF5 branches). If there are
volunteers, please contact Allen. We will go from there.****

** **

Thank you!****

** **

Elena****

** **

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820
www.hdfgroup.org
(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

****

** **

On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:****

****

Hey , Allen & Ward, next time I send an email to *both* of your group
lists, can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of
the answers.****

****

>>Building NetCDF statically is already an option, by passing
-DBUILD_SHARED_LIBS=OFF during configuration.
>>This will build the netcdf libraries and utilities statically, avoiding
the direct dependency on MSVCR100.dll.
>>They will, however, still inherit any downstream .dll dependencies from
the curl, hdf and zlib libraries. ****

****

Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:
****

****

And, yes, the solution is to build *all* downstream code in the *same*
way, either all static or all dynamic;
mixing both will get you linking errors.****

****

>>> You might be able to work around this by downloading (or building)
static versions of these libraries. ****

****

Yes, Ward, this can be done by editing the HDF Cmake generated VS
projects, and changing all of them to static, which I just did.****

****

In the curl case, you can do this by editing the file MakefileBuild.vc and
change all /MD (dynamic) to /MT (static)****

****

Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd****

****

>>Our binaries redistribute the MS CRT dlls that are used to build the
binaries.****

****

Hey, Allen, I was not asking for the *binaries* !****

****

I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them
*already* "static ready" if they wish...****

****

So that, next time I need to generate everything from scratch I don't have
to go each to each one of them (50 + projects ) and click the static button
50 times ...****

****

>>Because of the danger of using different CRTs (HDF lib uses one and user

>>program uses different one) and the possible memory corruption with
>>allocations, we build with /MD and provide the CRT with our binary.****

****

I don't pretend to know everything, but when I don't (often :slight_smile: ) I search
****

****

read this****

****

http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking
****

****

quote****

****

" i personally prefer statically linked. i hate scrambling around looking
for the right redist/dll/etc."****

****

that's what I do, I set to all static , and then move on to do more
interesting things, like writing software, than dealing with DLL
idiosyncracies.****

****

quote****

****

"Using /MT is risky if you create DLLs as well as an EXE. You'll end up
with multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT
would get its own heap, "****

****

Was this the problem you meant ?****

****

This might be true if you are distributing *binaries* with both the EXE
and DLL. Or if you are linking your code against a 3rd party library
*without* the code,
someone gave you a LIB and a DLL only.****

****

In the HDF, netCDF and NCO worlds none of this is true: all sources are
available , no secrets here :-)****

****

And you are lucky, Allen , because you only have the downstream ZLIB, and
netCDF only has curl****

****

Here's a list of all NCO dependencies****

****

zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :slight_smile:
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )**
**

****

I have Visual Studio projects for all these (except UDUnits and Regular
Expressions)
, because I need to build the *source* , as you can see many projects to
change to static/dynamic/32/64bit/debug/release combinations :-)****

****

Pedro****

****

****

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/****

****

****

****

----- Original Message -----****

From: "Allen Byrne" <byrn@hdfgroup.org>****

To: <hdf-forum@lists.hdfgroup.org>****

Sent: Wednesday, June 05, 2013 6:18 AM****

Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !****

** **

> Our binaries redistribute the MS CRT dlls that are used to build the
binaries.
> Because of the danger of using different CRTs (HDF lib uses one and user

> program uses different one) and the possible memory corruption with
> allocations, we build with /MD and provide the CRT with our binary.
>
> Hopefully, as more people use CMake and build a common knowledge of using

> CMake, those wishing to build alternative versions of HDF will share
their
> code changes.
>
> Allen
>
> On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:
>> Funny, I actually _prefer_ the DLL builds because I distribute the
runtime
>> C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
>> executables/libraries having each library linked statically against
their
>> own C/C++ libraries can also lead to problems because of how memory is
>> allocated/deallocated in each library version. There are 2 evils here
and
>> the idea is to pick the least of them. If anything I would petition the
>> HDFGroup to provide BOTH a dynamically linked AND a statically linked
>> runtime version.
>>
>> Just my 2 cents. Then again, I build my own HDF5 for our project
precisely
>> because of issues like this.
>> ___________________________________________________________
>> Mike Jackson Principal Software Engineer
>> BlueQuartz Software Dayton, Ohio
>> mike.jackson@bluequartz.net www.bluequartz.net
>>
>> On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
>> > Hi Allen, Ward
>> >
>> > I have a request regarding your new CMake Windows build system, could
you
>> > add an option to make the build static regarding the Microsoft
libraries
>> > (MSVCR100D.dll) ?
>> >
>> > Starting with version 4.3.1, NCO
>> >
>> > http://nco.sourceforge.net/
>> >
>> > uses the HDF5 and netCDF Windows libraries made with your CMake
system,
>> > and this is causing problems for NCO users, as explained here
>> >
>> > https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>> >
>> > and here
>> >
>> > https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>> >
>> >
>> > This is just a matter of changing the compiler flag to /MT(d)
>> >
>> > http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>> >
>> > Using a dynamic build is just a bad idea, because of these DLL issues.
>> >
>> > I have some Windows executables from code I did in the early 90's ,
that
>> > unfortunately I cannot run today, just because I linked them with
DLLs,
>> > with the DLLs from the Visual Studio from that time, that do not exist
>> > anymore (it seems every new version they change the Visual Studio
Dlls).
>> >
>> > Because of this I do not use Dlls, I know that eventually something
will
>> > go wrong :slight_smile:
>> >
>> > Pedro
>> >
>> > ------
>> > Pedro Vicente, Earth System Science
>> > University of California, Irvine
>> > http://www.ess.uci.edu/
>> >
>> >
>> > _______________________________________________
>> > Hdf-forum is for HDF software users discussion.
>> > Hdf-forum@lists.hdfgroup.org
>> >
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@lists.hdfgroup.org
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
> _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org***
*

** **

------------------------------

_______________________________________________
netcdfgroup mailing list
netcdfgroup@unidata.ucar.edu
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#18

Hi Allen

Here am I again complaining about missing DLLs (along with NCO users).

https://sourceforge.net/projects/nco/forums/forum/9829/topic/8553658/index/page/1

This time is not the Microsoft CRT as a DLL, but the HDF5 library as DLL.

It seems that there is not an option to build HDF5 statically under the current distro (I am using 1.8.11 built from source using Cmake).

Is this correct?

If yes, could you please add an option to allow this?

Thanks

···

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

----- Original Message ----- From: "Allen Byrne" <byrn@hdfgroup.org>
To: "Pedro Vicente" <pvicente@uci.edu>
Cc: <netcdfgroup@unidata.ucar.edu>; "HDF Users Discussion List" <hdf-forum@lists.hdfgroup.org>; "Ward Fisher" <wfisher@unidata.ucar.edu>
Sent: Thursday, September 19, 2013 7:05 AM
Subject: Re: [Hdf-forum] [netcdfgroup] Make the Cmake Windows build staticplease !

I wonder if the message I sent out about the MT solution never actually got
delivered? Yes it has been in the SVN repository since the long discussion
about this issue.

I will need to update the documentation in the files (will be done today)
however, just copy the code in the config/cmake/UserMacros/Windows_MT.cmake file
to the root UserMacros.cmake file and enble the option during configure.

Allen

On Wednesday, September 18, 2013 03:59:37 PM Pedro Vicente wrote:

Hi Allen

> "BUILD_SHARED_LIBS=OFF" should only control the library under build, > not
> how
> it is linked to CRT libraries. Which is Ward's understanding as well.

yes, my mistake

> We have implemeted this with our "UserOptions" cmake hooks. Feel free > to
> use
> the same technique.

where can I find these "UserOptions" documentation?

I am using hdf5-1.8.11.
I guess you mean this was added to the svn repo after hdf5-1.8.11?
how to get the svn repo ?

Pedro

------------
pvicente@uci.edu
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100

----- Original Message -----
From: "Allen Byrne" <byrn@hdfgroup.org>
To: "Ward Fisher" <wfisher@unidata.ucar.edu>
Cc: "Pedro Vicente" <pvicente@uci.edu>; "HDF Users Discussion List"
<hdf-forum@lists.hdfgroup.org>; <netcdfgroup@unidata.ucar.edu>
Sent: Wednesday, September 18, 2013 3:37 PM
Subject: Re: [netcdfgroup] Make the Cmake Windows build static please !

> We have implemeted this with our "UserOptions" cmake hooks. Feel free > to
> use
> the same technique.
>
> "BUILD_SHARED_LIBS=OFF" should only control the library under build, > not
> how
> it is linked to CRT libraries. Which is Ward's understanding as well.
>
> Allen
>
> On Wednesday, September 18, 2013 02:05:01 PM Ward Fisher wrote:
>> On 9/18/13 1:36 PM, Pedro Vicente wrote:
>> > Hi Ward
>> >

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#19

Hi Dana

My original thread title

"Make the CMake Windows build static"

is a bit misleading, sorry.

HDF5, in the NCO and netCDF cases, is only one of several input libraries.

You can think of both HDF5 and netCDF in 2 ways

1) A library that will be used as *input* for a program (EXE file) that any HDF/netCDF user wants to build, his/her own program.
In this case he will need your LIB (in case he wants to compile) or the DLL (if he just wants to run the EXE)

2) A final program destination already done (EXE) , like h5dump or ncdump, that can be either statically built (no DLL) or dynamically built (need DLL)

For NCO, only case 2) exists. NCO is just the final program, not an input library to anyone.
We only distribute EXE files, not LIB files, like you do.

If I am taking the HDF5 binaries you provide, and combine with other several other library binaries, and even other libraries source code ,
in a combination of several LIB, DLL, and EXE files, dynamic is the way to go, because of the reasons you mention.

Note: just the binaries. Not the whatever way I have to obtain them, in this case, a Cmake compiler flag.

But, on the other hand, if I take all the source code only from these “libraries”, and combine then into an EXE file statically build,
then I can consider all those “libraries” not libraries any more, just pieces of source files that will produce an EXE.

One note, though, and here I would like you to correct me, if I’m wrong.

I can separate all these individual “libraries”, like HDF5, into an intermediate input file, a LIB file,
that I use *only* for my compilation into an EXE, *not* to distribute to anyone.

Take the simplest case of 2 source files

a.c that I compile into a.LIB

main.c that I compile into main.EXE with a link input of a.LIB

If I compile both of these Visual Studio projects with the same static flag, then,
there is no DLL dependency, and I think the CRT issues you mention do not happen.

Do you agree with this?

When I started building Windows software in the 90’s with the Visual Studio from that time I was using the DLL option.

Why? It seemed like a good idea. Dozens of programs done, thousands of compiler builds, why not just make them share the system DLL?

Many of this software I kept both the source and the final EXE, stored in a backup drive. But I forgot to store also the DLL, just because I never cared about them,
I was not distributing software to anyone, just doing my college things.

Fast forward 20 years, I want to run the EXE from that program. what happens? DLL whatever year 1993 is missing, oh my , I forgot to backup it.
And now where can I find it?, nowhere :slight_smile:

The funny thing is that the EXE is still *binary* compatible regarding the Windows year 1993 version and Windows 7 today.
I just cannot run it because I don’t have the DLL. If I *had* used the static option, now, today, I could run that EXE.

So, that’s all what I was trying to say with my comments, “people, don’t use DLLs, they are bad”

Totally different case as compared to distributing HDF5 as a library.

CMake is only an automatic generator of Visual Studio projects.

Like it is now, I can go to your generated projects and change the compiler flags.

But I thought it would be a valuable feature for the community to have this choice already built directly into the Cmake step, like Ward did for netCDF,
and John also did, I think, in the original setup, but you took off the final distribution.

Is that right, John?
Because I think the option –enable-static/shared, is just a given thing with the system, just a matter of adding a one liner to the script ?

So, regarding your comment

“Personally, I'm not keen on making it easy for people to statically link to the CRT since it's a documented bad idea”

I don’t think you should be limiting our choices.

I am just asking to let us do your own choices in the way we compile our software.

Pedro

···

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

  ----- Original Message -----
  From: Dana Robinson
  To: HDF Users Discussion List
  Cc: netcdfgroup@unidata.ucar.edu
  Sent: Sunday, June 09, 2013 5:52 PM
  Subject: Re: [Hdf-forum] [netcdfgroup] Make the Cmake Windows build staticplease !

  Hi all,

  Just thought I'd toss in my two cents on this:

  Statically linking to other libraries when you are building your own library is not considered a very good idea on Windows. The reason for this is that Microsoft implements the C runtime (CRT) functionality on a per-library basis. i.e., each version of the CRT (debug vs. release, 10.0 vs. 9.0, etc.) has its own private version of the heap, file handles, and other CRT objects. This state is absolutely not shared among the dlls. This means that you have to be VERY careful to ensure that CRT objects are not manipulated by the wrong library (e.g., calling 'free' on a buffer allocated in a different version of the CRT). Not paying careful attention to this will result in subtle, difficult to find bugs down the road. When you statically link to a version of the CRT, you aren't fixing the problem, you are actually multiplying it. Each static link gives you a separate CRT state, just as if you mixed dll versions.

  In general, we try to be very good about this in the HDF5 library and it's not really an issue in normal use, though I'm pretty sure our help desk still fields questions that are resolved by ensuring all libraries and the executable are using the same CRT.

  Even if it's not that big of a problem for us, the official word from Microsoft is:

  "Because a DLL built by linking to a static CRT will have its own CRT state, it is not recommended to link statically to the CRT in a DLL unless the consequences of this are specifically desired and understood."

  http://msdn.microsoft.com/en-us/library/abx4dbyh(VS.80).aspx

  Yeah, the static link makes it super easy to build your software, but at the risk of corrupt applications. Personally, I'd rather solve the "where are my dlls" problem instead of the "why is my software mysteriously crashing or corrupting data" problem. The first one is a LOT more tractable.

  If anyone is sitting on a pile of money they need to spend, this might actually be a fixable problem. The Win32 libraries do not have this linkage issue, so if we extended our platform-independence layer so that our C calls mapped to Win32 calls, the problem should go away, with almost no change to the bulk of the source code.

  Personally, I'm not keen on making it easy for people to statically link to the CRT since it's a documented bad idea. If we're going to spend any money on Windows library issues, I'd rather see us properly distribute debug libraries.

  Cheers,

  Dana

  On Sat, Jun 8, 2013 at 3:48 PM, Pedro Vicente <pvicente@uci.edu> wrote:

    Hi Elena, John, Ward, Allen, Michael

    > Support for /MT(d) was removed from CMake because we do not have resources to support it with the HDF software (including HDF5).
    > If this is a desired feature, we will need someone to test it on a regular basis (daily with development and release HDF5 branches).

    > I had set up 4 nightly builds of hdf5, static and dynamic, with and
    > without fortran, and unfortunately, all goes wrong when you start
    > using ifort on windows because of the way the supplied fortran libs
    > are linked (once you link the intel fortran libraries your link errors
    > multiply and you have no control of those libraries so having an
    > option to change static/dynamic is no help!)

    > I would concur that
    > 1) Adding a version stamp to the dll/lib names is desirable (and easy to do)
    > 2) Adding a simple static/dynamic run time link option (and easy to do)

    OK, so, why not just add now the simplest configurations that are already working, and that Ward has already done for netCDF?

    The only minimal configuration that I think is certainly needed for everyone is item 2), the static/dynamic run time link option, the equivalent of GNU automake --enable-static/shared

    One of the things that I am interested in is to build a Cmake build system for NCO also, so what do you all think of a common CMake system for HDF, netCDF and NCO?

    I mean, things like

    1) common syntax options like netCDF's

    -DBUILD_SHARED_LIBS=OFF

    2) use of environment variables, with the same name for libraries and header files names;
    for example, I use on my projects for the location of the curl library, the name

    LIB_CURL

    and netCDF uses

    CURL_LIBRARY

    this would make things simpler for anyone that needs to build HDF, netCDF and NCO.

    Elena,
    if you need to discuss in detail, call me (same number still), or we can set Skype video/audio joint conference calls if needed.

    Pedro

    ------
    Pedro Vicente, Earth System Science
    University of California, Irvine
    http://www.ess.uci.edu/

      ----- Original Message -----
      From: Biddiscombe, John A.
      To: HDF Users Discussion List
      Cc: help@hdfgroup.org ; netcdfgroup@unidata.ucar.edu
      Sent: Friday, June 07, 2013 1:05 AM
      Subject: Re: [netcdfgroup] [Hdf-forum] Make the Cmake Windows build staticplease !

      I’d just like to second Elena’s comment below. As one of the people responsible for the cmake implementation of hdf5, I had a really hard time with this static/dynamic CRT link issue.

      I had set up 4 nightly builds of hdf5, static and dynamic, with and without fortran, and unfortunately, all goes wrong when you start using ifort on windows because of the way the supplied fortran libs are linked (once you link the intel fortran libraries your link errors multiply and you have no control of those libraries so having an option to change static/dynamic is no help!)

      So there was not a simple combination that was easy for all setups and we basically opted to remove most of the options to get a combination that worked - It took up so much of my time that I didn’t ‘finish the job cleanly’ and allow the user to decide for all cases.

      I would concur that

      1) Adding a version stamp to the dll/lib names is desirable (and easy to do)

      2) Adding a simple static/dynamic run time link option (and easy to do)

      I’m happy to test patches if people create them, but all my stuff works fine, so I’m not volunteering!

      JB

      [I would also personally favour dynamic linking in 99% of cases. When you have plugin based architectures where objects are created and destroyed in different modules, static linking is a no go. I do not suffer from DLL hell and actually find it much much much easier to distribute binaries to people on windows than linux]

      --

      John Biddiscombe, email:biddisco @.at.@ cscs.ch

      http://www.cscs.ch/

      CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07

      Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82

      From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Elena Pourmal
      Sent: 06 June 2013 17:01
      To: HDF Users Discussion List
      Cc: help@hdfgroup.org; netcdfgroup@unidata.ucar.edu
      Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

      All,

      Support for /MT(d) was removed from CMake because we do not have resources to support it with the HDF software (including HDF5).

      If this is a desired feature, we will need someone to test it on a regular basis (daily with development and release HDF5 branches). If there are volunteers, please contact Allen. We will go from there.

      Thank you!

      Elena

      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
      Elena Pourmal
      Director of Technical Services and Operations
      The HDF Group
      1800 So. Oak St., Suite 203,
      Champaign, IL 61820
      www.hdfgroup.org
      (217)531-6112 (office)
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

      On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:

      Hey , Allen & Ward, next time I send an email to *both* of your group lists, can you just do a "reply-all" ? :slight_smile: ... otherwise , we get half of the answers.

      >>Building NetCDF statically is already an option, by passing -DBUILD_SHARED_LIBS=OFF during configuration.
      >>This will build the netcdf libraries and utilities statically, avoiding the direct dependency on MSVCR100.dll.
      >>They will, however, still inherit any downstream .dll dependencies from the curl, hdf and zlib libraries.

      Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here :slight_smile:

      And, yes, the solution is to build *all* downstream code in the *same* way, either all static or all dynamic;
      mixing both will get you linking errors.

      >>> You might be able to work around this by downloading (or building) static versions of these libraries.

      Yes, Ward, this can be done by editing the HDF Cmake generated VS projects, and changing all of them to static, which I just did.

      In the curl case, you can do this by editing the file MakefileBuild.vc and change all /MD (dynamic) to /MT (static)

       Runtime library configuration
      !IF "$(RTLIBCFG)"=="static"
      #RTLIB = /MT
      RTLIB = /MTd
      RTLIB_DEBUG = /MTd
      !ELSE
      #RTLIB = /MD
      #RTLIB_DEBUG = /MDd

      >>Our binaries redistribute the MS CRT dlls that are used to build the binaries.

      Hey, Allen, I was not asking for the *binaries* !

      I was asking for an *option* in your CMake system that allows
      anyone who wants to generate the Visual Studio projects have them *already* "static ready" if they wish...

      So that, next time I need to generate everything from scratch I don't have to go each to each one of them (50 + projects ) and click the static button 50 times ...

      >>Because of the danger of using different CRTs (HDF lib uses one and user
      >>program uses different one) and the possible memory corruption with
      >>allocations, we build with /MD and provide the CRT with our binary.

      I don't pretend to know everything, but when I don't (often :slight_smile: ) I search

      read this

      http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking

      quote

      " i personally prefer statically linked. i hate scrambling around looking for the right redist/dll/etc."

      that's what I do, I set to all static , and then move on to do more interesting things, like writing software, than dealing with DLL idiosyncracies.

      quote

      "Using /MT is risky if you create DLLs as well as an EXE. You'll end up with multiple copies of the CRT in your program.
      This was especially a problem with earlier versions of VS where each CRT would get its own heap, "

      Was this the problem you meant ?

      This might be true if you are distributing *binaries* with both the EXE and DLL. Or if you are linking your code against a 3rd party library *without* the code,
      someone gave you a LIB and a DLL only.

      In the HDF, netCDF and NCO worlds none of this is true: all sources are available , no secrets here :slight_smile:

      And you are lucky, Allen , because you only have the downstream ZLIB, and netCDF only has curl

      Here's a list of all NCO dependencies

      zlib ,
      HDF5,
      curl,
      netCDF, including OpenDAP, thank you for that :slight_smile:
      ANTLR, a parser generator for ncap2
      GSL, the GNU scientific library
      UDUnits, Unidata units (Not yet available for Windows)
      Regular Expressions (Not yet available for Windows, tough one this one )

      I have Visual Studio projects for all these (except UDUnits and Regular Expressions)
      , because I need to build the *source* , as you can see many projects to change to static/dynamic/32/64bit/debug/release combinations :slight_smile:

      Pedro

      ------
      Pedro Vicente, Earth System Science
      University of California, Irvine
      http://www.ess.uci.edu/

      ----- Original Message -----

      From: "Allen Byrne" <byrn@hdfgroup.org>

      To: <hdf-forum@lists.hdfgroup.org>

      Sent: Wednesday, June 05, 2013 6:18 AM

      Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !

      > Our binaries redistribute the MS CRT dlls that are used to build the binaries.
      > Because of the danger of using different CRTs (HDF lib uses one and user
      > program uses different one) and the possible memory corruption with
      > allocations, we build with /MD and provide the CRT with our binary.
      >
      > Hopefully, as more people use CMake and build a common knowledge of using
      > CMake, those wishing to build alternative versions of HDF will share their
      > code changes.
      >
      > Allen
      >
      > On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:
      >> Funny, I actually _prefer_ the DLL builds because I distribute the runtime
      >> C/C++ libraries as allowed by MS. Depending on how one is using the HDF5
      >> executables/libraries having each library linked statically against their
      >> own C/C++ libraries can also lead to problems because of how memory is
      >> allocated/deallocated in each library version. There are 2 evils here and
      >> the idea is to pick the least of them. If anything I would petition the
      >> HDFGroup to provide BOTH a dynamically linked AND a statically linked
      >> runtime version.
      >>
      >> Just my 2 cents. Then again, I build my own HDF5 for our project precisely
      >> because of issues like this.
      >> ___________________________________________________________
      >> Mike Jackson Principal Software Engineer
      >> BlueQuartz Software Dayton, Ohio
      >> mike.jackson@bluequartz.net www.bluequartz.net
      >>
      >> On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@uci.edu> wrote:
      >> > Hi Allen, Ward
      >> >
      >> > I have a request regarding your new CMake Windows build system, could you
      >> > add an option to make the build static regarding the Microsoft libraries
      >> > (MSVCR100D.dll) ?
      >> >
      >> > Starting with version 4.3.1, NCO
      >> >
      >> > http://nco.sourceforge.net/
      >> >
      >> > uses the HDF5 and netCDF Windows libraries made with your CMake system,
      >> > and this is causing problems for NCO users, as explained here
      >> >
      >> > https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
      >> >
      >> > and here
      >> >
      >> > https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
      >> >
      >> >
      >> > This is just a matter of changing the compiler flag to /MT(d)
      >> >
      >> > http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
      >> >
      >> > Using a dynamic build is just a bad idea, because of these DLL issues.
      >> >
      >> > I have some Windows executables from code I did in the early 90's , that
      >> > unfortunately I cannot run today, just because I linked them with DLLs,
      >> > with the DLLs from the Visual Studio from that time, that do not exist
      >> > anymore (it seems every new version they change the Visual Studio Dlls).
      >> >
      >> > Because of this I do not use Dlls, I know that eventually something will
      >> > go wrong :slight_smile:
      >> >
      >> > Pedro
      >> >
      >> > ------
      >> > Pedro Vicente, Earth System Science
      >> > University of California, Irvine
      >> > http://www.ess.uci.edu/
      >> >
      >> >
      >> > _______________________________________________
      >> > Hdf-forum is for HDF software users discussion.
      >> > Hdf-forum@lists.hdfgroup.org
      >> > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
      >>
      >> _______________________________________________
      >> Hdf-forum is for HDF software users discussion.
      >> Hdf-forum@lists.hdfgroup.org
      >> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
      >
      > _______________________________________________
      > Hdf-forum is for HDF software users discussion.
      > Hdf-forum@lists.hdfgroup.org
      > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
      >
      > _______________________________________________
      Hdf-forum is for HDF software users discussion.
      Hdf-forum@lists.hdfgroup.org
      http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--------------------------------------------------------------------------

      _______________________________________________
      netcdfgroup mailing list
      netcdfgroup@unidata.ucar.edu
      For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

------------------------------------------------------------------------------

  _______________________________________________
  Hdf-forum is for HDF software users discussion.
  Hdf-forum@lists.hdfgroup.org
  http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org


#20

Hi Pedro,

Answers interspersed below...

**
Hi Dana

My original thread title

"Make the CMake Windows build static"

is a bit misleading, sorry.

HDF5, in the NCO and netCDF cases, is only one of several input libraries.

You can think of both HDF5 and netCDF in 2 ways

1) A library that will be used as *input* for a program (EXE file) that
any HDF/netCDF user wants to build, his/her own program.
In this case he will need your LIB (in case he wants to compile) or the
DLL (if he just wants to run the EXE)

2) A final program destination already done (EXE) , like h5dump or ncdump,
that can be either statically built (no DLL) or dynamically built (need DLL)

For NCO, only case 2) exists. NCO is just the final program, not an input
library to anyone.
We only distribute EXE files, not LIB files, like you do.

If I am taking the HDF5 binaries you provide, and combine with other
several other library binaries, and even other libraries source code ,
in a combination of several LIB, DLL, and EXE files, dynamic is the way
to go, because of the reasons you mention.

Note: just the binaries. Not the whatever way I have to obtain them, in
this case, a Cmake compiler flag.

But, on the other hand, if I take all the source code only from these
“libraries”, and combine then into an EXE file statically build,
then I can consider all those “libraries” not libraries any more, just
pieces of source files that will produce an EXE.

One note, though, and here I would like you to correct me, if I’m wrong.

I can separate all these individual “libraries”, like HDF5, into an
intermediate input file, a LIB file,
that I use *only* for my compilation into an EXE, *not* to distribute to
anyone.

Take the simplest case of 2 source files

a.c that I compile into a.LIB

main.c that I compile into main.EXE with a link input of a.LIB

If I compile both of these Visual Studio projects with the same static
flag, then,
there is no DLL dependency, and I think the CRT issues you mention do not
happen.

Do you agree with this?

No.

a.lib and main.exe will have separate CRT states. You will not be able to
malloc in a.c and free in main.exe without risking heap corruption in your
exe.

I have attached a helpful set of slides that explains this.

Note that what you described will "work" in the sense that the OS won't
complain. It's not very well documented, but it appears that the Win32
underpinnings are perfectly happy (this really isn't that surprising since
everything belongs to the same process and the Win32 layer doesn't care
about what's going on above it). It's just the bookkeeping information in
the CRTs that gets messed up and this may or may not manifest itself as an
error in your application. At any rate, it's not correct behavior.

This is not what happens in Linux and other Unix-like operating systems.
The standard C library is a central part of the Linux OS but Windows uses
the Win32 API exclusively. The bizarre linking issues on Windows seem to
be due to this non-central nature of the library. This is also why you
have to install the CRT on Windows, whereas on Linux it's just there.

The executive summary is that you should dynamically link ALL libraries
that expose CRT objects via the library API. This obviously includes the
Microsoft CRTs and, not so obviously, the HDF5 library.

Also, just to be clear, it is totally ok to write to memory allocated by a
different CRT. That doesn't affect internal CRT heap tracking. It's just
unsafe to resize or free the memory unless that CRT provided it.

When I started building Windows software in the 90’s with the Visual
Studio from that time I was using the DLL option.

Why? It seemed like a good idea. Dozens of programs done, thousands of
compiler builds, why not just make them share the system DLL?

Many of this software I kept both the source and the final EXE, stored in
a backup drive. But I forgot to store also the DLL, just because I never
cared about them,
I was not distributing software to anyone, just doing my college things.

Fast forward 20 years, I want to run the EXE from that program. what
happens? DLL whatever year 1993 is missing, oh my , I forgot to backup it.
And now where can I find it?, nowhere :slight_smile:

The funny thing is that the EXE is still *binary* compatible regarding the
Windows year 1993 version and Windows 7 today.
I just cannot run it because I don’t have the DLL. If I *had* used the
static option, now, today, I could run that EXE.

So, that’s all what I was trying to say with my comments, “people, don’t
use DLLs, they are bad”

Totally different case as compared to distributing HDF5 as a library.

CMake is only an automatic generator of Visual Studio projects.

Like it is now, I can go to your generated projects and change the
compiler flags.

But I thought it would be a valuable feature for the community to have
this choice already built directly into the Cmake step, like Ward did for
netCDF,
and John also did, I think, in the original setup, but you took off the
final distribution.

Is that right, John?
Because I think the option –enable-static/shared, is just a given thing
with the system, just a matter of adding a one liner to the script ?

So, regarding your comment

“Personally, I'm not keen on making it easy for people to statically link
to the CRT since it's a documented bad idea”

I don’t think you should be limiting our choices.

I am just asking to let us do your own choices in the way we compile our
software.

As for supporting static CRT linking in the HDF5 library, the problem we
face is support. If we supply it, people will use it, especially when it's
the "easy" thing, and then complain when it doesn't work for them. We
don't put it in CMake for the same reason you don't store knives with the
pointy ends up or allow smoking at the gas station.

Given what I've demonstrated in the attached slides, you can probably see
why I feel this way. I'm not 100% opposed, I'm just strongly opposed, and
I'd want such an option flagged so harshly that people would feel
personally insulted when they selected it. It's not enough to just
document it, because almost nobody reads documentation.

I understand that this sucks. Using standard C on Microsoft platforms has
been that way for a very long time. There's no C99 support, no standard
place to put libraries (the system directory does not count), you have to
rely on the PATH environment variable to find dlls and tools, POSIX support
barely exists, etc. AND then you have the CRT linking issue on top of it.
It's a complete nightmare, really.

Also, if I'm saying anything incorrect here, I would really appreciate
being corrected. I'd be especially keen on seeing any authoritative
sources that contradict what I've said. I've spent a lot of time digging
around for information on this topic and MS doesn't make it easy.

Cheers,

Dana

CRT State Stuff.pptx (52.7 KB)

···

On Sun, Jun 9, 2013 at 11:26 PM, Pedro Vicente <pvicente@uci.edu> wrote: