Weird HDF4 Behavior on macOS

HDF4
1

I’m having a weird issue with HDF4 on macOS compared to Linux. In my libraries, I have built HDF4 4.2.15 on both machines. We first noticed this with netcdf (via ncdump, asked about Odd ncdump + hdf4 behavior on macOS · Issue #2770 · Unidata/netcdf-c · GitHub) but I managed to finagle GitHub Copilot to tell me how to do things in “pure” HDF4. To wit, on Linux:

$ hdp-hdf4 dumpsds MOD04_L2.A2017090.2000.051.NRT.hdf | grep 'Variable Name = Longitude' -A 16
Variable Name = Longitude
	 Index = 0
	 Type= 32-bit floating point
	 Ref. = 4
	 Compression method = DEFLATE
		 Deflate level = 4
	 Compression ratio (original:compressed) = 1.16:1
	 Rank = 2
	 Number of attributes = 10
	 Dim0: Name=Cell_Along_Swath:mod04
		 Size = 203
		 Scale Type = number-type not set
		 Number of attributes = 0
	 Dim1: Name=Cell_Across_Swath:mod04
		 Size = 135
		 Scale Type = number-type not set
		 Number of attributes = 0

First, it’s called hdp-hdf4 for historic reasons (I inherited our library build system and it does --program-suffix="-hdf4"). But, focus on the Sizes of the two dims: 203 and 135.

Now, I run the same command on the same file on macOS:

❯ hdp-hdf4 dumpsds MOD04_L2.A2017090.2000.051.NRT.hdf | grep 'Variable Name = Longitude' -A 16
in sdsdumpfull: Attempting to allocate 0 items using 'read_nelts'!
Variable Name = Longitude
	 Index = 0
	 Type= 32-bit floating point
	 Ref. = 4
	 Compression method = DEFLATE
		 Deflate level = 4
	 Compression ratio (original:compressed) = 1.16:1
	 Rank = 2
	 Number of attributes = 10
	 Dim0: Name=Cell_Along_Swath:mod04
		 Size = -889192448
		 Scale Type = number-type not set
		 Number of attributes = 0
	 Dim1: Name=Cell_Across_Swath:mod04
		 Size = -2030043136
		 Scale Type = number-type not set
		 Number of attributes = 0

The sizes went nuts. ncdump is similar:

$ ncdump -hsc MOD04_L2.A2017090.0010.051.NRT.hdf | head
netcdf MOD04_L2.A2017090.0010.051.NRT {
dimensions:
	Cell_Along_Swath\:mod04 = 18446744072820359168 ;
	Cell_Across_Swath\:mod04 = 18446744071679508480 ;
	Solution_3_Land\:mod04 = 50331648 ;
	Solution_1_Land\:mod04 = 33554432 ;
	Solution_2_Land\:mod04 = 50331648 ;
	Solution_4_Land\:mod04 = 67108864 ;
	MODIS_Band_Land\:mod04 = 117440512 ;
	QA_Byte_Land\:mod04 = 83886080 ;

Per that GitHub Issue, it’s like an endianness problem, but all the configure output I see for hdf4 says it knows we aren’t on a big-endian platform!

Any ideas?

2

@matthew.thompson Please try the latest version of HDF4: HDF 4.2.16-2 and let us know if you still see such issue. Thanks!

3

@bmribler Well, I just grabbed that version and I can’t build it. :frowning:

I’m getting:

Making install in mfhdf
Making install in libsrc
  CC       array.lo
In file included from array.c:18:
./local_nc.h:473:51: error: unknown type name 'u_int'
HDFLIBAPI bool_t xdr_shorts(XDR *xdrs, short *sp, u_int cnt);
                                                  ^

I’m wondering if now I need extra flags if building with Clang on macOS? If I look online I see things like:

Note that I think before I was using 4.2.15 and I tried to take a look but GitHub says between 15 and 16-2:

Showing 921 changed files with 146,037 additions and 192,699 deletions.

Eep.

4

Oh. This might be Build fails on x86_64 OpenBSD 7.2 clang · Issue #151 · HDFGroup/hdf4 · GitHub

I’ll try and see if I can build a gcc+gfortran build instead of clang+gfortran to get around this. The code I work on is 98% Fortran or the like, so the C compiler often doesn’t matter to us.

5

I believe that Fortran is limited to 32-bit builds because of the use of pointers in the Fortran interface - so until we can rework the Fortran code, we recommend not using Fortran. There might even be restrictions on compiling hdf4 with Fortran on macs in the code somewhere?

6

Well, that latest error is a clang error, not a Fortran one. I don’t think the build even got to the Fortran bits yet.

7

We do build with clang on macs (no Fortran) with CMake.

8

Autotools configure: Cannot build shared fortran libraries. Please configure with --disable-fortran flag

Looks like that is all platforms. I think CMake allows it.

9

@byrn Configuring with:

          ./configure --prefix=$(prefix) \
                      --program-suffix="-hdf4"\
                      --includedir=$(prefix)/include/hdf \
                      --with-jpeg=$(prefix)/include/jpeg,$(prefix)/lib \
                      --with-szlib=$(prefix)/include/szlib,$(prefix)/lib \
                      --with-zlib=$(prefix)/include/zlib,$(prefix)/lib \
                      --disable-netcdf \
                      --disable-fortran \
                      CFLAGS="$(CFLAGS) $(NO_IMPLICIT_FUNCTION_ERROR) $(NO_IMPLICIT_INT_ERROR)" CC=$(CC) CXX=$(CXX) )

So no reference to fortran and --disable-fortran. I also do:

unset F90 FC

so that isn’t in my environment.

And during configure, I see:

Languages:
----------
                        Fortran: no

                           Java: no

I get:

Making install in mfhdf
Making install in libsrc
  CC       array.lo
In file included from array.c:18:
./local_nc.h:473:51: error: unknown type name 'u_int'
HDFLIBAPI bool_t xdr_shorts(XDR *xdrs, short *sp, u_int cnt);
                                                  ^
10

Could you also add: --enable-hdf4-xdr for Mac?

1 Like
11

Ahhh! That seems to be the magic sauce! Indeed, I went back to my original configure, and just added that, and it works.

Plus, the HDF4 file is now read correctly on macOS!

For my own edification: what does that do? :slight_smile:

ETA: Also: do you suspect this is only needed on macOS? Or should I enable it on Linux as well? Just wondering if how general to make the configure line.

12

The enable simply builds a local version of the xdr lib, which is based on the tirpc library usually distributed by unix/linix. It use to be the sun rpc library before that was removed by distributions.

We build with it on on all platforms as it simplifies the config lines.