HDF Newsletter #129

bljones · November 13, 2012, 8:18pm

To view from a browser, see:
http://www.hdfgroup.org/newsletters/newsletter129.html

···

--------------------------------------------------------------------------
The HDF Group Home Page: http://hdfgroup.org/
Helpdesk & Mailing Lists Info: http://hdfgroup.org/services/support.html
--------------------------------------------------------------------------

Newsletter #129
November 13, 2012

CONTENTS:

  - Release of HDF5-1.8.10
    . New Features and Changes
    . Future Changes to Supported Platforms

- Release of HDF4 File Content Map Writer 1.0.4

- Retirement of HDF-EOS5 to NetCDF4 Converter (eos5nc4)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Release of HDF5-1.8.10

The HDF5-1.8.10 release is now available from The HDF Group Downloads page:

http://www.hdfgroup.org/downloads/

It can also be obtained directly from:

http://www.hdfgroup.org/HDF5/release/obtain5.html

The HDF5-1.8.10 documentation is located in:

http://www.hdfgroup.org/HDF5/doc/

For information on HDF5, see the HDF5 home page:

http://www.hdfgroup.org/products/hdf5/index.html

New Features and Changes
------------------------

HDF5-1.8.10 is a minor release, but contains several important new
features and bug fixes. Changes in this release include the following:

o Two new query APIs were added to the Parallel library:

H5Pget_mpio_no_collective_cause - retrieves reasons why the
collective I/O was broken during read/write I/O access.

     H5Pget_mpio_actual_io_mode_f - Fortran counterpart to
          H5Pget_mpio_actual_io_mode (already in library for retrieving
          the parallel I/O type performed).

o The h5import utility now allows the use of h5dump output as input
into h5import.

o The conversion tests no longer fail in dt_arith.c on Mac OS X 10.7
or 10.8.

  o The windows pre-built binaries are now provided with debug information.
    This enables users to debug their HDF5 applications using the pre-built
    binaries.

This release contains many other changes and bug fixes not listed here,
including fixes to various tools. Please be sure to read the Release
Notes for a comprehensive list of new features and bug fixes:

http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.10-RELEASE.txt

Future Changes to Supported Platforms
-------------------------------------

We will be dropping support for the following platforms after this
release:

o VS 2008 and Windows XP
o Mac OS X Snow Leopard 10.6.8 32-bit

Also, we will no longer provide the VS project files for Windows after
this release, and will be moving to CMake 2.8.10.

Release of HDF4 File Content Map Writer 1.0.4

We are pleased to announce the release of HDF4 File Content Map Writer
version 1.0.4.

This release provides a fix for handling some NASA OCTS Level 3
products (e.g., O19963061996335.L3m_MO_CHLO) that involve palettes.

Please see the h4mapwriter web page for complete details regarding this
release:

http://www.hdfgroup.org/projects/h4map/h4map_writer.html

Retirement of HDF-EOS5 to NetCDF4 Converter (eos5nc4)

The HDF Group plans to retire the HDF-EOS5 to NetCDF-4 Converter (eos5nc4)
tool effective Dec 31, 2012.

If you are using the eos5nc4 tool and would like for it to continue to be
maintained, please either send a message to eoshelp@hdfgroup.org or post
a message to the HDF-EOS Forum by no later than December 19, 2012.

Please see the HDF-EOS Forum announcement for more information:

http://www.hdfeos.org/forums/showthread.php?p=1254#post1254

-------------------------------------------------------------
For questions regarding these or other HDF issues, contact:

help@hdfgroup.org

mike.jackson · November 14, 2012, 3:03am

Hello,
First congratulations on the new release. I would like some clarifications on the "Supported" platform changes. Specifically the "VS2008 and Windows XP". I can read that in 2 different ways:

1: The *combination* of VS2008 running on Windows XP is deprecated
2: VS2008 on *any* version of Windows AND Windows XP

Thanks

···

___________________________________________________________
Mike Jackson Principal Software Engineer
BlueQuartz Software Dayton, Ohio
mike.jackson@bluequartz.net www.bluequartz.net

On Nov 13, 2012, at 3:18 PM, Barbara Jones wrote:

To view from a browser, see:
http://www.hdfgroup.org/newsletters/newsletter129.html

--------------------------------------------------------------------------
The HDF Group Home Page: http://hdfgroup.org/
Helpdesk & Mailing Lists Info: http://hdfgroup.org/services/support.html
--------------------------------------------------------------------------

                           Newsletter #129
                          November 13, 2012

CONTENTS:

- Release of HDF5-1.8.10
  . New Features and Changes
  . Future Changes to Supported Platforms

- Release of HDF4 File Content Map Writer 1.0.4

- Retirement of HDF-EOS5 to NetCDF4 Converter (eos5nc4)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Release of HDF5-1.8.10

The HDF5-1.8.10 release is now available from The HDF Group Downloads page:

   http://www.hdfgroup.org/downloads/

It can also be obtained directly from:

   http://www.hdfgroup.org/HDF5/release/obtain5.html

The HDF5-1.8.10 documentation is located in:

   http://www.hdfgroup.org/HDF5/doc/

For information on HDF5, see the HDF5 home page:

   http://www.hdfgroup.org/products/hdf5/index.html

New Features and Changes
------------------------

HDF5-1.8.10 is a minor release, but contains several important new
features and bug fixes. Changes in this release include the following:

o Two new query APIs were added to the Parallel library:

   H5Pget_mpio_no_collective_cause - retrieves reasons why the
        collective I/O was broken during read/write I/O access.

   H5Pget_mpio_actual_io_mode_f - Fortran counterpart to
        H5Pget_mpio_actual_io_mode (already in library for retrieving
        the parallel I/O type performed).

o The h5import utility now allows the use of h5dump output as input
  into h5import.

o The conversion tests no longer fail in dt_arith.c on Mac OS X 10.7
  or 10.8.

o The windows pre-built binaries are now provided with debug information.
  This enables users to debug their HDF5 applications using the pre-built
  binaries.

This release contains many other changes and bug fixes not listed here,
including fixes to various tools. Please be sure to read the Release
Notes for a comprehensive list of new features and bug fixes:

  http://www.hdfgroup.org/ftp/HDF5/current/src/hdf5-1.8.10-RELEASE.txt

Future Changes to Supported Platforms
-------------------------------------

We will be dropping support for the following platforms after this
release:

o VS 2008 and Windows XP
o Mac OS X Snow Leopard 10.6.8 32-bit

Also, we will no longer provide the VS project files for Windows after
this release, and will be moving to CMake 2.8.10.

Release of HDF4 File Content Map Writer 1.0.4

We are pleased to announce the release of HDF4 File Content Map Writer
version 1.0.4.

This release provides a fix for handling some NASA OCTS Level 3
products (e.g., O19963061996335.L3m_MO_CHLO) that involve palettes.

Please see the h4mapwriter web page for complete details regarding this
release:

     http://www.hdfgroup.org/projects/h4map/h4map_writer.html

Retirement of HDF-EOS5 to NetCDF4 Converter (eos5nc4)

The HDF Group plans to retire the HDF-EOS5 to NetCDF-4 Converter (eos5nc4)
tool effective Dec 31, 2012.

If you are using the eos5nc4 tool and would like for it to continue to be
maintained, please either send a message to eoshelp@hdfgroup.org or post
a message to the HDF-EOS Forum by no later than December 19, 2012.

Please see the HDF-EOS Forum announcement for more information:

http://www.hdfeos.org/forums/showthread.php?p=1254#post1254

-------------------------------------------------------------
For questions regarding these or other HDF issues, contact:

  help@hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

derobins · November 14, 2012, 4:42am

Hi Mike,

Hello,
First congratulations on the new release. I would like some
clarifications on the "Supported" platform changes. Specifically the
"VS2008 and Windows XP". I can read that in 2 different ways:

1: The *combination* of VS2008 running on Windows XP is deprecated
2: VS2008 on *any* version of Windows AND Windows XP

Unless I am misinformed,

- VS2008 will not be supported.
- Windows XP will not be supported.

From what I know if the code, I'd imagine this shouldn't be an issue for

most people. It mainly affects what binaries we produce and our test
systems. CMake will still produce VS2008 projects and we don't have very
much Windows-specific code. I can't think of anything off the top of my
head that won't work on any NT-based Windows system.

Dana

···

On Tue, Nov 13, 2012 at 9:03 PM, Michael Jackson < mike.jackson@bluequartz.net> wrote:

epourmal · November 14, 2012, 3:52pm

Mike,

Good questions. Thank you for asking and giving us a chance to clarify.

···

On Nov 13, 2012, at 10:42 PM, Dana Robinson wrote:

Hi Mike,

On Tue, Nov 13, 2012 at 9:03 PM, Michael Jackson <mike.jackson@bluequartz.net> wrote:
Hello,
First congratulations on the new release. I would like some clarifications on the "Supported" platform changes. Specifically the "VS2008 and Windows XP". I can read that in 2 different ways:

1: The *combination* of VS2008 running on Windows XP is deprecated
2: VS2008 on *any* version of Windows AND Windows XP

Unless I am misinformed,

- VS2008 will not be supported.
- Windows XP will not be supported.

Dana's statement is correct.

We will try to make our announcements less ambiguous next time!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820

(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

From what I know if the code, I'd imagine this shouldn't be an issue for most people. It mainly affects what binaries we produce and our test systems. CMake will still produce VS2008 projects and we don't have very much Windows-specific code. I can't think of anything off the top of my head that won't work on any NT-based Windows system.

Dana
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

andrew.collette · November 16, 2012, 5:33pm

Hi all,

Unless I am misinformed,

- VS2008 will not be supported.
- Windows XP will not be supported.

From what I know if the code, I'd imagine this shouldn't be an issue for
most people. It mainly affects what binaries we produce and our test
systems. CMake will still produce VS2008 projects and we don't have very
much Windows-specific code. I can't think of anything off the top of my
head that won't work on any NT-based Windows system.

I just wanted to mention that dropping VS2008 support is a very
serious issue for anyone working with Python. All major Python
versions currently available (2.6, 2.7 and 3.2) are built with VS2008.
Python extension modules (and any of their dependencies which share C
standard library bits like malloc and free) have to be built with the
same compiler in order to avoid mixing C runtimes. Right now h5py
(and I would expect, PyTables) is built against a version of HDF5
compiled with VS2008 for this reason. In particular, if you allocate
any memory on the Python side which is freed by HDF5 (or vice versa)
you have to use the same compiler.

In practical terms, what this means for us is that the version of HDF5
shipping with h5py for Python 2.6, 2.7 and 3.2 will be frozen at
1.8.10. In particular, HDF5 1.10 will also be unsupported on all
currently available Python platforms, unless we can figure out a way
around the C runtime issue.

Would the HDF Group accept bug reports/patches from the community to
fix issues with VS2008 as they come up?

Andrew

derobins · November 16, 2012, 5:45pm

Hi all,

I just wanted to mention that dropping VS2008 support is a very

serious issue for anyone working with Python. All major Python
versions currently available (2.6, 2.7 and 3.2) are built with VS2008.
Python extension modules (and any of their dependencies which share C
standard library bits like malloc and free) have to be built with the
same compiler in order to avoid mixing C runtimes. Right now h5py
(and I would expect, PyTables) is built against a version of HDF5
compiled with VS2008 for this reason. In particular, if you allocate
any memory on the Python side which is freed by HDF5 (or vice versa)
you have to use the same compiler.

Ah, I hadn't thought of that. I'll mention that to the appropriate people.

Does anyone know when the Python folks plan to upgrade to VS2010? VS2008
is four years old and is now the n-2 version. That's getting a little long
in the tooth.

In practical terms, what this means for us is that the version of HDF5
shipping with h5py for Python 2.6, 2.7 and 3.2 will be frozen at
1.8.10. In particular, HDF5 1.10 will also be unsupported on all
currently available Python platforms, unless we can figure out a way
around the C runtime issue.

There is no way around the C runtime issue that I know of.

Do people who use h5py, etc. use our binaries (which I assume) or do they
build HDF5 via CMake?

Would the HDF Group accept bug reports/patches from the community to
fix issues with VS2008 as they come up?

I would assume so, but I don't think we've had a lot of problems with
VS-specific issues in HDF5. Usually Windows bugs apply to all versions of
Visual Studio and Windows.

Cheers,

Dana

andre · November 16, 2012, 5:52pm

Do people who use h5py, etc. use our binaries (which I assume) or do they build HDF5 via CMake?

I use the Enthought python distribution - which builds both h5py and PyTables as well as HDF5.

http://www.enthought.com/

unfortunately, I am not sure how they build them. On my Mac OS, it is just a package installer.
If no one else knows, I can ask one of the Enthought folks. I suspect there are reasonable number of people who use Enthought to get their HDF5/h5py/PyTables builds (for those who don't know - they make installers for many platforms, Mac OS, Linux, Windows) and they offer their build for free to anyone with a .edu email address.

Andre

andrew.collette · November 16, 2012, 5:57pm

Hi Dana,

Ah, I hadn't thought of that. I'll mention that to the appropriate people.

Does anyone know when the Python folks plan to upgrade to VS2010? VS2008 is
four years old and is now the n-2 version. That's getting a little long in
the tooth.

The newest version of Python (3.3) is built with VS2010. I think they
stuck with VS2008 for so long because of the extensions issue.

There is no way around the C runtime issue that I know of.

Do people who use h5py, etc. use our binaries (which I assume) or do they
build HDF5 via CMake?

On Windows, we ship a copy of HDF5 along with the h5py installer,
which is built by the h5py team. It's customized to disable the SZIP
writer for licensing issues, and to rename the DLL to avoid a conflict
with PyTables. Right now that version is built with VS2008 (and on a
WinXP build node, for added irony). I build from the included project
file at the moment but I'm fine with switching to CMake.

I would assume so, but I don't think we've had a lot of problems with
VS-specific issues in HDF5. Usually Windows bugs apply to all versions of
Visual Studio and Windows.

If this is the case, may I ask what the motivation is to drop VS2008
support? Is it to reduce the testing burden? I understand the HDF
Group can't support every old compiler forever, but I would certainly
be willing to chip in with testing, etc. because this is such an
important part of our infrastructure.

Andrew

derobins · November 16, 2012, 6:07pm

Hi Andrew,

On Windows, we ship a copy of HDF5 along with the h5py installer,

which is built by the h5py team. It's customized to disable the SZIP
writer for licensing issues, and to rename the DLL to avoid a conflict
with PyTables. Right now that version is built with VS2008 (and on a
WinXP build node, for added irony). I build from the included project
file at the moment but I'm fine with switching to CMake.

You'll have to switch to CMake for the next release since we've dropped the
VS projects and solutions for the next release. CMake is actually a much,
much better build environment for HDF5. The projects and solutions we'd
been including with the source distribution had been deprecated for some
time.

> I would assume so, but I don't think we've had a lot of problems with
> VS-specific issues in HDF5. Usually Windows bugs apply to all versions
of
> Visual Studio and Windows.

If this is the case, may I ask what the motivation is to drop VS2008
support? Is it to reduce the testing burden? I understand the HDF
Group can't support every old compiler forever, but I would certainly
be willing to chip in with testing, etc. because this is such an
important part of our infrastructure.

It's mainly a test and build issue. It costs time and resources to build
and test on a large number of platforms and we try to keep the load due to
Windows small since we receive no funding for Windows development. We do
have CDash set up to track testing by external parties. I can put you in
touch with Allen if you'd like to investigate that.

Cheers,

Dana

Dr_Mike_Mueller · November 16, 2012, 10:18pm

If would like to use a different VS version, you would need to compile
Python itself and all C extension with this version. So even if you would
get Python to compile with a different VS version, which is rather unlikely,
you still have a lot of trouble compiling *all* other C extension packages.
Since under Windows binary installs of C extension modules are very common,
this makes this option practically impossible if you don't want to spent an
unreasonable amount of time on it, not talking about the expertise you need
for this.

I tend to use MinGW32 (http://www.mingw.org/) to compile C extension for
Python on Windows. I know from experience it can compile and link against
VS2003 and VS2008 for different Python versions. I remember doing it at least
for 2.5, 2.6 and 2.7 using source code distributions of packages with pip or
setuptools/distribute. MinGW32 can figure out how to link against the right
library version.

PyTables has a make file for Windows that uses MinGW32 and supports several
Python versions as well 32 and 64 bit platforms:
https://github.com/PyTables/PyTables/blob/develop/mswindows/Makefile_windows

So one solution would be using MinGW32. In the ideal case you only make MinGW32
your default complier and than use the normal setup.py you used with VS.
Well, if there is a problem you need to solve it.
There my be issues with lib and include paths and such. Which often can be
solved setting some environmental variables.

I use a file called pydistutils.cfg in my home directory which contains:

[build]
compiler=mingw32

This works for me. Alternatively, you can specify the compiler as an
command line argument:

setup.py build_ext --compiler mingw32

HTH,
Mike

···

Am 16.11.12 18:57, schrieb Andrew Collette:

Hi Dana,

Ah, I hadn't thought of that. I'll mention that to the appropriate people.

Does anyone know when the Python folks plan to upgrade to VS2010? VS2008 is
four years old and is now the n-2 version. That's getting a little long in
the tooth.

The newest version of Python (3.3) is built with VS2010. I think they
stuck with VS2008 for so long because of the extensions issue.

epourmal · November 16, 2012, 6:48pm

Andrew and all,

Thank you for bringing up the issue of VS2008 support. As Dana mentioned we don't have capacity to support all compilers, but we definitely need to find a solution to support HDF5 Python users.

Please give me some time to think what we can do and how we can approach the problem. Hopefully there will be some resolution after the Holidays.

Thank you!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820

(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Nov 16, 2012, at 12:07 PM, Dana Robinson wrote:

Hi Andrew,

On Windows, we ship a copy of HDF5 along with the h5py installer,
which is built by the h5py team. It's customized to disable the SZIP
writer for licensing issues, and to rename the DLL to avoid a conflict
with PyTables. Right now that version is built with VS2008 (and on a
WinXP build node, for added irony). I build from the included project
file at the moment but I'm fine with switching to CMake.

You'll have to switch to CMake for the next release since we've dropped the VS projects and solutions for the next release. CMake is actually a much, much better build environment for HDF5. The projects and solutions we'd been including with the source distribution had been deprecated for some time.

> I would assume so, but I don't think we've had a lot of problems with
> VS-specific issues in HDF5. Usually Windows bugs apply to all versions of
> Visual Studio and Windows.

If this is the case, may I ask what the motivation is to drop VS2008
support? Is it to reduce the testing burden? I understand the HDF
Group can't support every old compiler forever, but I would certainly
be willing to chip in with testing, etc. because this is such an
important part of our infrastructure.

It's mainly a test and build issue. It costs time and resources to build and test on a large number of platforms and we try to keep the load due to Windows small since we receive no funding for Windows development. We do have CDash set up to track testing by external parties. I can put you in touch with Allen if you'd like to investigate that.

Cheers,

Dana

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

andrew.collette · November 16, 2012, 10:32pm

Hi Mike,

I tend to use MinGW32 (http://www.mingw.org/) to compile C extension for
Python on Windows. I know from experience it can compile and link against
VS2003 and VS2008 for different Python versions. I remember doing it at least
for 2.5, 2.6 and 2.7 using source code distributions of packages with pip or
setuptools/distribute. MinGW32 can figure out how to link against the right
library version.

Thanks for the info! But in this case I think we have a slightly
different problem. The generally available Python builds (from
Python.org and distributors) are VS2008, and if HDF5 itself isn't
built with VS2008, we have two separate versions of malloc and free
from the two separate msvcrt*.dll runtimes. At the end of the day
what we really need is *HDF5* built with the 2008 runtime dll. This
is why dropping VS2008 support in HDF5 is concerning.

Andrew

Dr_Mike_Mueller · November 16, 2012, 10:56pm

Hi Andrew,

Hi Mike,

I tend to use MinGW32 (http://www.mingw.org/) to compile C extension for
Python on Windows. I know from experience it can compile and link against
VS2003 and VS2008 for different Python versions. I remember doing it at least
for 2.5, 2.6 and 2.7 using source code distributions of packages with pip or
setuptools/distribute. MinGW32 can figure out how to link against the right
library version.

Thanks for the info! But in this case I think we have a slightly
different problem. The generally available Python builds (from
Python.org and distributors) are VS2008, and if HDF5 itself isn't
built with VS2008, we have two separate versions of malloc and free
from the two separate msvcrt*.dll runtimes. At the end of the day
what we really need is *HDF5* built with the 2008 runtime dll. This
is why dropping VS2008 support in HDF5 is concerning.

I see your point. Your are still at the HDF5 level not at the Python
extension level yet. But maybe you can still look at how MinGW32
compiles Python extensions. It links against the right VS version
run time libraries. So using MinGW32 to compile HDF5 itself and set the
lib and include paths to VS2008 might work. So dropping VS2008 support
would mean no VS project file. But a make (or cmake) file that takes
care of all the compiler flags and linking issues to build against
VS2008 run time libraries might work. Does it make sense or am I am
making things too simple here?

Mike

···

Am 16.11.12 23:32, schrieb Andrew Collette: