I'm looking at the preliminary work needed to add HDF5 to our project
builds, but encountered a few errors which I was hoping someone might be
able to provide pointers or suggestions for.
On Linux (Ubuntu 16.04) and MacOS X (10.11 with latest Xcode), I get a
single test failure. In both cases, I built with CMake (Unix Makefiles
on Ubuntu, Ninja on MacOS X, to make sure it wasn't a generator-specific
bug), and I didn't specify any hdf5-specific configuration options, just
the install prefix:
Start 632: H5CLEAR-clear_open_chk-latest_log_v3_F
632: Test command:
/Users/rleigh/code/ome-cmake-superbuild/h/hdf5-build/bin/clear_open_chk
"h5clear_log_v3.h5"
632: Test timeout computed to be: 600
632/2317 Test #632: H5CLEAR-clear_open_chk-latest_log_v3_F
This doesn't look platform- or compiler-specific since it fails in
exactly the same way on both. Is this a known issue?
On Windows with CMake/Ninja, I get
[4/1416] Running utility command for h5watch-files
FAILED: cmd.exe /C "cd /D
V:\ome-cmake-superbuild\b\hdf5-build\hl\tools\h5watch && "C:\Program
Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-help1.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-help1.ddl
&& ...
[ elided 10 kilobytes of commands! ]
..."C:\Program Files (x86)\CMake\bin\cmake.exe" -E
copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-ext-two-width.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-ext-two-width.ddl"
The command line is too long.
(I also had a similar failure on Linux, but don't have a log for it
right now.)
Is the CMake logic here correct? Or could this be split up into a
number of separate custom commands/targets which are run separately
rather than in a single shot which is exceeding the practical limits of
at least two systems.
Kind regards,
Roger Leigh
···
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
There is a typo in the CMake test call - it should be using the "latest"
version of the hdf5 file. This will be fixed in the next release of 1.10.
Allen
···
On Tuesday, June 07, 2016 04:16:51 PM Roger Leigh wrote:
Hi folks,
I'm looking at the preliminary work needed to add HDF5 to our project
builds, but encountered a few errors which I was hoping someone might be
able to provide pointers or suggestions for.
On Linux (Ubuntu 16.04) and MacOS X (10.11 with latest Xcode), I get a
single test failure. In both cases, I built with CMake (Unix Makefiles
on Ubuntu, Ninja on MacOS X, to make sure it wasn't a generator-specific
bug), and I didn't specify any hdf5-specific configuration options, just
the install prefix:
Start 632: H5CLEAR-clear_open_chk-latest_log_v3_F
632: Test command:
/Users/rleigh/code/ome-cmake-superbuild/h/hdf5-build/bin/clear_open_chk
"h5clear_log_v3.h5"
632: Test timeout computed to be: 600
632/2317 Test #632: H5CLEAR-clear_open_chk-latest_log_v3_F
This doesn't look platform- or compiler-specific since it fails in
exactly the same way on both. Is this a known issue?
On Windows with CMake/Ninja, I get
[4/1416] Running utility command for h5watch-files
FAILED: cmd.exe /C "cd /D
V:\ome-cmake-superbuild\b\hdf5-build\hl\tools\h5watch && "C:\Program
Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-help1.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-help1.ddl
&& ...
[ elided 10 kilobytes of commands! ]
..."C:\Program Files (x86)\CMake\bin\cmake.exe" -E
copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-ext-two-width.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-ext-two-w
idth.ddl" The command line is too long.
(I also had a similar failure on Linux, but don't have a log for it
right now.)
Is the CMake logic here correct? Or could this be split up into a
number of separate custom commands/targets which are run separately
rather than in a single shot which is exceeding the practical limits of
at least two systems.
Kind regards,
Roger Leigh
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
Thanks. Is there a commit for that I could cherry-pick, or a diff I
could pick up in the interim?
Regarding the command length overflow on Windows and Linux, I've had a
look and it's due to the amount of POST_BUILD custom commands which are
being created. These are concatenated into a single mega-command, which
exceeds the OS argv limits. The attached patch reduces it sufficiently
by converting these into regular dependencies and targets to alleviate
the problem, so they are then regular separate dependency-driven
actions, but likely needs doing across the board--it's not intended to
apply right now but rather to demonstrate a possible solution. I'd be
happy to work on a more comprehensive solution by extending this to all
custom command usage if that's useful to you.
With this patch, I can use the Ninja generator on Windows. There are
some unrelated failures I still need to investigate which I'll follow up
with separately.
If the test data is modified by the tests and needs re-copying, then
this strategy won't be sufficient on its own. But ctest runs appear to
be idempotent, so this doesn't look like an issue. If it is, then we
could do the copying in a test wrapper (see below).
Could I possibly also point you to this example:
This is a wrapper to run tests, which resolves all the needed run-time
paths via cmake generator expressions, and which makes the tests
completely generator-agnostic and work across the board on Windows. A
similar test wrapper could be written for hdf5, with a hdf5_add_test
around add_test to make use of it. Again, I'd be happy to look into
doing this if you'd find this useful.
Is hdf5 in a public VCS? I couldn't see it on your github HDFGroup
group, unless I overlooked it or it's under a different name?
There is a typo in the CMake test call - it should be using the "latest"
version of the hdf5 file. This will be fixed in the next release of 1.10.
Allen
On Tuesday, June 07, 2016 04:16:51 PM Roger Leigh wrote:
Hi folks,
I'm looking at the preliminary work needed to add HDF5 to our project
builds, but encountered a few errors which I was hoping someone might be
able to provide pointers or suggestions for.
On Linux (Ubuntu 16.04) and MacOS X (10.11 with latest Xcode), I get a
single test failure. In both cases, I built with CMake (Unix Makefiles
on Ubuntu, Ninja on MacOS X, to make sure it wasn't a generator-specific
bug), and I didn't specify any hdf5-specific configuration options, just
the install prefix:
Start 632: H5CLEAR-clear_open_chk-latest_log_v3_F
632: Test command:
/Users/rleigh/code/ome-cmake-superbuild/h/hdf5-build/bin/clear_open_chk
"h5clear_log_v3.h5"
632: Test timeout computed to be: 600
632/2317 Test #632: H5CLEAR-clear_open_chk-latest_log_v3_F
This doesn't look platform- or compiler-specific since it fails in
exactly the same way on both. Is this a known issue?
On Windows with CMake/Ninja, I get
[4/1416] Running utility command for h5watch-files
FAILED: cmd.exe /C "cd /D
V:\ome-cmake-superbuild\b\hdf5-build\hl\tools\h5watch && "C:\Program
Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-help1.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-help1.ddl
&& ...
[ elided 10 kilobytes of commands! ]
..."C:\Program Files (x86)\CMake\bin\cmake.exe" -E
copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-ext-two-width.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-ext-two-w
idth.ddl" The command line is too long.
(I also had a similar failure on Linux, but don't have a log for it
right now.)
Is the CMake logic here correct? Or could this be split up into a
number of separate custom commands/targets which are run separately
rather than in a single shot which is exceeding the practical limits of
at least two systems.
Kind regards,
Roger Leigh
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
There is not a patch and there will soon be a normal 1.10 branch in our
repository that will have that plus a number of other fixes/changes. This
should be available soon, hopefully by the end of the month. To fix the test,
edit line 253 in the CMakeTests.cmake file and add "latest-" to the h5 file
name.
I like the patch and will try to integrate and test it this month. If you have
any other CMake suggestions, please send them in, we are always looking to
improve the code.
We have a number of changes for our code and repositories coming soon, so stay
tuned.
Allen
···
On Wednesday, June 08, 2016 01:21:53 PM Roger Leigh wrote:
Thanks. Is there a commit for that I could cherry-pick, or a diff I
could pick up in the interim?
Regarding the command length overflow on Windows and Linux, I've had a
look and it's due to the amount of POST_BUILD custom commands which are
being created. These are concatenated into a single mega-command, which
exceeds the OS argv limits. The attached patch reduces it sufficiently
by converting these into regular dependencies and targets to alleviate
the problem, so they are then regular separate dependency-driven
actions, but likely needs doing across the board--it's not intended to
apply right now but rather to demonstrate a possible solution. I'd be
happy to work on a more comprehensive solution by extending this to all
custom command usage if that's useful to you.
With this patch, I can use the Ninja generator on Windows. There are
some unrelated failures I still need to investigate which I'll follow up
with separately.
If the test data is modified by the tests and needs re-copying, then
this strategy won't be sufficient on its own. But ctest runs appear to
be idempotent, so this doesn't look like an issue. If it is, then we
could do the copying in a test wrapper (see below).
https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/bzip2/patc
hes/cmake.diff#L226 This is a wrapper to run tests, which resolves all the
needed run-time paths via cmake generator expressions, and which makes the
tests
completely generator-agnostic and work across the board on Windows. A
similar test wrapper could be written for hdf5, with a hdf5_add_test
around add_test to make use of it. Again, I'd be happy to look into
doing this if you'd find this useful.
Is hdf5 in a public VCS? I couldn't see it on your github HDFGroup
group, unless I overlooked it or it's under a different name?
Kind regards,
Roger
On 07/06/2016 18:05, Allen Byrne wrote:
> There is a typo in the CMake test call - it should be using the "latest"
> version of the hdf5 file. This will be fixed in the next release of 1.10.
>
> Allen
>
We are switching to Bitbucket; it will be public after the HDF5 1.10.1 release later this summer.
Roger,
Allen will let you know when the typo fix is in.
Thank you for reporting the problem!
Elena
···
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Jun 8, 2016, at 8:21 AM, Roger Leigh <rleigh@dundee.ac.uk<mailto:rleigh@dundee.ac.uk>> wrote:
Thanks. Is there a commit for that I could cherry-pick, or a diff I
could pick up in the interim?
Regarding the command length overflow on Windows and Linux, I've had a
look and it's due to the amount of POST_BUILD custom commands which are
being created. These are concatenated into a single mega-command, which
exceeds the OS argv limits. The attached patch reduces it sufficiently
by converting these into regular dependencies and targets to alleviate
the problem, so they are then regular separate dependency-driven
actions, but likely needs doing across the board--it's not intended to
apply right now but rather to demonstrate a possible solution. I'd be
happy to work on a more comprehensive solution by extending this to all
custom command usage if that's useful to you.
With this patch, I can use the Ninja generator on Windows. There are
some unrelated failures I still need to investigate which I'll follow up
with separately.
If the test data is modified by the tests and needs re-copying, then
this strategy won't be sufficient on its own. But ctest runs appear to
be idempotent, so this doesn't look like an issue. If it is, then we
could do the copying in a test wrapper (see below).
Could I possibly also point you to this example:
This is a wrapper to run tests, which resolves all the needed run-time
paths via cmake generator expressions, and which makes the tests
completely generator-agnostic and work across the board on Windows. A
similar test wrapper could be written for hdf5, with a hdf5_add_test
around add_test to make use of it. Again, I'd be happy to look into
doing this if you'd find this useful.
Is hdf5 in a public VCS? I couldn't see it on your github HDFGroup
group, unless I overlooked it or it's under a different name?
Kind regards,
Roger
On 07/06/2016 18:05, Allen Byrne wrote:
There is a typo in the CMake test call - it should be using the "latest"
version of the hdf5 file. This will be fixed in the next release of 1.10.
Allen
On Tuesday, June 07, 2016 04:16:51 PM Roger Leigh wrote:
Hi folks,
I'm looking at the preliminary work needed to add HDF5 to our project
builds, but encountered a few errors which I was hoping someone might be
able to provide pointers or suggestions for.
On Linux (Ubuntu 16.04) and MacOS X (10.11 with latest Xcode), I get a
single test failure. In both cases, I built with CMake (Unix Makefiles
on Ubuntu, Ninja on MacOS X, to make sure it wasn't a generator-specific
bug), and I didn't specify any hdf5-specific configuration options, just
the install prefix:
Start 632: H5CLEAR-clear_open_chk-latest_log_v3_F
632: Test command:
/Users/rleigh/code/ome-cmake-superbuild/h/hdf5-build/bin/clear_open_chk
"h5clear_log_v3.h5"
632: Test timeout computed to be: 600
632/2317 Test #632: H5CLEAR-clear_open_chk-latest_log_v3_F
This doesn't look platform- or compiler-specific since it fails in
exactly the same way on both. Is this a known issue?
On Windows with CMake/Ninja, I get
[4/1416] Running utility command for h5watch-files
FAILED: cmd.exe /C "cd /D
V:\ome-cmake-superbuild\b\hdf5-build\hl\tools\h5watch && "C:\Program
Files (x86)\CMake\bin\cmake.exe" -E copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-help1.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-help1.ddl
&& ...
[ elided 10 kilobytes of commands! ]
..."C:\Program Files (x86)\CMake\bin\cmake.exe" -E
copy_if_different
V:/ome-cmake-superbuild/b/hdf5-source/hl/tools/testfiles/w-ext-two-width.ddl
V:/ome-cmake-superbuild/b/hdf5-build/hl/tools/h5watch/testfiles/w-ext-two-w
idth.ddl" The command line is too long.
(I also had a similar failure on Linux, but don't have a log for it
right now.)
Is the CMake logic here correct? Or could this be split up into a
number of separate custom commands/targets which are run separately
rather than in a single shot which is exceeding the practical limits of
at least two systems.
Kind regards,
Roger Leigh
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
We are switching to Bitbucket; it will be public after the HDF5 1.10.1
release later this summer.
Roger,
Allen will let you know when the typo fix is in.
Thank you for reporting the problem!
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks. Is there a commit for that I could cherry-pick, or a diff I
could pick up in the interim?
Regarding the command length overflow on Windows and Linux, I've had a
look and it's due to the amount of POST_BUILD custom commands which are
being created. These are concatenated into a single mega-command, which
exceeds the OS argv limits. The attached patch reduces it sufficiently
by converting these into regular dependencies and targets to alleviate
the problem, so they are then regular separate dependency-driven
actions, but likely needs doing across the board--it's not intended to
apply right now but rather to demonstrate a possible solution. I'd be
happy to work on a more comprehensive solution by extending this to all
custom command usage if that's useful to you.
With this patch, I can use the Ninja generator on Windows. There are
some unrelated failures I still need to investigate which I'll follow up
with separately.
If the test data is modified by the tests and needs re-copying, then
this strategy won't be sufficient on its own. But ctest runs appear to
be idempotent, so this doesn't look like an issue. If it is, then we
could do the copying in a test wrapper (see below).
https://github.com/ome/ome-cmake-superbuild/blob/develop/packages/bzip2/patc
hes/cmake.diff#L226 This is a wrapper to run tests, which resolves all the
needed run-time paths via cmake generator expressions, and which makes the
tests
completely generator-agnostic and work across the board on Windows. A
similar test wrapper could be written for hdf5, with a hdf5_add_test
around add_test to make use of it. Again, I'd be happy to look into
doing this if you'd find this useful.
Is hdf5 in a public VCS? I couldn't see it on your github HDFGroup
group, unless I overlooked it or it's under a different name?
Kind regards,
Roger
On 07/06/2016 18:05, Allen Byrne wrote:
There is a typo in the CMake test call - it should be using the "latest"
version of the hdf5 file. This will be fixed in the next release of 1.10.
I can't find it on bitbucket or github, nor see any pointers from your
website (even to SVN). I could easily have missed it since I'm fairly
rubbish at noticing things, so apologies if I've overlooked it.
Many thanks,
Roger
The University of Dundee is a registered Scottish Charity, No: SC015096
We are switching to Bitbucket; it will be public after the HDF5 1.10.1
release later this summer.
Did this switch occur?
I can't find it on bitbucket or github, nor see any pointers from your
website (even to SVN). I could easily have missed it since I'm fairly
rubbish at noticing things, so apologies if I've overlooked it.
Many thanks,
Roger
The University of Dundee is a registered Scottish Charity, No: SC015096
I was looking on bitbucket.org, which explains why I didn't find it. I
take it this is a custom instance, or can existing bitbucket accounts be
used here? Can anyone sign up for an account to open pull requests? Is
it also being used for issue reporting and tracking?
What is the meaning of the project names here? Why is HDFFR hdf4 and
HDFFV hdf5? Why are they not just called HDF4 and HDF5? Is
"HDFFV/hdf5" is the canonical location of the hdf5 source?
I would never have found this without your email, because I don't see
any pointers to this on your website. Would it be possible to add some
links?
It would be nice to have some useful links here. If I come to the
website looking for your source repositories, there's absolutely nothing
that I can see that takes me there.
- It would be useful to have on the download page alongside the source
release link (git tag, repo link)
- It would be useful to have as one of the community resources
- It would maybe be useful in other places as well, e.g. issue tracking
if you're using that
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
I tried to log in with my bitbucket credentials with no success, so it seems
indeed to be a private instance. The JIRA instance displays nothing publically
either.
Regards,
Pierre
···
On Mon, Nov 14, 2016 at 05:55:29PM +0000, Roger Leigh wrote:
On 14/11/16 16:42, Elena Pourmal wrote:
>Yes, switch did occur. Please see https://git.hdfgroup.org/projects. It
>looks like we failed to announce it.
OK, thanks.
I was looking on bitbucket.org, which explains why I didn't find it. I
take it this is a custom instance, or can existing bitbucket accounts be
used here? Can anyone sign up for an account to open pull requests? Is
it also being used for issue reporting and tracking?
What is the meaning of the project names here? Why is HDFFR hdf4 and
HDFFV hdf5? Why are they not just called HDF4 and HDF5? Is
"HDFFV/hdf5" is the canonical location of the hdf5 source?
I would never have found this without your email, because I don't see
any pointers to this on your website. Would it be possible to add some
links?
It would be nice to have some useful links here. If I come to the
website looking for your source repositories, there's absolutely nothing
that I can see that takes me there.
- It would be useful to have on the download page alongside the source
release link (git tag, repo link)
- It would be useful to have as one of the community resources
- It would maybe be useful in other places as well, e.g. issue tracking
if you're using that
Thanks for your consideration,
Roger
--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
College of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364
The University of Dundee is a registered Scottish Charity, No: SC015096
--
-----------------------------------------------------------
Pierre de Buyl
KU Leuven - Institute for Theoretical Physics
T +32 16 3 27355
W http://pdebuyl.be/
-----------------------------------------------------------
