[PATCH] Relative filenames for external files

Hdf-forum archives
1

Hello,

currently relative file names set with H5Pset_external() will be interpreted as relative to the current directory.

The attached patch (against SVN trunk r24959) changes this to relative to the HDF5 file, which is probably what most users want when they use H5Pset_external().

Best regards,
Steffen Kie�

relative-external-files.patch (8.04 KB)

2

Hi Steffen,

Thank you for the patch. I created an issue in our JIRA database HDFFV-8740. Unfortunately, we had code freeze for 1.8.13 already, and will not be able to consider it for the upcoming release, but we will try to do our best fro the next one.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 4, 2014, at 10:10 AM, Steffen Kieß <Steffen.Kiess@ipvs.uni-stuttgart.de> wrote:

Hello,

currently relative file names set with H5Pset_external() will be interpreted as relative to the current directory.

The attached patch (against SVN trunk r24959) changes this to relative to the HDF5 file, which is probably what most users want when they use H5Pset_external().

Best regards,
Steffen Kieß
<relative-external-files.patch>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

3

Hello,

Hi Steffen,

Thank you for the patch. I created an issue in our JIRA database
HDFFV-8740. Unfortunately, we had code freeze for 1.8.13 already, and
will not be able to consider it for the upcoming release, but we will
try to do our best fro the next one.

what's the status on this? The patch still applies cleanly to the current SVN trunk. Is there anything I can do to help?

Best regards,
Steffen Kie�

···

On 2014-04-10 21:14, Elena Pourmal wrote:

On Apr 4, 2014, at 10:10 AM, Steffen Kie� > <Steffen.Kiess@ipvs.uni-stuttgart.de > <mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

Hello,

currently relative file names set with H5Pset_external() will be
interpreted as relative to the current directory.

The attached patch (against SVN trunk r24959) changes this to relative
to the HDF5 file, which is probably what most users want when they use
H5Pset_external().

Best regards,
Steffen Kie�
<relative-external-files.patch>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

4

Hi Steffen,

Hello,

Hi Steffen,

Thank you for the patch. I created an issue in our JIRA database
HDFFV-8740. Unfortunately, we had code freeze for 1.8.13 already, and
will not be able to consider it for the upcoming release, but we will
try to do our best fro the next one.

what's the status on this? The patch still applies cleanly to the current SVN trunk. Is there anything I can do to help?

Thank you for your question! It was very timely.

Your patch was reviewed before 1.8.14 release but didn’t make it into the 1.8.14 release due to a few missing items (see below).

Would it be possible for you to create:
   1. A test for this new functionality that passes at least on Linux and Windows.
      Please take a look at the test/dset.c and test/external.c files and add the new tests where they fit best. Separate test file is also OK, but then you will need to modify
      autoconf and CMake related files, and I don’t think it makes sense in this particular case.
   2. Add the tests to the h5dump, h5ls, h5diff and h5repack tools.
   3. A short paragraph that describes the issue and the fix for it for the RELEASE.txt file.
   4. Enhancements to the HDF5 Reference Manual and User’s Guide entries for H5Pset_external where applicable.

Code freeze for 1.8.15 is April 1, 2015. We will need the patches for 1. and 2. before March 25 to tests it on the platforms beyond Linux and Windows.
Documentation patches should be provided by April 15.

Please let me know if you have any questions.

Thank you!

Elena

···

On Mar 6, 2015, at 4:31 AM, Steffen Kieß <Steffen.Kiess@ipvs.uni-stuttgart.de> wrote:

On 2014-04-10 21:14, Elena Pourmal wrote:

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Best regards,
Steffen Kieß

On Apr 4, 2014, at 10:10 AM, Steffen Kieß >> <Steffen.Kiess@ipvs.uni-stuttgart.de >> <mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

Hello,

currently relative file names set with H5Pset_external() will be
interpreted as relative to the current directory.

The attached patch (against SVN trunk r24959) changes this to relative
to the HDF5 file, which is probably what most users want when they use
H5Pset_external().

Best regards,
Steffen Kieß
<relative-external-files.patch>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org <mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

5

Hello,

Your patch was reviewed before 1.8.14 release but didn�t make it into the 1.8.14 release due to a few missing items (see below).

I've attached a new version of the patch.

Would it be possible for you to create:
    1. A test for this new functionality that passes at least on Linux and Windows.
       Please take a look at the test/dset.c and test/external.c files and add the new tests where they fit best. Separate test file is also OK, but then you will need to modify
       autoconf and CMake related files, and I don�t think it makes sense in this particular case.

I've added tests in test/external.c which test absolute and relative filenames and on windows path-absolute (\dir\path).

The tests work on Linux and Windows.

    2. Add the tests to the h5dump, h5ls, h5diff and h5repack tools.

I'm not sure what test to add here, as the functionality of these tools is not affected. (The output of h5dump / h5ls / h5diff should not change in any way, and h5repack should not do anything different.)

    3. A short paragraph that describes the issue and the fix for it for the RELEASE.txt file.

I've added that.

    4. Enhancements to the HDF5 Reference Manual and User�s Guide entries for H5Pset_external where applicable.

(I'm not sure where the sources for the Reference Manual and the User's Guide are, so I'm not sure who will apply these changes / how I can submit patches for that.)

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetExternal

A paragraph can be added to the Description:

If 'name' is a relative filename, it will be considered relative to the HDF5 file. In this case when the HDF5 file is moved, the external files should be moved along with it.

And to History:
1.8.15 Relative filenames are now considered relative to the HDF5 file, not to the current directory.

http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html

To last paragraph of "5.5.4. External Storage Properties":
All referenced external data files must exist before performing raw data I/O on the dataset. This is normally not a problem since those files are being managed directly by the application or indirectly through some other library. However, if the file is transferred from its original context, care must be taken to assure that all the external files are accessible in the new location.

Add:
If relative filenames are used, they will be considered relative to the location of the HDF5 file.

Code freeze for 1.8.15 is April 1, 2015. We will need the patches for 1. and 2. before March 25 to tests it on the platforms beyond Linux and Windows.
Documentation patches should be provided by April 15.

Best regards,
Steffen Kie�

relative-external-files-v2.patch (16.4 KB)

···

On 2015-03-08 22:52, Elena Pourmal wrote:

6

Steffen,

Thank you for the new patch! I’ll let you know if we have any questions.

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 11, 2015, at 1:28 PM, Steffen Kieß <Steffen.Kiess@ipvs.uni-stuttgart.de<mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

(I'm not sure where the sources for the Reference Manual and the User's Guide are, so I'm not sure who will apply these changes / how I can submit patches for that.)

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetExternal

A paragraph can be added to the Description:

If 'name' is a relative filename, it will be considered relative to the HDF5 file. In this case when the HDF5 file is moved, the external files should be moved along with it.

And to History:
1.8.15 Relative filenames are now considered relative to the HDF5 file, not to the current directory.

http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html

To last paragraph of "5.5.4. External Storage Properties":
All referenced external data files must exist before performing raw data I/O on the dataset. This is normally not a problem since those files are being managed directly by the application or indirectly through some other library. However, if the file is transferred from its original context, care must be taken to assure that all the external files are accessible in the new location.

Add:
If relative filenames are used, they will be considered relative to the location of the HDF5 file.

7

Hello,

Steffen,

Thank you for the new patch! I�ll let you know if we have any questions.

is there any update on this? Is there anything I can do to help?

Best regards,
Steffen Kie�

···

On 2015-03-12 00:08, Elena Pourmal wrote:

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 11, 2015, at 1:28 PM, Steffen Kie� > <Steffen.Kiess@ipvs.uni-stuttgart.de > <mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

(I'm not sure where the sources for the Reference Manual and the
User's Guide are, so I'm not sure who will apply these changes / how I
can submit patches for that.)

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetExternal

A paragraph can be added to the Description:

If 'name' is a relative filename, it will be considered relative to
the HDF5 file. In this case when the HDF5 file is moved, the external
files should be moved along with it.

And to History:
1.8.15 Relative filenames are now considered relative to the HDF5
file, not to the current directory.

http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html

To last paragraph of "5.5.4. External Storage Properties":
All referenced external data files must exist before performing raw
data I/O on the dataset. This is normally not a problem since those
files are being managed directly by the application or indirectly
through some other library. However, if the file is transferred from
its original context, care must be taken to assure that all the
external files are accessible in the new location.

Add:
If relative filenames are used, they will be considered relative to
the location of the HDF5 file.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

8

Hi Steffen,

···

On Mar 30, 2015, at 2:49 AM, Steffen Kieß <Steffen.Kiess@ipvs.uni-stuttgart.de> wrote:

Hello,

On 2015-03-12 00:08, Elena Pourmal wrote:

Steffen,

Thank you for the new patch! I’ll let you know if we have any questions.

is there any update on this? Is there anything I can do to help?

Yes, there is.

On Friday, March 27, we reviewed the patch and concluded that we cannot accept it as it is. Applications that rely on the current behavior will be broken.
Would you like to have a telecon with us to discuss the issue and see how we can move forward with the fix? We really appreciate your effort and would like to resolve the issue.

Thank you!

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Best regards,
Steffen Kieß

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Mar 11, 2015, at 1:28 PM, Steffen Kieß >> <Steffen.Kiess@ipvs.uni-stuttgart.de >> <mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

(I'm not sure where the sources for the Reference Manual and the
User's Guide are, so I'm not sure who will apply these changes / how I
can submit patches for that.)

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetExternal

A paragraph can be added to the Description:

If 'name' is a relative filename, it will be considered relative to
the HDF5 file. In this case when the HDF5 file is moved, the external
files should be moved along with it.

And to History:
1.8.15 Relative filenames are now considered relative to the HDF5
file, not to the current directory.

http://www.hdfgroup.org/HDF5/doc/UG/10_Datasets.html

To last paragraph of "5.5.4. External Storage Properties":
All referenced external data files must exist before performing raw
data I/O on the dataset. This is normally not a problem since those
files are being managed directly by the application or indirectly
through some other library. However, if the file is transferred from
its original context, care must be taken to assure that all the
external files are accessible in the new location.

Add:
If relative filenames are used, they will be considered relative to
the location of the HDF5 file.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

9

Hello,

Hi Steffen,

Hello,

Steffen,

Thank you for the new patch! I�ll let you know if we have any questions.

is there any update on this? Is there anything I can do to help?

Yes, there is.

On Friday, March 27, we reviewed the patch and concluded that we cannot accept it as it is. Applications that rely on the current behavior will be broken.
Would you like to have a telecon with us to discuss the issue and see how we can move forward with the fix? We really appreciate your effort and would like to resolve the issue.

A teleconference would be a possibility, but as I'm living in Europe there might be some problems scheduling it.

If you want to avoid changing the behavior for existing applications, I can update the patch to:
- Add a new dataset access property which controls how resolving external files behaves (relative to current directory or relative to HDF5 file, plus maybe an option to disable resolving external files completely)
- The new property would default to the current behavior (relative to current directory)
- Add an environment variable to override the default behavior

Best regards,
Steffen Kie�

···

On 2015-03-31 01:06, Elena Pourmal wrote:

On Mar 30, 2015, at 2:49 AM, Steffen Kie� <Steffen.Kiess@ipvs.uni-stuttgart.de> wrote:

On 2015-03-12 00:08, Elena Pourmal wrote:

10

Hi Steffen,

Scheduling a telecon with Europe is not a problem. We have such telecons every week :slight_smile:

Could you please, tell me the dates next week or the week after (any time between 15:00-17:00 your time) when you are available?

Thank you!

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 14, 2015, at 8:14 AM, Steffen Kieß <Steffen.Kiess@ipvs.uni-stuttgart.de<mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

Hello,

On 2015-03-31 01:06, Elena Pourmal wrote:
Hi Steffen,

On Mar 30, 2015, at 2:49 AM, Steffen Kieß <Steffen.Kiess@ipvs.uni-stuttgart.de<mailto:Steffen.Kiess@ipvs.uni-stuttgart.de>> wrote:

Hello,

On 2015-03-12 00:08, Elena Pourmal wrote:
Steffen,

Thank you for the new patch! I’ll let you know if we have any questions.

is there any update on this? Is there anything I can do to help?

Yes, there is.

On Friday, March 27, we reviewed the patch and concluded that we cannot accept it as it is. Applications that rely on the current behavior will be broken.
Would you like to have a telecon with us to discuss the issue and see how we can move forward with the fix? We really appreciate your effort and would like to resolve the issue.

A teleconference would be a possibility, but as I'm living in Europe there might be some problems scheduling it.

If you want to avoid changing the behavior for existing applications, I can update the patch to:
- Add a new dataset access property which controls how resolving external files behaves (relative to current directory or relative to HDF5 file, plus maybe an option to disable resolving external files completely)
- The new property would default to the current behavior (relative to current directory)
- Add an environment variable to override the default behavior

Best regards,
Steffen Kieß

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5