Multiply Opened Files

Hdf-forum archives
1

I found this in the reference manual about H5Fopen():
"In some cases, such as files on a local Unix file system, the HDF5 library
can detect that a file is multiply opened and will maintain coherent access
among the file identifiers"

Does HDF5 maintain multi-open coherent access on Windows also?

Ideally I'd like to have the file locked so that multiple opens are not
possible but I can't figure out a way to do that on Windows. Something like
the flock() method for Unix I've seen on this forum would be fine but I
only see file locking available via CreateFile() in the Windows API.

- David

2

Hmm. Maybe I am asking a stupid question here. . .but are you talking about having a file opened multiple times *within* the same executable or in different executables?

Reason I ask is that I think the former problem is what the comment in H5Fopen is referring to but your inquiy regarding 'locking' seems to apply to the latter case.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of David <list@aue.org<mailto:list@aue.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, November 16, 2015 9:46 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] Multiply Opened Files

I found this in the reference manual about H5Fopen():
"In some cases, such as files on a local Unix file system, the HDF5 library can detect that a file is multiply opened and will maintain coherent access among the file identifiers"

Does HDF5 maintain multi-open coherent access on Windows also?

Ideally I'd like to have the file locked so that multiple opens are not possible but I can't figure out a way to do that on Windows. Something like the flock() method for Unix I've seen on this forum would be fine but I only see file locking available via CreateFile() in the Windows API.

- David

3

Hi David,

Yes, a similar mechanism to that used on POSIX is used on Windows. We use the file index and volume serial number to detect if a file is being re-opened, which should work on NTFS but has caveats on FAT-based systems. Keep in mind that this is only from the same process; we don't detect if files are open across processes. The upcoming 1.10.0 will include a file locking mechanism that should mitigate certain bad use cases like multiple processes opening a file for write access, though. That should work on Windows as well, though we are still in the process of generating a Win32 equivalent.

Dana Robinson
Software Engineer
The HDF Group

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Monday, November 16, 2015 12:46 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Multiply Opened Files

I found this in the reference manual about H5Fopen():
"In some cases, such as files on a local Unix file system, the HDF5 library can detect that a file is multiply opened and will maintain coherent access among the file identifiers"
Does HDF5 maintain multi-open coherent access on Windows also?

Ideally I'd like to have the file locked so that multiple opens are not possible but I can't figure out a way to do that on Windows. Something like the flock() method for Unix I've seen on this forum would be fine but I only see file locking available via CreateFile() in the Windows API.

- David

4

Presently I have a single executable which is probably an important fact
for my question about the internal HDF5 lib multi-open detection. An OS
file lock could work for either case I believe.

David

···

On Mon, Nov 16, 2015 at 9:53 AM, Miller, Mark C. <miller86@llnl.gov> wrote:

Hmm. Maybe I am asking a stupid question here. . .but are you talking
about having a file opened multiple times *within* the same executable or
in different executables?

Reason I ask is that I think the former problem is what the comment in
H5Fopen is referring to but your inquiy regarding 'locking' seems to apply
to the latter case.

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of David
<list@aue.org>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Date: Monday, November 16, 2015 9:46 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Multiply Opened Files

I found this in the reference manual about H5Fopen():
"In some cases, such as files on a local Unix file system, the HDF5
library can detect that a file is multiply opened and will maintain
coherent access among the file identifiers"

Does HDF5 maintain multi-open coherent access on Windows also?

Ideally I'd like to have the file locked so that multiple opens are not
possible but I can't figure out a way to do that on Windows. Something like
the flock() method for Unix I've seen on this forum would be fine but I
only see file locking available via CreateFile() in the Windows API.

- David

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

5

Hi Dana,

Thank you for your answer. The upcoming locking stuff sounds perfect. What
is the timeline for 1.10.0 release?

It looks like I can use Win32 LockFile() on offset 0 to the max file size
in the meantime.
I didn't realize before that LockFile allowed ranges past the present eof.

Thanks,
David

···

On Mon, Nov 16, 2015 at 9:58 AM, Dana Robinson <derobins@hdfgroup.org> wrote:

Hi David,

Yes, a similar mechanism to that used on POSIX is used on Windows. We use
the file index and volume serial number to detect if a file is being
re-opened, which should work on NTFS but has caveats on FAT-based systems.
Keep in mind that this is only from the same process; we don't detect if
files are open across processes. The upcoming 1.10.0 will include a file
locking mechanism that should mitigate certain bad use cases like multiple
processes opening a file for write access, though. That should work on
Windows as well, though we are still in the process of generating a Win32
equivalent.

Dana Robinson

Software Engineer

The HDF Group

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *David
*Sent:* Monday, November 16, 2015 12:46 PM
*To:* HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Subject:* [Hdf-forum] Multiply Opened Files

I found this in the reference manual about H5Fopen():
"In some cases, such as files on a local Unix file system, the HDF5
library can detect that a file is multiply opened and will maintain
coherent access among the file identifiers"

Does HDF5 maintain multi-open coherent access on Windows also?

Ideally I'd like to have the file locked so that multiple opens are not
possible but I can't figure out a way to do that on Windows. Something like
the flock() method for Unix I've seen on this forum would be fine but I
only see file locking available via CreateFile() in the Windows API.

- David

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

6

Hi Dana,

This sounds interesting to me. We have a Windows use case and I'd like to inquire to see if this new HDF5 release addresses it.

The problem we have involves multiple executables but only one writer.

We run Ale3d tutorials. Ale3d is a simulation application and it writes HDF5 files.

We view the resulting files in another application called VisIt.

The problem on Windows is that when VisIt has the file open for reading, a subsequent run of Ale3d is unable to write to the file until VisIt has closed it.

We don't need VisIt to see the new contents of the file until *after* VisIt re-opens the file. However, in this case, the problem is that because one application, VisIt, has the file open for reading, another application, Ale3d, is unable to open (and eventually overwrite) the file.

The behavior is specific to Windows. We don't have same issues on Linux or OS X.

Does newest feature you refer to here address this?

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Monday, November 16, 2015 9:58 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] Multiply Opened Files

Hi David,

Yes, a similar mechanism to that used on POSIX is used on Windows. We use the file index and volume serial number to detect if a file is being re-opened, which should work on NTFS but has caveats on FAT-based systems. Keep in mind that this is only from the same process; we don't detect if files are open across processes. The upcoming 1.10.0 will include a file locking mechanism that should mitigate certain bad use cases like multiple processes opening a file for write access, though. That should work on Windows as well, though we are still in the process of generating a Win32 equivalent.

Dana Robinson
Software Engineer
The HDF Group

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of David
Sent: Monday, November 16, 2015 12:46 PM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] Multiply Opened Files

I found this in the reference manual about H5Fopen():
"In some cases, such as files on a local Unix file system, the HDF5 library can detect that a file is multiply opened and will maintain coherent access among the file identifiers"
Does HDF5 maintain multi-open coherent access on Windows also?

Ideally I'd like to have the file locked so that multiple opens are not possible but I can't figure out a way to do that on Windows. Something like the flock() method for Unix I've seen on this forum would be fine but I only see file locking available via CreateFile() in the Windows API.

- David