Hi All,
I wanted to check to see if anyone has run into this case and has some
suggestions for what to do.
I have two HDF5 applications; A and B. A generates (writes) some HDF5
files that B reads and does some visualization of the files A produces.
I'd like to be able to operate (on Windows) in such a way that I
frequently re-run A, producing a file say 'foo.h5' which I might already
have open in application B. All I do in application B to see the new
contents of 'foo.h5' is to 're-open' the file.
However, on Windows, the problem is that if A attempts to write to
'foo.h5' while B has it open, the Windows somehow prevents this with a
'this file is already in use by another application. Please close the
file in other applications before attempting to modify it' error.
Does anyone know if any special HDF5 or Windows incantations with which
to open 'foo.h5' in application B such that application A will NOT hit
this Windows error?
Mark
···
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Mark
I have the same use case. Solver runs in one process and
postprocessing in another. I don't have a problem. AFAIK HDF5 does not
do some kind of locking. Do you somehow lock the file? Is it possible
that you have another open handle somewhere? If you are using
CreateFile take good care that the flags are consistent. I use my own
locking mechanism for the files to synchronize processes and threads.
I'd be happy to provide you with more info.
···
2010/5/18, Mark Miller <miller86@llnl.gov>:
Hi All,
I wanted to check to see if anyone has run into this case and has some
suggestions for what to do.
I have two HDF5 applications; A and B. A generates (writes) some HDF5
files that B reads and does some visualization of the files A produces.
I'd like to be able to operate (on Windows) in such a way that I
frequently re-run A, producing a file say 'foo.h5' which I might already
have open in application B. All I do in application B to see the new
contents of 'foo.h5' is to 're-open' the file.
However, on Windows, the problem is that if A attempts to write to
'foo.h5' while B has it open, the Windows somehow prevents this with a
'this file is already in use by another application. Please close the
file in other applications before attempting to modify it' error.
Does anyone know if any special HDF5 or Windows incantations with which
to open 'foo.h5' in application B such that application A will NOT hit
this Windows error?
Mark
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Since I am working with applications that are DESIGNED to work on both
Linux and Windows, it is conceivable that on Windows, I am inadvertently
using the non-Windows VFD for HDF5 to open the files. For example, I
could inadvertently be using sec2 VFD when running on Windows. Could
that be the source of the problem?
Mark
···
On Tue, 2010-05-18 at 20:03 +0200, Dimitris Servis wrote:
Mark
I have the same use case. Solver runs in one process and
postprocessing in another. I don't have a problem. AFAIK HDF5 does not
do some kind of locking. Do you somehow lock the file? Is it possible
that you have another open handle somewhere? If you are using
CreateFile take good care that the flags are consistent. I use my own
locking mechanism for the files to synchronize processes and threads.
I'd be happy to provide you with more info.
2010/5/18, Mark Miller <miller86@llnl.gov>:
> Hi All,
>
> I wanted to check to see if anyone has run into this case and has some
> suggestions for what to do.
>
> I have two HDF5 applications; A and B. A generates (writes) some HDF5
> files that B reads and does some visualization of the files A produces.
>
> I'd like to be able to operate (on Windows) in such a way that I
> frequently re-run A, producing a file say 'foo.h5' which I might already
> have open in application B. All I do in application B to see the new
> contents of 'foo.h5' is to 're-open' the file.
>
> However, on Windows, the problem is that if A attempts to write to
> 'foo.h5' while B has it open, the Windows somehow prevents this with a
> 'this file is already in use by another application. Please close the
> file in other applications before attempting to modify it' error.
>
> Does anyone know if any special HDF5 or Windows incantations with which
> to open 'foo.h5' in application B such that application A will NOT hit
> this Windows error?
>
> Mark
>
> --
> Mark C. Miller, Lawrence Livermore National Laboratory
> ================!!LLNL BUSINESS ONLY!!================
> miller86@llnl.gov urgent: miller86@pager.llnl.gov
> T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Mark
For example, I
could inadvertently be using sec2 VFD when running on Windows. Could
that be the source of the problem?
In the most recent hdf5 source base, (certainly the cmakeified version I'm using) - The default VFD is set to ... (some stuff snipped out)
IF (WINDOWS)
SET (H5_HAVE_WINDOWS 1)
SET (H5_DEFAULT_VFD H5FD_WINDOWS)
ELSE (WINDOWS)
SET (H5_DEFAULT_VFD H5FD_SEC2)
ENDIF (WINDOWS)
So unless the application is setting it manually, then the windows VFD should be used. If it isn't using it, I'm not sure how you'd know - can debug messages be enabled?
I'm surprised that Dimitris is not getting a locking issue, I've always found that windows stops you writing to a file once another app has it open. If the post-processing app Opens/Reads/Closes the file as quickly as possible, and the simulation doesn't write whilst it is open, all should be fine. How one synchronizes is a good question though.
However, if it is using another VFD, one option (that is a bit drastic) - If you don't have the source to the application and it is using a DLL, you could recompile the hdf5 lib, force the Windows VFD to be used (by overriding any calls to set_fapl code) and then replace the hdf5 dll with your 'modified' version.
JB
Hi all,
sorry for the delayed response.
John, AFAIK the restriction on not being able to open a write handle on a
file when a read handle is open is impose by HDF5. At the OS level it is up
to the flags passed to the CreateFile function how it will behave. In that
case you can very well open handles that all read, write - or any
combination thereof - to the file. My own locking mechanism locks the file
before it is needed. Locking guarantees that a handle with proper semantics
for the operation can be acquired.
Mark,
unless you're doing it on purpose you must be using the Windows native
functions. In that case you'd confront the problem that many people have
that writing in an HDF5 file from different threads or processes is not
synchronised. I think the error you get means that you open the file from
somewhere else and keep the file open but I believe this should work if it
was the HDF5 handle. The windows driver uses _open in a permissive way.
···
2010/5/20 Biddiscombe, John A. <biddisco@cscs.ch>
Mark
> For example, I
> could inadvertently be using sec2 VFD when running on Windows. Could
> that be the source of the problem?
In the most recent hdf5 source base, (certainly the cmakeified version I'm
using) - The default VFD is set to ... (some stuff snipped out)
IF (WINDOWS)
SET (H5_HAVE_WINDOWS 1)
SET (H5_DEFAULT_VFD H5FD_WINDOWS)
ELSE (WINDOWS)
SET (H5_DEFAULT_VFD H5FD_SEC2)
ENDIF (WINDOWS)
So unless the application is setting it manually, then the windows VFD
should be used. If it isn't using it, I'm not sure how you'd know - can
debug messages be enabled?
I'm surprised that Dimitris is not getting a locking issue, I've always
found that windows stops you writing to a file once another app has it open.
If the post-processing app Opens/Reads/Closes the file as quickly as
possible, and the simulation doesn't write whilst it is open, all should be
fine. How one synchronizes is a good question though.
However, if it is using another VFD, one option (that is a bit drastic) -
If you don't have the source to the application and it is using a DLL, you
could recompile the hdf5 lib, force the Windows VFD to be used (by
overriding any calls to set_fapl code) and then replace the hdf5 dll with
your 'modified' version.
JB
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
