Re-read of metadata

Hi all,

one of the major problems that I have in synchronizing reads and writes with
HDF5 files and locks is the update of metadata. Here' s some pseudoflows
that show the problem:

Process 1:
1. Create file file.h5
2. Start process 2
3. Wait until file can be locked
4. Read /dataset

Process 2:
1. Open file file.h5
2. Get exclusive lock
3. Write /dataset
4. Unlock and close the file

My locks work on the file handle returned by H5Fget_vfd_handle and always
lock the first byte. This is the only way I could find to synchronize
between threads and processes. this works fine so far, what I would like to
do now is this: Process 1 at step (4) will not read the dataset, because the
file handle in process 1 is opened before process B writes into the file.
What I would like to do is have a way to "refresh" the metadata. The ways to
do that are:

1. Close and reopen the file, but that will lose me the lock
2. Lock on an OS handle and then open the HDF5 handle. That will work on
Linux but not windows as the HDF5 handle will not be able to read.
3. Do it in a VFD driver but then locking/unlocking is not controlled by the
program.

What I really would like to have is a function that simply forces the
metadata to be re-read from the same open file handle. Is there any such
function that is available for sharing? Otherwise I will try to write one.

Best

-- dimitris

Hi Dimitris,

···

On Jan 20, 2011, at 8:05 AM, Dimitris Servis wrote:

Hi all,

one of the major problems that I have in synchronizing reads and writes with HDF5 files and locks is the update of metadata. Here' s some pseudoflows that show the problem:

Process 1:
1. Create file file.h5
2. Start process 2
3. Wait until file can be locked
4. Read /dataset

Process 2:
1. Open file file.h5
2. Get exclusive lock
3. Write /dataset
4. Unlock and close the file

My locks work on the file handle returned by H5Fget_vfd_handle and always lock the first byte. This is the only way I could find to synchronize between threads and processes. this works fine so far, what I would like to do now is this: Process 1 at step (4) will not read the dataset, because the file handle in process 1 is opened before process B writes into the file. What I would like to do is have a way to "refresh" the metadata. The ways to do that are:

1. Close and reopen the file, but that will lose me the lock
2. Lock on an OS handle and then open the HDF5 handle. That will work on Linux but not windows as the HDF5 handle will not be able to read.
3. Do it in a VFD driver but then locking/unlocking is not controlled by the program.

What I really would like to have is a function that simply forces the metadata to be re-read from the same open file handle. Is there any such function that is available for sharing? Otherwise I will try to write one.

  We have implemented this functionality for the 1.10 release.

    Quincey

Hi Quincey

thanks for the response, is this code available (AFAIK there's only 1.9.x
development snapshots)? When is it planned to be released officially?

thanks again!

Best

-- dimitris

···

2011/1/20 Quincey Koziol <koziol@hdfgroup.org>

Hi Dimitris,

On Jan 20, 2011, at 8:05 AM, Dimitris Servis wrote:

> Hi all,
>
> one of the major problems that I have in synchronizing reads and writes
with HDF5 files and locks is the update of metadata. Here' s some
pseudoflows that show the problem:
>
> Process 1:
> 1. Create file file.h5
> 2. Start process 2
> 3. Wait until file can be locked
> 4. Read /dataset
>
> Process 2:
> 1. Open file file.h5
> 2. Get exclusive lock
> 3. Write /dataset
> 4. Unlock and close the file
>
> My locks work on the file handle returned by H5Fget_vfd_handle and always
lock the first byte. This is the only way I could find to synchronize
between threads and processes. this works fine so far, what I would like to
do now is this: Process 1 at step (4) will not read the dataset, because the
file handle in process 1 is opened before process B writes into the file.
What I would like to do is have a way to "refresh" the metadata. The ways to
do that are:
>
> 1. Close and reopen the file, but that will lose me the lock
> 2. Lock on an OS handle and then open the HDF5 handle. That will work on
Linux but not windows as the HDF5 handle will not be able to read.
> 3. Do it in a VFD driver but then locking/unlocking is not controlled by
the program.
>
> What I really would like to have is a function that simply forces the
metadata to be re-read from the same open file handle. Is there any such
function that is available for sharing? Otherwise I will try to write one.

        We have implemented this functionality for the 1.10 release.

               Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Dimitris,

Hi Quincey

thanks for the response, is this code available (AFAIK there's only 1.9.x development snapshots)?

  Right now, this code is available in a branch, but I can create a snapshot for you to experiment with, if you'd like. Send me email off the list, if so.

When is it planned to be released officially?

  Elena and I are working on a release schedule for the 1.10.0 release, but it's not done yet. We'll post something to the list when we have something available.

  Quincey

···

On Jan 20, 2011, at 9:00 AM, Dimitris Servis wrote:

thanks again!

Best

-- dimitris

2011/1/20 Quincey Koziol <koziol@hdfgroup.org>
Hi Dimitris,

On Jan 20, 2011, at 8:05 AM, Dimitris Servis wrote:

> Hi all,
>
> one of the major problems that I have in synchronizing reads and writes with HDF5 files and locks is the update of metadata. Here' s some pseudoflows that show the problem:
>
> Process 1:
> 1. Create file file.h5
> 2. Start process 2
> 3. Wait until file can be locked
> 4. Read /dataset
>
> Process 2:
> 1. Open file file.h5
> 2. Get exclusive lock
> 3. Write /dataset
> 4. Unlock and close the file
>
> My locks work on the file handle returned by H5Fget_vfd_handle and always lock the first byte. This is the only way I could find to synchronize between threads and processes. this works fine so far, what I would like to do now is this: Process 1 at step (4) will not read the dataset, because the file handle in process 1 is opened before process B writes into the file. What I would like to do is have a way to "refresh" the metadata. The ways to do that are:
>
> 1. Close and reopen the file, but that will lose me the lock
> 2. Lock on an OS handle and then open the HDF5 handle. That will work on Linux but not windows as the HDF5 handle will not be able to read.
> 3. Do it in a VFD driver but then locking/unlocking is not controlled by the program.
>
> What I really would like to have is a function that simply forces the metadata to be re-read from the same open file handle. Is there any such function that is available for sharing? Otherwise I will try to write one.

       We have implemented this functionality for the 1.10 release.

               Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org