read-only multi-threaded access with forced object closing

Hello,
After writing an application that allows the user to flip between loading and saving a H5 file, a few objects within the file weren't closing properly which didn't allow the file to close when I called H5Fclose. Going from a loading state (H5open) to a saving state (H5create or H5open with RDWR access) complained that the file was still open so I implemented a simple routine to close all open objects within a file before calling H5Fclose:

    void close_all_objects_within_file (hid_t file_id)
    {
      ssize_t numOpenObjs = H5Fget_obj_count(file_id, H5F_OBJ_ALL);
      hid_t *obj_id_list = (hid_t*)(malloc(numOpenObjs * sizeof(hid_t)));
      ssize_t numReturnedOpenObjs = H5Fget_obj_ids(file_id, H5F_OBJ_ALL, numOpenObjs, obj_id_list);
      for (ssize_t i = 0; i < numReturnedOpenObjs; ++i)
      {
        H5Oclose(obj_id_list[i]);
      }
      free(obj_id_list);
    }

This worked fine until the application became multi-threaded, and a call to close_all_objects_within_file(file_id) was crashing the program. It seems that the object resources are shared between the threads that have opened the same file, and the first thread to call this routine kills any chance of another thread to use that resource. Is this correct?

If this is case, is there a way I can check (in the for-loop above) if the object is being referenced by another thread before doing an H5Oclose? Or perhaps there's a better way to close all the resources of a file? Or maybe I'm going about this the wrong way, in which case I'm very much open to suggestions.

Many thanks in advance,
Richard.

Hi Richard,

Hello,
After writing an application that allows the user to flip between loading and saving a H5 file, a few objects within the file weren't closing properly which didn't allow the file to close when I called H5Fclose. Going from a loading state (H5open) to a saving state (H5create or H5open with RDWR access) complained that the file was still open so I implemented a simple routine to close all open objects within a file before calling H5Fclose:

  void close_all_objects_within_file (hid_t file_id)
  {
    ssize_t numOpenObjs = H5Fget_obj_count(file_id, H5F_OBJ_ALL);
    hid_t *obj_id_list = (hid_t*)(malloc(numOpenObjs * sizeof(hid_t)));
    ssize_t numReturnedOpenObjs = H5Fget_obj_ids(file_id, H5F_OBJ_ALL, numOpenObjs, obj_id_list);
    for (ssize_t i = 0; i < numReturnedOpenObjs; ++i)
    {
      H5Oclose(obj_id_list[i]);
    }
    free(obj_id_list);
  }

This worked fine until the application became multi-threaded, and a call to close_all_objects_within_file(file_id) was crashing the program. It seems that the object resources are shared between the threads that have opened the same file, and the first thread to call this routine kills any chance of another thread to use that resource. Is this correct?

  Yes.

If this is case, is there a way I can check (in the for-loop above) if the object is being referenced by another thread before doing an H5Oclose? Or perhaps there's a better way to close all the resources of a file? Or maybe I'm going about this the wrong way, in which case I'm very much open to suggestions.

  You probably want to use the H5Pset_fclose_degree() routine.

    Quincey

···

On Jul 7, 2010, at 3:47 AM, Richard Khoury wrote:

Does read-only HDF usage tolerate multi-threading? I was under the
impression that enabling threading simply added a mutex around critical
operations. Is there a difference between platforms, e.g. Linux & Windows?

Cheers,

Sebastian

···

On Thu, Jul 8, 2010 at 6:00 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi Richard,

On Jul 7, 2010, at 3:47 AM, Richard Khoury wrote:

> Hello,
> After writing an application that allows the user to flip between loading
and saving a H5 file, a few objects within the file weren't closing properly
which didn't allow the file to close when I called H5Fclose. Going from a
loading state (H5open) to a saving state (H5create or H5open with RDWR
access) complained that the file was still open so I implemented a simple
routine to close all open objects within a file before calling H5Fclose:
>
> void close_all_objects_within_file (hid_t file_id)
> {
> ssize_t numOpenObjs = H5Fget_obj_count(file_id, H5F_OBJ_ALL);
> hid_t *obj_id_list = (hid_t*)(malloc(numOpenObjs * sizeof(hid_t)));
> ssize_t numReturnedOpenObjs = H5Fget_obj_ids(file_id, H5F_OBJ_ALL,
numOpenObjs, obj_id_list);
> for (ssize_t i = 0; i < numReturnedOpenObjs; ++i)
> {
> H5Oclose(obj_id_list[i]);
> }
> free(obj_id_list);
> }
>
> This worked fine until the application became multi-threaded, and a call
to close_all_objects_within_file(file_id) was crashing the program. It seems
that the object resources are shared between the threads that have opened
the same file, and the first thread to call this routine kills any chance of
another thread to use that resource. Is this correct?

        Yes.

> If this is case, is there a way I can check (in the for-loop above) if
the object is being referenced by another thread before doing an H5Oclose?
Or perhaps there's a better way to close all the resources of a file? Or
maybe I'm going about this the wrong way, in which case I'm very much open
to suggestions.

        You probably want to use the H5Pset_fclose_degree() routine.

               Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hey Quincey,

Quincey Koziol wrote:

Hi Richard,

Hello,
After writing an application that allows the user to flip between loading and saving a H5 file, a few objects within the file weren't closing properly which didn't allow the file to close when I called H5Fclose. Going from a loading state (H5open) to a saving state (H5create or H5open with RDWR access) complained that the file was still open so I implemented a simple routine to close all open objects within a file before calling H5Fclose:

  void close_all_objects_within_file (hid_t file_id)
  {
    ssize_t numOpenObjs = H5Fget_obj_count(file_id, H5F_OBJ_ALL);
    hid_t *obj_id_list = (hid_t*)(malloc(numOpenObjs * sizeof(hid_t)));
    ssize_t numReturnedOpenObjs = H5Fget_obj_ids(file_id, H5F_OBJ_ALL, numOpenObjs, obj_id_list);
    for (ssize_t i = 0; i < numReturnedOpenObjs; ++i)
    {
      H5Oclose(obj_id_list[i]);
    }
    free(obj_id_list);
  }

This worked fine until the application became multi-threaded, and a call to close_all_objects_within_file(file_id) was crashing the program. It seems that the object resources are shared between the threads that have opened the same file, and the first thread to call this routine kills any chance of another thread to use that resource. Is this correct?
    
  Yes.

If this is case, is there a way I can check (in the for-loop above) if the object is being referenced by another thread before doing an H5Oclose? Or perhaps there's a better way to close all the resources of a file? Or maybe I'm going about this the wrong way, in which case I'm very much open to suggestions.
    
  You probably want to use the H5Pset_fclose_degree() routine.

H5Pset_fclose_degree() did the trick.

Thanks again for the help.

Richard.

···

On Jul 7, 2010, at 3:47 AM, Richard Khoury wrote:

Hi Sebastian,

Does read-only HDF usage tolerate multi-threading? I was under the impression that enabling threading simply added a mutex around critical operations. Is there a difference between platforms, e.g. Linux & Windows?

  You must still enable thread-safety for the HDF5 library even when performing read-only access to files. There are plenty of internal [memory] data structures that will likely blow up currently without the global mutex to only let one thread in at a time.

  Currently, we don't support multi-threaded access on Windows, although we are making some progress in that area currently.

  Quincey

···

On Jul 8, 2010, at 9:46 AM, Sebastian Good wrote:

Cheers,

Sebastian

On Thu, Jul 8, 2010 at 6:00 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:
Hi Richard,

On Jul 7, 2010, at 3:47 AM, Richard Khoury wrote:

> Hello,
> After writing an application that allows the user to flip between loading and saving a H5 file, a few objects within the file weren't closing properly which didn't allow the file to close when I called H5Fclose. Going from a loading state (H5open) to a saving state (H5create or H5open with RDWR access) complained that the file was still open so I implemented a simple routine to close all open objects within a file before calling H5Fclose:
>
> void close_all_objects_within_file (hid_t file_id)
> {
> ssize_t numOpenObjs = H5Fget_obj_count(file_id, H5F_OBJ_ALL);
> hid_t *obj_id_list = (hid_t*)(malloc(numOpenObjs * sizeof(hid_t)));
> ssize_t numReturnedOpenObjs = H5Fget_obj_ids(file_id, H5F_OBJ_ALL, numOpenObjs, obj_id_list);
> for (ssize_t i = 0; i < numReturnedOpenObjs; ++i)
> {
> H5Oclose(obj_id_list[i]);
> }
> free(obj_id_list);
> }
>
> This worked fine until the application became multi-threaded, and a call to close_all_objects_within_file(file_id) was crashing the program. It seems that the object resources are shared between the threads that have opened the same file, and the first thread to call this routine kills any chance of another thread to use that resource. Is this correct?

       Yes.

> If this is case, is there a way I can check (in the for-loop above) if the object is being referenced by another thread before doing an H5Oclose? Or perhaps there's a better way to close all the resources of a file? Or maybe I'm going about this the wrong way, in which case I'm very much open to suggestions.

       You probably want to use the H5Pset_fclose_degree() routine.

               Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Delighted to hear that progress is being made, especially on the Windows
side. Read-only multi-threaded access would be quite lovely.

Cheers!

Sebastian

···

On Thu, Jul 8, 2010 at 10:00 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi Sebastian,

On Jul 8, 2010, at 9:46 AM, Sebastian Good wrote:

Does read-only HDF usage tolerate multi-threading? I was under the
impression that enabling threading simply added a mutex around critical
operations. Is there a difference between platforms, e.g. Linux & Windows?

You must still enable thread-safety for the HDF5 library even when
performing read-only access to files. There are plenty of internal [memory]
data structures that will likely blow up currently without the global mutex
to only let one thread in at a time.

Currently, we don't support multi-threaded access on Windows, although we
are making some progress in that area currently.

Quincey

Cheers,

Sebastian

On Thu, Jul 8, 2010 at 6:00 AM, Quincey Koziol <koziol@hdfgroup.org>wrote:

Hi Richard,

On Jul 7, 2010, at 3:47 AM, Richard Khoury wrote:

> Hello,
> After writing an application that allows the user to flip between
loading and saving a H5 file, a few objects within the file weren't closing
properly which didn't allow the file to close when I called H5Fclose. Going
from a loading state (H5open) to a saving state (H5create or H5open with
RDWR access) complained that the file was still open so I implemented a
simple routine to close all open objects within a file before calling
H5Fclose:
>
> void close_all_objects_within_file (hid_t file_id)
> {
> ssize_t numOpenObjs = H5Fget_obj_count(file_id, H5F_OBJ_ALL);
> hid_t *obj_id_list = (hid_t*)(malloc(numOpenObjs * sizeof(hid_t)));
> ssize_t numReturnedOpenObjs = H5Fget_obj_ids(file_id, H5F_OBJ_ALL,
numOpenObjs, obj_id_list);
> for (ssize_t i = 0; i < numReturnedOpenObjs; ++i)
> {
> H5Oclose(obj_id_list[i]);
> }
> free(obj_id_list);
> }
>
> This worked fine until the application became multi-threaded, and a call
to close_all_objects_within_file(file_id) was crashing the program. It seems
that the object resources are shared between the threads that have opened
the same file, and the first thread to call this routine kills any chance of
another thread to use that resource. Is this correct?

        Yes.

> If this is case, is there a way I can check (in the for-loop above) if
the object is being referenced by another thread before doing an H5Oclose?
Or perhaps there's a better way to close all the resources of a file? Or
maybe I'm going about this the wrong way, in which case I'm very much open
to suggestions.

        You probably want to use the H5Pset_fclose_degree() routine.

               Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org