HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same
hdf5 file as read-only. I have searched over months on the internet trying
to determine if HDF5 can surely handle this case and have found conflicting
results.

I have built a test in C++, using the hdf5 C API, to determine if the read
is possible. The test occasional segfaults on H5Close(). I am trying to
determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same
file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

Did you build the HDF5 library with thread-safety enabled?

https://www.hdfgroup.org/hdf5-quest.html#tsafe

Yes, the documentation is inconsistent. :frowning:

G.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 5:31 PM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same hdf5 file as read-only. I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

I have built a test in C++, using the hdf5 C API, to determine if the read is possible. The test occasional segfaults on H5Close(). I am trying to determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

Hi Isaac,

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :slight_smile:

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

Dana Robinson
Software Engineer
The HDF Group

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same hdf5 file as read-only. I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

I have built a test in C++, using the hdf5 C API, to determine if the read is possible. The test occasional segfaults on H5Close(). I am trying to determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

Did you build the HDF5 library with thread-safety enabled?

Yes I did.

···

On Mon, Aug 29, 2016 at 9:59 AM, Gerd Heber <gheber@hdfgroup.org> wrote:

Did you build the HDF5 library with thread-safety enabled?

https://www.hdfgroup.org/hdf5-quest.html#tsafe

Yes, the documentation is inconsistent. L

G.

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Isaac Gerg
*Sent:* Friday, August 26, 2016 5:31 PM
*To:* hdf-forum@lists.hdfgroup.org
*Subject:* [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same
hdf5 file as read-only. I have searched over months on the internet trying
to determine if HDF5 can surely handle this case and have found conflicting
results.

I have built a test in C++, using the hdf5 C API, to determine if the read
is possible. The test occasional segfaults on H5Close(). I am trying to
determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same
file as online documentation seems to be inconsistent.

Thanks in advance,

Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Dana,

I am not sure I understand your question. In each thread, I open an hdf
file, read from it, and then call H5close().

Normally, you don't need to call H5close() explicitly, btw. It will be

called when the program exits.

I did not know about this.

In reading the documentation...

"This function is generally called when the application calls exit(), but

may be called earlier in the event of an emergency shutdown or out of a
desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in
exit() OR that its preferred by hdf library that the user calls H5close()
right before calling exit().

Thank you,
Isaac

···

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org> wrote:

Hi Isaac,

Are you calling H5close() from one thread and then expecting the other
thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful
after you shut the library down :slight_smile:

Normally, you don't need to call H5close() explicitly, btw. It will be
called when the program exits.

From the reference manual (https://www.hdfgroup.org/
HDF5/doc/RM/RM_H5.html#Library-Close ):

"This function is generally called when the application calls exit(), but
may be called earlier in the event of an emergency shutdown or out of a
desire to free all resources used by the HDF5 library."

Dana Robinson

Software Engineer

The HDF Group

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Isaac Gerg
*Sent:* Friday, August 26, 2016 6:31 PM
*To:* hdf-forum@lists.hdfgroup.org
*Subject:* [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same
hdf5 file as read-only. I have searched over months on the internet trying
to determine if HDF5 can surely handle this case and have found conflicting
results.

I have built a test in C++, using the hdf5 C API, to determine if the read
is possible. The test occasional segfaults on H5Close(). I am trying to
determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same
file as online documentation seems to be inconsistent.

Thanks in advance,

Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Can you send us an example that fails (occasionally)? G.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 9:05 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

Did you build the HDF5 library with thread-safety enabled?

Yes I did.

On Mon, Aug 29, 2016 at 9:59 AM, Gerd Heber <gheber@hdfgroup.org<mailto:gheber@hdfgroup.org>> wrote:
Did you build the HDF5 library with thread-safety enabled?

https://www.hdfgroup.org/hdf5-quest.html#tsafe

Yes, the documentation is inconsistent. :frowning:

G.

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 5:31 PM
To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same hdf5 file as read-only. I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

I have built a test in C++, using the hdf5 C API, to determine if the read is possible. The test occasional segfaults on H5Close(). I am trying to determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Isaac,

It sounds like you want to call H5Fclose, not H5close. Try just closing the file and not calling H5close at all. Does that work?

Dana

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:25 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Dana,

I am not sure I understand your question. In each thread, I open an hdf file, read from it, and then call H5close().

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

I did not know about this.

In reading the documentation...

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

Thank you,
Isaac

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>> wrote:
Hi Isaac,

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :slight_smile:

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

Dana Robinson
Software Engineer
The HDF Group

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same hdf5 file as read-only. I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

I have built a test in C++, using the hdf5 C API, to determine if the read is possible. The test occasional segfaults on H5Close(). I am trying to determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Thanks Dana, I see what you mean and where I misread.

···

On Mon, Aug 29, 2016 at 11:29 AM, Dana Robinson <derobins@hdfgroup.org> wrote:

Hi Isaac,

It sounds like you want to call H5Fclose, not H5close. Try just closing
the file and not calling H5close at all. Does that work?

Dana

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Isaac Gerg
*Sent:* Monday, August 29, 2016 11:25 AM
*To:* HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Subject:* Re: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Dana,

I am not sure I understand your question. In each thread, I open an hdf
file, read from it, and then call H5close().

> Normally, you don't need to call H5close() explicitly, btw. It will be
called when the program exits.

I did not know about this.

In reading the documentation...

> "This function is generally called when the application calls exit(),
but may be called earlier in the event of an emergency shutdown or out of a
desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called
in exit() OR that its preferred by hdf library that the user calls
H5close() right before calling exit().

Thank you,

Isaac

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org> > wrote:

Hi Isaac,

Are you calling H5close() from one thread and then expecting the other
thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful
after you shut the library down :slight_smile:

Normally, you don't need to call H5close() explicitly, btw. It will be
called when the program exits.

From the reference manual (https://www.hdfgroup.org/
HDF5/doc/RM/RM_H5.html#Library-Close ):

"This function is generally called when the application calls exit(), but
may be called earlier in the event of an emergency shutdown or out of a
desire to free all resources used by the HDF5 library."

Dana Robinson

Software Engineer

The HDF Group

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Isaac Gerg
*Sent:* Friday, August 26, 2016 6:31 PM
*To:* hdf-forum@lists.hdfgroup.org
*Subject:* [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same
hdf5 file as read-only. I have searched over months on the internet trying
to determine if HDF5 can surely handle this case and have found conflicting
results.

I have built a test in C++, using the hdf5 C API, to determine if the read
is possible. The test occasional segfaults on H5Close(). I am trying to
determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same
file as online documentation seems to be inconsistent.

Thanks in advance,

Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Interesting... I am using H5Fclose().

If 2 different threads call this function for the same file (but different
file handles), does HDF handle this?

···

On Mon, Aug 29, 2016 at 11:29 AM, Dana Robinson <derobins@hdfgroup.org> wrote:

Hi Isaac,

It sounds like you want to call H5Fclose, not H5close. Try just closing
the file and not calling H5close at all. Does that work?

Dana

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Isaac Gerg
*Sent:* Monday, August 29, 2016 11:25 AM
*To:* HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
*Subject:* Re: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Dana,

I am not sure I understand your question. In each thread, I open an hdf
file, read from it, and then call H5close().

> Normally, you don't need to call H5close() explicitly, btw. It will be
called when the program exits.

I did not know about this.

In reading the documentation...

> "This function is generally called when the application calls exit(),
but may be called earlier in the event of an emergency shutdown or out of a
desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called
in exit() OR that its preferred by hdf library that the user calls
H5close() right before calling exit().

Thank you,

Isaac

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org> > wrote:

Hi Isaac,

Are you calling H5close() from one thread and then expecting the other
thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful
after you shut the library down :slight_smile:

Normally, you don't need to call H5close() explicitly, btw. It will be
called when the program exits.

From the reference manual (https://www.hdfgroup.org/
HDF5/doc/RM/RM_H5.html#Library-Close ):

"This function is generally called when the application calls exit(), but
may be called earlier in the event of an emergency shutdown or out of a
desire to free all resources used by the HDF5 library."

Dana Robinson

Software Engineer

The HDF Group

*From:* Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] *On
Behalf Of *Isaac Gerg
*Sent:* Friday, August 26, 2016 6:31 PM
*To:* hdf-forum@lists.hdfgroup.org
*Subject:* [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same
hdf5 file as read-only. I have searched over months on the internet trying
to determine if HDF5 can surely handle this case and have found conflicting
results.

I have built a test in C++, using the hdf5 C API, to determine if the read
is possible. The test occasional segfaults on H5Close(). I am trying to
determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same
file as online documentation seems to be inconsistent.

Thanks in advance,

Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Isaac,

Yes. There can be multiple file IDs for the same physical file. Closing one of them will not affect the other.

Dana

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:57 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

Interesting... I am using H5Fclose().

If 2 different threads call this function for the same file (but different file handles), does HDF handle this?

On Mon, Aug 29, 2016 at 11:29 AM, Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>> wrote:
Hi Isaac,

It sounds like you want to call H5Fclose, not H5close. Try just closing the file and not calling H5close at all. Does that work?

Dana

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Isaac Gerg
Sent: Monday, August 29, 2016 11:25 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Dana,

I am not sure I understand your question. In each thread, I open an hdf file, read from it, and then call H5close().

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

I did not know about this.

In reading the documentation...

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

To me, "generally" implies that h5close() is not guaranteed to be called in exit() OR that its preferred by hdf library that the user calls H5close() right before calling exit().

Thank you,
Isaac

On Mon, Aug 29, 2016 at 11:14 AM, Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>> wrote:
Hi Isaac,

Are you calling H5close() from one thread and then expecting the other thread to make HDF5 calls? Because HDF5 calls are unlikely to be successful after you shut the library down :slight_smile:

Normally, you don't need to call H5close() explicitly, btw. It will be called when the program exits.

From the reference manual (https://www.hdfgroup.org/HDF5/doc/RM/RM_H5.html#Library-Close ):

"This function is generally called when the application calls exit(), but may be called earlier in the event of an emergency shutdown or out of a desire to free all resources used by the HDF5 library."

Dana Robinson
Software Engineer
The HDF Group

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>] On Behalf Of Isaac Gerg
Sent: Friday, August 26, 2016 6:31 PM
To: hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] HDF5 -- multithreading support or not?

Hi Folks,

I am a need for a windows application to have two threads read the same hdf5 file as read-only. I have searched over months on the internet trying to determine if HDF5 can surely handle this case and have found conflicting results.

I have built a test in C++, using the hdf5 C API, to determine if the read is possible. The test occasional segfaults on H5Close(). I am trying to determine if the segfault is due to HDF5 or my own error.

TL;DR - can HDF5 on windows support having 2 threads read from the same file as online documentation seems to be inconsistent.

Thanks in advance,
Isaac

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5