HDF5 files on NFS?

Hdf-forum archives
1

Hi All,

In case my previous question was lost between other issues I sent before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is not possible?
How to achieve this? Is there just some "switch" before compiling the library source?

Regards,
Rafal

2

Rafal,

There is a patch to disable flock for the previous release, hdf5
1.10.0-patch1. This might be an interim solution for you. Please see this
recent message from HDF5 support:

https://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2016-December/010026.html

I am not sure what kind of flock treatment is in the recently announced
1.10.1-pre1 pre-release.

--Dave

···

On Fri, Mar 31, 2017 at 3:06 AM, Rafal Lichwala <syriusz@man.poznan.pl> wrote:

Hi All,

In case my previous question was lost between other issues I sent before,
I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is not
possible?
How to achieve this? Is there just some "switch" before compiling the
library source?

Regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

3

I haven't had trouble using the serial HDF5 library on NFS. For parallel HDF5, I believe there are certain versions of NFS and specific configurations required, and from what I understand it will certainly not be fast. I think you really need a file system that is built for parallel I/O to use parallel HDF5 effectively.

I don't know the answer regarding the requirement of flock().

Jarom

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Rafal Lichwala
Sent: Friday, March 31, 2017 2:07 AM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 files on NFS?

Hi All,

In case my previous question was lost between other issues I sent before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is not possible?
How to achieve this? Is there just some "switch" before compiling the library source?

Regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

4

I'm not familiar with the ramifications, and doing so is kind of a hack, but it appears you can disable flock() by passing "ac_cv_func_flock=no" to "configure" before compiling.

Sean

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Rafal Lichwala <syriusz@man.poznan.pl>
Sent: Friday, March 31, 2017 4:06:57 AM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 files on NFS?

Hi All,

In case my previous question was lost between other issues I sent
before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is
not possible?
How to achieve this? Is there just some "switch" before compiling the
library source?

Regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

5

Thank you All for the hints!
I will try both: the patch and configure switch to disable flock().
Thanks!

Regards,
Rafal

W dniu 2017-03-31 o 22:48, Sean Byland pisze:

···

I'm not familiar with the ramifications, and doing so is kind of a hack,
but it appears you can disable flock() by passing "ac_cv_func_flock=no"
to "configure" before compiling.

Sean

------------------------------------------------------------------------
*From:* Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of
Rafal Lichwala <syriusz@man.poznan.pl>
*Sent:* Friday, March 31, 2017 4:06:57 AM
*To:* hdf-forum@lists.hdfgroup.org
*Subject:* [Hdf-forum] HDF5 files on NFS?

Hi All,

In case my previous question was lost between other issues I sent
before, I'd like to ask again with hope someone knows the answer here:

Is is possible to use HDF5 library on NFS filesystems where flock() is
not possible?
How to achieve this? Is there just some "switch" before compiling the
library source?

Regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--

***
Rafał Lichwała
Poznańskie Centrum Superkomputerowo-Sieciowe
ul. Jana Pawła II nr 10
61-139 Poznań
e-mail: syriusz@man.poznan.pl
***

6

Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an environment variable that you can use to disable file locking (HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

    Thank you All for the hints!
    I will try both: the patch and configure switch to disable flock().
    Thanks!
    
    Regards,
    Rafal
    
    W dniu 2017-03-31 o 22:48, Sean Byland pisze:

···

On 4/2/17, 11:38 PM, "Hdf-forum on behalf of Rafal Lichwala" <hdf-forum-bounces@lists.hdfgroup.org on behalf of syriusz@man.poznan.pl> wrote:
    > I'm not familiar with the ramifications, and doing so is kind of a hack,
    > but it appears you can disable flock() by passing "ac_cv_func_flock=no"
    > to "configure" before compiling.
    >
    >
    > Sean
    >
    > ------------------------------------------------------------------------
    > *From:* Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of
    > Rafal Lichwala <syriusz@man.poznan.pl>
    > *Sent:* Friday, March 31, 2017 4:06:57 AM
    > *To:* hdf-forum@lists.hdfgroup.org
    > *Subject:* [Hdf-forum] HDF5 files on NFS?
    >
    > Hi All,
    >
    > In case my previous question was lost between other issues I sent
    > before, I'd like to ask again with hope someone knows the answer here:
    >
    > Is is possible to use HDF5 library on NFS filesystems where flock() is
    > not possible?
    > How to achieve this? Is there just some "switch" before compiling the
    > library source?
    >
    > Regards,
    > Rafal
    >
    >
    > _______________________________________________
    > Hdf-forum is for HDF software users discussion.
    > Hdf-forum@lists.hdfgroup.org
    > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    > Twitter: https://twitter.com/hdf5
    >
    >
    > _______________________________________________
    > Hdf-forum is for HDF software users discussion.
    > Hdf-forum@lists.hdfgroup.org
    > http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    > Twitter: https://twitter.com/hdf5
    >
    
    --
    
    ***
    Rafał Lichwała
    Poznańskie Centrum Superkomputerowo-Sieciowe
    ul. Jana Pawła II nr 10
    61-139 Poznań
    e-mail: syriusz@man.poznan.pl
    ***
    
    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@lists.hdfgroup.org
    http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
    Twitter: https://twitter.com/hdf5

7

Dana,

Is HDF5_USE_FILE_LOCKING applied at configure time, build time, or run time?

--Dave

···

On Mon, Apr 3, 2017 at 4:27 PM, Dana Robinson <derobins@hdfgroup.org> wrote:

Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an
environment variable that you can use to disable file locking
(HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock
calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

<snip>

8

Sorry, I am confused about the meaning of "FALSE" in this context. Is that
actual setting the five-character string "FALSE", or the digit "0"? Thanks
for your help.

--Dave

···

On Mon, Apr 3, 2017 at 5:14 PM, Dave Allured - NOAA Affiliate < dave.allured@noaa.gov> wrote:

Dana,

Is HDF5_USE_FILE_LOCKING applied at configure time, build time, or run
time?

--Dave

On Mon, Apr 3, 2017 at 4:27 PM, Dana Robinson <derobins@hdfgroup.org> > wrote:

Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an
environment variable that you can use to disable file locking
(HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock
calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

<snip>

9

Hi Dave,

The environment variable is checked at runtime when the file is opened.

We check for the five-character string "FALSE" (case-sensitive). No other false-y value will work.

Dana

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Dave Allured - NOAA Affiliate <dave.allured@noaa.gov>
Reply-To: HDF List <hdf-forum@lists.hdfgroup.org>
Date: Monday, April 3, 2017 at 4:23 PM
To: HDF List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] HDF5 files on NFS?

Sorry, I am confused about the meaning of "FALSE" in this context. Is that actual setting the five-character string "FALSE", or the digit "0"? Thanks for your help.

--Dave

On Mon, Apr 3, 2017 at 5:14 PM, Dave Allured - NOAA Affiliate <dave.allured@noaa.gov<mailto:dave.allured@noaa.gov>> wrote:
Dana,

Is HDF5_USE_FILE_LOCKING applied at configure time, build time, or run time?

--Dave

On Mon, Apr 3, 2017 at 4:27 PM, Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>> wrote:
Hi all,

Sorry I've missed most of this discussion, but HDF5 1.10.1 checks an environment variable that you can use to disable file locking (HDF5_USE_FILE_LOCKING). Setting this to "FALSE" disables all file lock calls.

Cheers,

Dana Robinson
Software Engineer
The HDF Group

<snip>