Turn off file locking

Hdf-forum archives
1

Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to write and read data files. My current computing resources aren't sufficient for much longer, so I got an account on a new computing resource. Unfortunately, the administrators on this new system have disabled file locking to improve performance of their Lustre file system. HDF5 demands file locks, which I don't need, and I can't find a way to turn that off. Is there a way to tell HDF5 to not request a file lock, so that I can run on a system that doesn't allow file locking?

Thank you,
Brendan

2

If you're on Linux, you could look at the "eatmydata" command. This
LD_PRELOADs a small library which overrides the glibc locking calls and
simply stubs them out. Depending upon how the locking is implemented by
HDF5, this may or may not be a possible solution, but thought it might
be something you might find worth investigating.

Regards,
Roger

···

On 28/06/2016 17:09, Krueger, Brendan K wrote:

Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to
write and read data files. My current computing resources aren't
sufficient for much longer, so I got an account on a new computing
resource. Unfortunately, the administrators on this new system have
disabled file locking to improve performance of their Lustre file
system. HDF5 demands file locks, which I don't need, and I can't find a
way to turn that off. Is there a way to tell HDF5 to not request a file
lock, so that I can run on a system that doesn't allow file locking?

--
Dr Roger Leigh -- Open Microscopy Environment
Wellcome Trust Centre for Gene Regulation and Expression,
School of Life Sciences, University of Dundee, Dow Street,
Dundee DD1 5EH Scotland UK Tel: (01382) 386364

The University of Dundee is a registered Scottish Charity, No: SC015096

3

Hi Brendan,

There is a source patch for HDF5 1.10 that disables file locking located here:

<https://www.hdfgroup.org/HDF5/release/obtainsrc5110.html#patch

Does that solve your problem?

Dana Robinson
Software Engineer
The HDF Group
<https://www.hdfgroup.org/HDF5/release/obtainsrc5110.html#patch>

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Krueger, Brendan K" <bkkrueger@lanl.gov<mailto:bkkrueger@lanl.gov>>
Reply-To: HDF List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, June 28, 2016 at 12:09
To: HDF List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] Turn off file locking

Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to write and read data files. My current computing resources aren't sufficient for much longer, so I got an account on a new computing resource. Unfortunately, the administrators on this new system have disabled file locking to improve performance of their Lustre file system. HDF5 demands file locks, which I don't need, and I can't find a way to turn that off. Is there a way to tell HDF5 to not request a file lock, so that I can run on a system that doesn't allow file locking?

Thank you,
Brendan

4

Hi Dana,

I installed HDF5 using the patch that you pointed me to and ran a test case. It looks like everything works. Thank you very much!

Brendan

···

---------------------------------
Brendan K. Krueger
Scientist
XCP-2: Eulerian Codes
Los Alamos National Laboratory
www.linkedin.com/in/bkkrueger
(505) 667-0243

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of Dana Robinson <derobins@hdfgroup.org<mailto:derobins@hdfgroup.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, June 28, 2016 at 11:45 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] Turn off file locking

Hi Brendan,

There is a source patch for HDF5 1.10 that disables file locking located here:

<https://www.hdfgroup.org/HDF5/release/obtainsrc5110.html#patch

Does that solve your problem?

Dana Robinson
Software Engineer
The HDF Group
<https://www.hdfgroup.org/HDF5/release/obtainsrc5110.html#patch>

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Krueger, Brendan K" <bkkrueger@lanl.gov<mailto:bkkrueger@lanl.gov>>
Reply-To: HDF List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Tuesday, June 28, 2016 at 12:09
To: HDF List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: [Hdf-forum] Turn off file locking

Hello,

I've written a code that uses HDF5 (serial mode, C++ if that matters) to write and read data files. My current computing resources aren't sufficient for much longer, so I got an account on a new computing resource. Unfortunately, the administrators on this new system have disabled file locking to improve performance of their Lustre file system. HDF5 demands file locks, which I don't need, and I can't find a way to turn that off. Is there a way to tell HDF5 to not request a file lock, so that I can run on a system that doesn't allow file locking?

Thank you,
Brendan

5

Hi, that patch is no longer available, and I’m having the exact same problem with a Lustre file system using the latest HDF5 code. I would hope file locking could be disabled by a config option perhaps?

6

Never mind - I just discovered the HDF5_USE_FILE_LOCKING=FALSE environment variable setting.