External file limit when generating links to non HDF5 files

Is there a known limit of external files, non HDF5 files when creating an HDF5 data set. The call being used is H5Pset_external. Running on Linux cluster Rhel5, HDF5 5.1-8.10. The limit is a "unusual number" 2730.

Hello!

This same issue recently came up with the helpdesk.

There is no internal limit, so this is likely a system-specific resource issue.
However, we would very much like to understand the issue, and try to reproduce it,
if we can.

When does it fail? Does it happen when creating or accessing data? There might be
something we can suggest to alleviate the problem if we understand the issue.
For example, if the failure happens when reading, then it might help to not open
all files at once (or something like that).

If you have example code that we can use to reproduce the issue, that would
be very helpful, as well.

Thanks!
-Barbara Jones

HDF Helpdesk
help@hdfgroup.org

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Donald Bridwell
Sent: Tuesday, April 29, 2014 2:14 PM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] External file limit when generating links to non HDF5 files

Is there a known limit of external files, non HDF5 files when creating an HDF5 data set. The call being used is H5Pset_external. Running on Linux cluster Rhel5, HDF5 5.1-8.10. The limit is a "unusual number" 2730.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

Could it be related to the limited number of open files in linux, such as mentioned in

?

      Werner

···

On 29.04.2014 21:34, Barbara Jones wrote:

Hello!

This same issue recently came up with the helpdesk.

There is no internal limit, so this is likely a system-specific resource issue.
However, we would very much like to understand the issue, and try to reproduce it,
if we can.

When does it fail? Does it happen when creating or accessing data? There might be
something we can suggest to alleviate the problem if we understand the issue.
For example, if the failure happens when reading, then it might help to not open
all files at once (or something like that).

If you have example code that we can use to reproduce the issue, that would
be very helpful, as well.

Thanks!
-Barbara Jones

HDF Helpdesk
help@hdfgroup.org

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Donald Bridwell
Sent: Tuesday, April 29, 2014 2:14 PM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] External file limit when generating links to non HDF5 files

Is there a known limit of external files, non HDF5 files when creating an HDF5 data set. The call being used is H5Pset_external. Running on Linux cluster Rhel5, HDF5 5.1-8.10. The limit is a "unusual number" 2730.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362