Hello,
We are hitting a limit of 2048 open files per process when using HDF5 (we use file families), the limit seems to be coming from the C run-time used by HDF5:https://msdn.microsoft.com/en-us/library/6e3b887c.aspx
Is there a way around this limit?
At some point MySQL had the same issue due to the C run-time limit and their solution was to switch to using the Win API for file ops:http://bugs.mysql.com/bug.php?id=24509
Hi,
This is a known issue in HDF5 on Windows. There are plans to develop a virtual file driver that uses the Win32 API instead of POSIX but, unfortunately, this has not yet moved to the front of the priority queue (At one point there was a Windows VFD, but it did not use Win32 file I/O. We deprecated it and removed the code since it did not do anything special.).
If your company might be interested in funding the work, I can put you in touch with the business development people here. Given the way our file drivers are encapsulated, I don't think it would be a very difficult or lengthy project.
Dana Robinson
Software Engineer
The HDF Group
ยทยทยท
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Johhny Depp
Sent: Friday, April 10, 2015 6:36 AM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 2048 Open Files Limit (Windows)
Hello,
We are hitting a limit of 2048 open files per process when using HDF5 (we use file families), the limit seems to be coming from the C run-time used by HDF5:
https://msdn.microsoft.com/en-us/library/6e3b887c.aspx
Is there a way around this limit?
At some point MySQL had the same issue due to the C run-time limit and their solution was to switch to using the Win API for file ops:
http://bugs.mysql.com/bug.php?id=24509