If a disk becomes full while writing datasets, you get an OSError with errno 28, which is expected. However, h5py.File.close() also throws an OSError exception. If you terminate the application, you get further OSErrors with same errno, probably from open datasets.
If you remove the file on OS level with ‘rm’, it is no longer listed, however, the disk is still full (as shown with ‘df’). Only if the application is terminated, the disk space is freed.
What is the proper way to close an HDF file, if the disk is full?
I have put together a small example script to demonstrate the problem: diskfull.py (1.6 KB)
I used a fixed path inside the script to point to /tmp dir, which is rather small. When running it on my PC, I get this output:
xspadmin@xspdemo03:~/tmp$ ./diskfull.py
error at index 559: No space left on device
close failed: No space left on device
Segmentation fault
The segfault is probably during destruction of the Writer class and the h5py.File class therein.
The other problem is, that as long as the script is running (i.e. you have some code to wait for user input before deleting ‘w’), then the HDF file is not released and you still see it in ‘lsof’, meaning the disk is still full, even if you just called File.close().
Surround the critical parts of your code, especially file operations, with try-except blocks to catch and handle exceptions. When you encounter a disk full condition, log the exception, and consider closing the file properly.
import h5py
try:
# Your HDF file operations here
with h5py.File(‘your_file.hdf5’, ‘w’) as file:
# Your dataset writing operations
# …
except OSError as e:
if e.errno == 28: # Disk full error
print(“Disk full error. Closing the file.”)
# Close the file if it’s still open
if file:
file.close()
else:
# Handle other OSError cases
print(f"Error: {e}")
except OSError as e:
if e.errno == 28: # Disk full error
print(“Disk full error. Closing the file.”)
# Close the file if it’s still open
if file:
file.close()
else:
# Handle other OSError cases
print(f"Error: {e}")
There is currently no “proper” way to close an HDF5 file when no storage is left on the device(s) you write to. Leaving aside the question of what “proper” could mean, the problem is that the state of an HDF5 file that was opened for writing consists of modified or new bytes in memory and bytes in storage. In a “disk full”-situation, you ask the library to decide what part of the state to scrap to “save your skin.” In the current implementation, the library has no concept of transactions or logic to make such decisions. Could this be implemented? Of course. Who wants to contribute and support the development?
Unfortunately, file.close() also throws an OSError exception, thus a single try-except is not working in all situations. Especially, if you have many datasets opened, then close() will throw again for each one while trying to close them.
I think it’s pretty likely that this is a bug, probably in the HDF5 library but it might just be limited to h5py. I’m assuming that in @andreas.beckmann 's application the failure occurred after the f5fclose() on the last open HDF5 file ID and there were no other open objects associated with the file which would prevent closing even without the disk full.
The documentation on the h5fclose function is silent about the behavior of the function in a disk full situation.
That said, in unix OS’s when a file is deleted (unlinked) the disk space isn’t recovered until all the the file descriptors associated with it are closed. This is a feature in that it allows an application to open a file, delete the file (without closing), and then use the disk as temporary data storage area via the open file descriptor. The disk space is automatically recovered when the application exits–even if it crashes. It’s the last bit which is nice because it prevents the disk from filling up if a buggy application creates a lot of temporary files and then crashes before closing (as long as the file was unlinked after open).
This functionality isn’t the goal here, however. It should be possible for an HDF5 application to encounter a disk full situation and handle it–including recovering the disk space–without exiting. In other words, call unlink to delete the file if h5fclose() returns a error. The implication from this report is that can’t be done and that’s the bug.
If there were a mechanism to extract the unix file descriptor corresponding to the HDF5 then the user could close the file by bypassing the HDF5 library, but I think the best solution would be for h5fclose to always close the unix file id on disk full, returning an error if necessary.
The first step is probably to reproduce the problem in C to be rule out h5py as a cause.
There is H5Fget_vfd_handle() to get the underlying file descriptor, but just closing this is a very bad idea, because HDF5 will still try to use the fd number. If your process opens another file afterwards, the fd number will be reused, so HDF5 might write to & close a completely different file!
A full disk isn’t the only scenario where you can have this problem - e.g. if your file is on a network filesystem and the connection goes down, writing can fail. We’ve also seen it in h5py where people wrap a file like object and then close that object before closing the HDF5 file.
To cope with this properly, I think HDF5 needs some way to ‘abandon’ an open file - i.e. discarding any unwritten data, accepting that the file on disk is likely corrupt or incomplete.
I agree with your last statement. Main goal should be to release all file handles, so that users are able to remove the file outside the context of the application to free up disk space.
I wanted to ask if recent HDF5 (2.1.0) has any updates in regards to this functionality.
I got to a similar problem, I have a daemon writer receiving data from X-ray detector and writing them using HDF5 library on disk. When getting to a disk full error, I get error on H5Dwrite, which I can handle. However, after I do H5Fclose, HDF5 library keeps the file descriptor open and H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) is non-zero. I don’t care about the broken file consistency, I would like to delete it ASAP. I would like to know if there is a way to recover global state of the library (no stale descriptors, no stale HDF5 objects, etc.), or is terminating and restarting process the reasonable solution in this case?
@filip.leonarski, thank you for reviving this. In our SHINES project, it has been recognized as a first-rate safety issue. At its heart is the issue that there is no HDF5 story for ENOSPC, there is no callback for “the VFD just hit ENOSPC.” Short of implementing journaling. There are a few options to consider:
A space reservation API — H5Freserve(file, n_bytes) that calls posix_fallocate/equivalent through the VFD before a large write campaign. Catches ENOSPC up-front instead of mid-flush.
A “poisoned file” state — once a write fails, mark the in-memory file struct unrecoverable: future writes/flushes return immediately, and H5Fclose releases memory & FDs without attempting to flush the dirty MDC. Today, this has to be hacked by setting the cache to discard-on-evict.
A pre-flush watchdog hook in the metadata cache that calls statvfs (or VFD equivalent) before issuing a flush batch and fails fast if the headroom is below a configurable threshold.
A VFD-level error callback property (H5Pset_vfd_error_cb) fired on write/truncate failures, with a return code letting the caller choose retry / fail-fast / panic.
(I’m sure the experts can come up with additional options/see the shortcomings in these proposals.)
As a blunt instrument, you could implement option 4 without changing the core library by writing a pass-through VFD that wraps sec2/core (forget MPI for now) and intercepts non-zero returns from the inner driver’s write/truncate callbacks. This would be the lowest-friction place to experiment before proposing a public API. Claude suggested this skeleton:
Invoked from the VFD write/truncate paths and from H5AC flush failures. The handler can:
Call _exit(2) (skips atexit, leaves files as-is on disk - what you asked for).
Call abort() (core dump for forensics).
Return CONTINUE to fall back to today’s behavior.
I think this would be interesting to collaborate on and produce an RFC that lays out all aspects of the problem and what’s acceptable for different stakeholders. Then we’ll have a better idea of what to implement.
I’ve followed your suggestion on pass-through VFD for sec2. I’ve got one with GPT help, so it is just a prototype, but given it might be helpful for someone I write my experience below. The logic of the pass-through VFD is as follows:
If sec2 reports error, the file is marked as poisoned, and error is transferred upstream, also error information is stored internally in the pass-through VFD,
All other read/write operations on the poisoned file return error immediately,
Pass-through VFD can be switched externally to special state - then all write operations always return success upstream.
Pass-through VFD is restarted, and error code can be recovered.
This seems to work. During H5Fclose any non-zero return value on write/truncate/flush is interrupting closing and H5Fget_obj_count(H5F_OBJ_ALL, H5F_OBJ_ALL) is non zero afterwards. I’ve also encountered seg faults when closing the process. Therefore I need to strictly return 0 when running H5Fclose. Otherwise I got problem of two open files, one tripping ENOSPC, and being also unable to close the second one. Obvious downside is that after H5Fclose one needs to manually check for pass-through VFD status to alert upstream code of corrupted code.
We are still trying to understand root cause of the ENOSPC problem. It seems that wrong configuration of GPFS quota tripped disk space error, even though space was technically there (df showed 500 TB available, while our files where <10 GB). Therefore statvfs solution sounds to me not enough, such error can well happen when space apparently is there. Reservation API could help, but I’m not sure how it interacts with parallel file systems.
I’ll be happy to share my experience looking into fix for the issue - it made me very scary, regarding getting to undefined state after getting disk space problem.
