Hi,
I have a question. Yesterday my HDF5 file got corrupted due to disk-space shortage. Maybe I could have saved the file by a check on the available disk-space before I tried to add data to the file (using system-tools). However this will not help in case, the program itself is responsible for the shortage of disk-space (by adding a lot of data) or someone adds during the execution a lot of data to the system. Thus I would like to catch the errors returned by the HDF5 library and then close all resources leaving the HDF5 file in a proper state. Is that possible, or is the file lost after a failed write action?
Would it make a difference using an other file driver? H5FD_STDIO instead**of**H5FD_SEC2?
Would it make a difference using other setting for |xfer_plist_id |in the call to H5Dwrite?
Would it be possible in a future release (1.10) to restore file to a previous defined state?
Many thanks for your time.
Best regards,
Richard van Hees
···
Hi Richard,
I have a question. Yesterday my HDF5 file got corrupted due to disk-space
shortage.
...
Thus I would like to catch the
errors returned by the HDF5 library and then close all resources leaving the
HDF5 file in a proper state. Is that possible, or is the file lost after a
failed write action?
While it can be dramatically less space efficient, I tend to write
entirely new HDF5 files, close them, flush to disk, and then rename(3)
to (more-or-less) atomically overwrite the old file. This works
regardless of the underlying HDF5 driver. All one needs is the
guarantee that the driver really flushes when asked to flushed. As
always, it leaves one at the mercy of the OS.
My use case is maintaining the last N simulation restart files where I
never want to clobber any old ones before I know the most recent has
been successfully written. The approach may not be useful when you
want to add data to a large, pre-existing file as it would require a
large file copy.
- Rhys
This is tricky due to the large amount of data that the HDF5 library caches in memory... We're making progress on a feature which should address it, but won't be released for several months yet.
Quincey
···
On Mar 14, 2012, at 7:48 AM, Richard van Hees wrote:
Hi,
I have a question. Yesterday my HDF5 file got corrupted due to disk-space shortage. Maybe I could have saved the file by a check on the available disk-space before I tried to add data to the file (using system-tools). However this will not help in case, the program itself is responsible for the shortage of disk-space (by adding a lot of data) or someone adds during the execution a lot of data to the system. Thus I would like to catch the errors returned by the HDF5 library and then close all resources leaving the HDF5 file in a proper state. Is that possible, or is the file lost after a failed write action?
Would it make a difference using an other file driver? H5FD_STDIO instead of H5FD_SEC2?
Would it make a difference using other setting for xfer_plist_id in the call to H5Dwrite?
Would it be possible in a future release (1.10) to restore file to a previous defined state?
Would it make a difference using an other file driver? H5FD_STDIO instead of
H5FD_SEC2?
Just an FYI - The STDIO driver was developed as more of a "tutorial"
VFD than a production VFD and is missing some functionality that
requires POSIX behavior.
You should pretty much always use the SEC2 driver instead of the STDIO
driver; even on Windows.
Cheers,
Dana
