Hi everyone, I’m testing the reliability of hdf5 and I managed to create a corrupted file.
This was not a random event as I was able to reproduce the creation of such a file.
My method is simple, I launched a jupyter notebook which uses the code below to store a list of large 2d images in a group, each image has its own dataset.
import numpy as np
import random
import h5py
import time
import tracemalloc
def generate_random_image(height=600, width=512):
"""generate a 2d image with random value for 0 to 9000 """
return np.random.randint(0, 9001, size=(height, width), dtype=np.int16)
tracemalloc.start()
start_time=time.time()
hdf5_file_path = 'random_image.h5'
with h5py.File(hdf5_file_path, 'w') as hdf5_file:
k=10
image_list=[generate_random_image(10000,10000) for _ in range(k)]
print("image list created")
for i,image in enumerate(image_list):
print(i)
hdf5_file.create_dataset(f'image/frame_{i:05}', data=image,compression="gzip")
#add the number of image in the group
hdf5_file['image'].attrs['nbr_frame']=len(image_list)
write_time=time.time()-start_time
print(f"writing time {write_time} s")
current, peak = tracemalloc.get_traced_memory()
print(f"RAM allocated {current / 10**6}MB; Max {peak / 10**6}MB")
tracemalloc.stop()
To create the corrupted file, the notebook must be stopped while the hdf5 dataset is being written. To do this, I split-screened the jupyter notebook home page in the runing tab on one side and the notebook runing on the other. I then waited about 0.5 seconds after seeing printed “2” on the notebook output and pressed the stop button on the home page to kill the process.
By doing this, I was able to create a corrupted file each time I did this.
I tried to open the file with h5web vs code extension and h5py and for h5py I got :
233 if swmr and swmr_support :
234 flags |= h5f.ACC_SWMR_READ
--> 235 fid = h5f.open(name, flags, fapl=fapl)
236 elif mode == 'r+' :
237 fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
File h5py\_objects.pyx:54, in h5py._objects.with_phil.wrapper()
File h5py\_objects.pyx:55, in h5py._objects.with_phil.wrapper()
File h5py\h5f.pyx:102, in h5py.h5f.open()
OSError: Unable to open file synchronously (wrong version number in object header)
Im realy currious to know if someone can reproduce it and if it would be possible to get the images previously stored inside.
Thank for reading
R
