HDF5 File Builds Correctly Only with Certain Width and Height


#1

Hi, I have some code that creates and writes to an Hdf5 file. However, the size of this file depends on a user input resolution for the images they would like saved. The large resolution 2448x2048 works perfectly fine. However, when those variables switch to 1224 x 1024 the file can no longer be opened in the H5 Viewer. This is weird because the file size is exactly what I would expect. For example, if I took 1 image the file size is roughly 1224*1024 + some overhead. The large resolution opens and works and I can see all of the data, but the smaller version does not work. I’m perplexed because all I’m changing is the value of the parameter I pass in.

I’ll post some code here:
This initializes my file:


This writes to the file and extends it when a new image is grabbed ignore the mutexes and whatnot:
image
image


#2

What is (if any) the error message from HDFView? Have you tried h5dump -pBH <file.h5>? What does that say? G.


#3

I have tried h5dump, opening in MATLAB and h5 view and the message is usually something along the lines of could not be opened.

On h5View: Error ppening file “filename”
With h5dump: h5dump error: unable to open file “thisonedidntwork.h5”
In MATLAB it just says could not open file.

If I try all of these operations on my other file which used the 2448x2048 as an input everything works perfectly. This is reproducible also, not just a one time thing.


#4

I don’t know if this would help, but here is the h5dump from the file that worked correctly:

HDF5 “ThisOneWorked.h5” {
SUPER_BLOCK {
SUPERBLOCK_VERSION 0
FREELIST_VERSION 0
SYMBOLTABLE_VERSION 0
OBJECTHEADER_VERSION 0
OFFSET_SIZE 8
LENGTH_SIZE 8
BTREE_RANK 16
BTREE_LEAF 4
ISTORE_K 32
FILE_SPACE_STRATEGY H5F_FSPACE_STRATEGY_FSM_AGGR
FREE_SPACE_PERSIST FALSE
FREE_SPACE_SECTION_THRESHOLD 1
FILE_SPACE_PAGE_SIZE 4096
USER_BLOCK {
USERBLOCK_SIZE 0
}
}
GROUP “/” {
DATASET “DataSet” {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 2048, 2448, 59 ) / ( 2048, 2448, H5S_UNLIMITED ) }
STORAGE_LAYOUT {
CHUNKED ( 2048, 2448, 1 )
SIZE 295796736
}
FILTERS {
NONE
}
FILLVALUE {
FILL_TIME H5D_FILL_TIME_IFSET
VALUE H5D_FILL_VALUE_DEFAULT
}
ALLOCATION_TIME {
H5D_ALLOC_TIME_INCR
}
}
}
}


#5

If you add “–enable-error-stack” to h5dump you should get the error stack from hdf5 that might indicate why the file cannot be opened.

Allen


#6

I got the exact same error as without that flag. Thanks for the suggestion thought, I didn’t see that flag earlier.


#7

Sorry, for file open errors you need --enable-error-stack=2

Allen


#8

Ok here’s the huge error message, I’m reading through it now, but I’ll post it here as well in case someone has insights.

HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 0:
#000: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5F.c line 793 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3500 in H5VL_file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#002: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3465 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#003: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLnative_file.c line 100 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#004: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Fint.c line 1716 in H5F_open(): unable to read root group
major: File accessibility
minor: Unable to open file
#005: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Groot.c line 239 in H5G_mkroot(): can’t check if symbol table message exists
major: Symbol table
minor: Can’t get value
#006: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Omessage.c line 883 in H5O_msg_exists(): unable to protect object header
major: Object header
minor: Unable to protect metadata
#007: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Oint.c line 1082 in H5O_protect(): unable to load object header
major: Object header
minor: Unable to protect metadata
#008: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5AC.c line 1312 in H5AC_protect(): H5C_protect() failed
major: Object cache
minor: Unable to protect metadata
#009: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5C.c line 2346 in H5C_protect(): can’t load entry
major: Object cache
minor: Unable to load metadata into cache
#010: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5C.c line 6689 in H5C_load_entry(): incorrect metadatda checksum after all read attempts
major: Object cache
minor: Read failed
#011: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Ocache.c line 220 in H5O__cache_get_final_load_size(): can’t deserialize object header prefix
major: Object header
minor: Unable to decode value
#012: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Ocache.c line 1232 in H5O__prefix_deserialize(): bad object header version number
major: Object header
minor: Wrong version number
HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 0:
#000: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5F.c line 793 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3500 in H5VL_file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#002: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3465 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#003: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLnative_file.c line 100 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#004: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Fint.c line 1564 in H5F_open(): unable to open file: name = ‘ThisOneDidnt.h5’, tent_flags = 0
major: File accessibility
minor: Unable to open file
#005: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5FD.c line 741 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#006: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5FDfamily.c line 697 in H5FD_family_open(): file names not unique
major: File accessibility
minor: File already exists
HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 0:
#000: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5F.c line 793 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3500 in H5VL_file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#002: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3465 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#003: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLnative_file.c line 100 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#004: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Fint.c line 1564 in H5F_open(): unable to open file: name = ‘ThisOneDidnt.h5’, tent_flags = 0
major: File accessibility
minor: Unable to open file
#005: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5FD.c line 741 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
HDF5-DIAG: Error detected in HDF5 (1.12.0) thread 0:
#000: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5F.c line 793 in H5Fopen(): unable to open file
major: File accessibility
minor: Unable to open file
#001: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3500 in H5VL_file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#002: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLcallback.c line 3465 in H5VL__file_open(): open failed
major: Virtual Object Layer
minor: Can’t open object
#003: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5VLnative_file.c line 100 in H5VL__native_file_open(): unable to open file
major: File accessibility
minor: Unable to open file
#004: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5Fint.c line 1564 in H5F_open(): unable to open file: name = ‘ThisOneDidnt.h5’, tent_flags = 0
major: File accessibility
minor: Unable to open file
#005: C:\autotest\hdf5112-StdRelease-dist-10vs14\build\hdfsrc\src\H5FD.c line 741 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
h5dump error: unable to open file “ThisOneDidnt.h5”


#9

The only thing that jumps out at me from the error message is an issue with the headers. After some searching it looks like theres a 64kb limit to the headers, but I don’t think that would be the issue as the bigger file size works perfectly fine and the smaller one doesn’t. Is there something magic about having a height of 1024 and a width of 1224 that just breaks stuff? I would sincerely doubt so.


#10

Ok so I went in and enabled szip compression and now everything works totally fine including the compression. This leads me to think there was some issue with the way I chunked my images, but I don’t think so. For now I’ll just be happy that it works, but if you can figure out what was wrong that would be great because there could still be a bug with my code especially pertaining to the chunks.

For example, my chunk is basically 2D, but I gave it a depth dimension of 1 just in case because the overall matrix was 3D. Was this wrong?


#11

The problem may not be with HDF5, but with the writing application. The error message suggests that the HDF5 file isn’t in a sane state. The root group can’t be read with the default driver (POSIX) and then the library tries a few other things, which are noise at this point (unless you used the family or other VFD). Maybe the file wasn’t closed properly by the writing application? Hard to know with the information provided. G.