Repair corrupted file


#1

Happy New Year!

I’m unable to open file using HdfView 3.1.3 (Ubuntu 20.04) and most other tools neither work with this file. It gives me error:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:///opt/hdfview/lib/app/mods/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hdfview/lib/app/extra/slf4j-simple-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hdfview/lib/app/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]

(SWT:236228): Gtk-WARNING **: 04:44:21.238: Negative content width -9 (allocation 4, extents 6x7) while allocating gadget (node separator, owner GtkSeparatorToolItem)

(SWT:236228): Gtk-WARNING **: 04:44:21.238: Negative content width -9 (allocation 4, extents 6x7) while allocating gadget (node separator, owner GtkSeparatorToolItem)

(SWT:236228): Gtk-WARNING **: 04:44:21.239: Negative content width -9 (allocation 4, extents 6x7) while allocating gadget (node separator, owner GtkSeparatorToolItem)

(SWT:236228): Gtk-CRITICAL **: 04:44:21.287: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar

(SWT:236228): Gtk-CRITICAL **: 04:44:21.288: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar

(SWT:236228): Gtk-CRITICAL **: 04:44:21.288: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar

(SWT:236228): Gtk-CRITICAL **: 04:44:21.289: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
Exception in thread "main" java.lang.InternalError: H5Gget_obj_info_full: retrieval of object info failed
	at jarhdf5@1.10.7/hdf.hdf5lib.H5.H5Gget_obj_info_full(Native Method)
	at jarhdf5@1.10.7/hdf.hdf5lib.H5.H5Gget_obj_info_full(H5.java:3605)
	at hdf.object.h5.H5File.depth_first(H5File.java:2433)
	at hdf.object.h5.H5File.depth_first(H5File.java:2495)
	at hdf.object.h5.H5File.depth_first(H5File.java:2495)
	at hdf.object.h5.H5File.loadIntoMemory(H5File.java:2374)
	at hdf.object.h5.H5File.open(H5File.java:2349)
	at hdf.object.h5.H5File.open(H5File.java:2220)
	at hdf.object.h5.H5File.open(H5File.java:1028)
	at hdf.view.TreeView.DefaultTreeView.initFile(DefaultTreeView.java:2463)
	at hdf.view.TreeView.DefaultTreeView.openFile(DefaultTreeView.java:2438)
	at hdf.view.HDFView.openLocalFile(HDFView.java:1820)
	at hdf.view.HDFView$2.widgetSelected(HDFView.java:480)
	at swt/org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:252)
	at swt/org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:89)
	at swt/org.eclipse.swt.widgets.Display.sendEvent(Display.java:5685)
	at swt/org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1423)
	at swt/org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4953)
	at swt/org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:4446)
	at hdf.view.HDFView.runMainWindow(HDFView.java:385)
	at hdf.view.HDFView$39.run(HDFView.java:2604)
	at swt/org.eclipse.swt.widgets.Synchronizer.syncExec(Synchronizer.java:236)
	at swt/org.eclipse.swt.widgets.Display.syncExec(Display.java:5795)
	at hdf.view.HDFView.main(HDFView.java:2595)
Failed to launch JVM

Most likely I have somehow broken the file while using my custom C++ app. I guess the problem occured after renaming objects H5Lmove. There was few hundreds of objects and I needed to rename some of them using custom python API. The code had bugs and it took a while to fix it.

It is still possible to open and work with the file using C/C++, python but I suspect that some strange things happen (not sure though).

I’m looking for a way to repair the file, or probably copy all objects (groups, datasets, attributes with data) to another file using C API. Also it would be pretty nice to understand how I did I broke it to not repeat that in the future.

I’ve tried using h5clear -s <myfile.h5> but that doesn’t help.
h5repack <in.h5> <out.h5> gives error:
Error occurred while repacking


#2

Can you send us the output of

h5dump --enable-error-stack=2 <filename> 

?

And what about

h5stat <filename>

?

How many objects do you see?

Have you tried the h5copy command-line tool? And then there is H5Ocopy for your code.

Are you consistently checking return values and handling resources accordingly?

Happy New Year!

G.


#3

Thank you for response,
h5dump gives the following error:

h5dump --enable-error-stack=2 directional_wells.h5
HDF5-DIAG: Error detected in HDF5 (1.12.1) thread 0:
  #000: /home/kerim/Documents/Colada/d/HDF5/src/H5L.c line 1516 in H5Lvisit_by_name2(): link visitation failed
    major: Links
    minor: Iteration failed
  #001: /home/kerim/Documents/Colada/d/HDF5/src/H5VLcallback.c line 5173 in H5VL_link_specific(): unable to execute link specific callback
    major: Virtual Object Layer
    minor: Can't operate on object
  #002: /home/kerim/Documents/Colada/d/HDF5/src/H5VLcallback.c line 5136 in H5VL__link_specific(): unable to execute link specific callback
    major: Virtual Object Layer
    minor: Can't operate on object
  #003: /home/kerim/Documents/Colada/d/HDF5/src/H5VLnative_link.c line 364 in H5VL__native_link_specific(): link visitation failed
    major: Links
    minor: Iteration failed
  #004: /home/kerim/Documents/Colada/d/HDF5/src/H5Gint.c line 1118 in H5G_visit(): can't visit links
    major: Symbol table
    minor: Iteration failed
  #005: /home/kerim/Documents/Colada/d/HDF5/src/H5Gobj.c line 673 in H5G__obj_iterate(): can't iterate over symbol table
    major: Symbol table
    minor: Iteration failed
  #006: /home/kerim/Documents/Colada/d/HDF5/src/H5Gstab.c line 537 in H5G__stab_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #007: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1196 in H5B_iterate(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #008: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1155 in H5B__iterate_helper(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #009: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1155 in H5B__iterate_helper(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #010: /home/kerim/Documents/Colada/d/HDF5/src/H5Gnode.c line 976 in H5G__node_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #011: /home/kerim/Documents/Colada/d/HDF5/src/H5Gobj.c line 673 in H5G__obj_iterate(): can't iterate over symbol table
    major: Symbol table
    minor: Iteration failed
  #012: /home/kerim/Documents/Colada/d/HDF5/src/H5Gstab.c line 537 in H5G__stab_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #013: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1196 in H5B_iterate(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #014: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1155 in H5B__iterate_helper(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #015: /home/kerim/Documents/Colada/d/HDF5/src/H5Gnode.c line 976 in H5G__node_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #016: /home/kerim/Documents/Colada/d/HDF5/src/H5Gobj.c line 673 in H5G__obj_iterate(): can't iterate over symbol table
    major: Symbol table
    minor: Iteration failed
  #017: /home/kerim/Documents/Colada/d/HDF5/src/H5Gstab.c line 537 in H5G__stab_iterate(): iteration operator failed
    major: Symbol table
    minor: Can't move to next iterator location
  #018: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1196 in H5B_iterate(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #019: /home/kerim/Documents/Colada/d/HDF5/src/H5B.c line 1155 in H5B__iterate_helper(): B-tree iteration failed
    major: B-Tree node
    minor: Iteration failed
  #020: /home/kerim/Documents/Colada/d/HDF5/src/H5Gnode.c line 955 in H5G__node_iterate(): unable to get symbol table node name
    major: Symbol table
    minor: Can't get value
  #021: /home/kerim/Documents/Colada/d/HDF5/src/H5HL.c line 390 in H5HL_offset_into(): unable to offset into local heap data block
    major: Heap
    minor: Can't get value
h5dump error: internal error (file /home/kerim/Documents/Colada/d/HDF5/tools/src/h5dump/h5dump.c:line 1550)
H5tools-DIAG: Error detected in HDF5:tools (1.12.1) thread 0:
  #000: /home/kerim/Documents/Colada/d/HDF5/tools/lib/h5tools_utils.c line 794 in init_objs(): finding shared objects failed
    major: Failure in tools library
    minor: error in function
  #001: /home/kerim/Documents/Colada/d/HDF5/tools/lib/h5trav.c line 1058 in h5trav_visit(): traverse failed
    major: Failure in tools library
    minor: error in function
  #002: /home/kerim/Documents/Colada/d/HDF5/tools/lib/h5trav.c line 290 in traverse(): H5Lvisit_by_name failed
    major: Failure in tools library
    minor: error in function

h5stat:

h5stat directional_wells.h5
Filename: directional_wells.h5
h5stat error: unable to traverse objects/links in file "directional_wells.h5"

Fortunately it seems that I am able to copy the data using h5copy:

h5copy --input directional_wells.h5 --output directional_wells_out.h5 -s "/88-AX-10" -d "/88-AX-10"

This copies /88-AX-10 group with its attributes, children, and everything that is inside of it to the destination. I believe I should be able to repair the file by consistently copying objects from one file to another.

I guess so but as I said the problem began after I’ve used bugged python script aimed at renaming links
I was renaming soft links. For example I used to have /88-AX-10/ACTIVE_DEV and I needed to rename it to /88-AX-10/ACTIVE. The ACTIVE_DEV object is a soft link. I can’t say for sure but probably some of these soft links were pointing to nowhere at the time when I was renaming them (is it possible?).

From H5L_MOVE documentation I can see the following:
Exercise care in moving links as it is possible to render data in a file inaccessible with H5L_MOVE. If the link being moved is on the only path leading to an HDF5 object, that object may become permanently inaccessible in the file..
I don’t fully understand this. If possible could you please explain this a little bit more.


#4

HDF groups (including the root group) are collections of links. A link is a uni-directional, named association between a source and a destination. The (implicit) source of each link is the group to which it belongs. The destination is typically another HDF5 object in the same or another HDF5 file. There are several ways to specify a link’s destination, which introduces different link “flavors.”

  1. A hard link: the destination is another HDF5 object in the same file, represented by its address or token.
  2. A soft link: the destination is specified as an HDF5 path name.
  3. An external link: the destination is specified as a pair of (HDF5 file path name, HDF5 path name)
  4. A user-defined link: the destination is specified in a user-defined form (byte array)

Unlike 1., link flavors 2.-4. can be “dangling,” i.e., the destination cannot be resolved to an HDF5 object.

Moving a link means just that. You just take the link from one group’s link collection to another group’s link collection. (Ignoring corner cases…) In all this, the destination of the link is untouched.

It is possible that an HDF5 object is the destination of more than one link, but it better be the destination of at least one link. Otherwise, the object would be unreachable from the root group via link traversal. This is what the comment is getting at. Being able to move links means being able to make things “disappear.”

Let’s assume our profile looks like this:

/group1/dset1
/group1/dest2
/group2

Moving the link group1 from the root group to the group linked as group2 would yield:

/group2/group1/dset1
/group2/group1/dest2

Under what circumstances could links disappear?

  1. Undefined library behavior, e.g., you move links as a side effect of traversing objects in an HDF5 file
  2. You move a link to an anonymous group (H5Gcreate_anon) and forget to link it
  3. A bug in the application causing H5Lmove to fail (I don’t remember what the precise pre- and post-conditions for H5Lmove are.)
  4. A bug in the HDF5 library

OK?

G.


#5

Thank you very much for the explaination!