memory backed files

HDF5 Library VOLs and VFDs
21

Hi,
I am not seem any error message because my code does not provide any. The second H5Fopen on my snippet fails by returning a negative value (-1). Have you give it a try?

22

OK, I can reproduce the issue. Before we go there, a few comments.

Since you are interested only in the object type, use H5O_INFO_BASIC. Otherwise, this is a very expensive operation, especially, in a callback.

No, this is usually not the case, because the core VFD buffer is allocated in increments of memory_increments whereas the buffer obtained from the on-disk file is not subject to such increment constraints. If the latter size were divisible by memory_increments, then there’d be a chance of size equality.

OK?

The error message I see is this:

HDF5-DIAG: Error detected in HDF5 (1.13.2-1) thread 0:
  #000: H5F.c line 837 in H5Fopen(): unable to synchronously open file
    major: File accessibility
    minor: Unable to open file
  #001: H5F.c line 797 in H5F__open_api_common(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #002: H5VLcallback.c line 3686 in H5VL_file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLcallback.c line 3498 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #004: H5VLnative_file.c line 128 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #005: H5Fint.c line 1964 in H5F_open(): unable to read superblock
    major: File accessibility
    minor: Read failed
  #006: H5Fsuper.c line 450 in H5F__super_read(): unable to load superblock
    major: File accessibility
    minor: Unable to protect metadata
  #007: H5AC.c line 1396 in H5AC_protect(): H5C_protect() failed
    major: Object cache
    minor: Unable to protect metadata
  #008: H5C.c line 2368 in H5C_protect(): can't load entry
    major: Object cache
    minor: Unable to load metadata into cache
  #009: H5C.c line 7315 in H5C__load_entry(): incorrect metadata checksum after all read attempts
    major: Object cache
    minor: Read failed

I looked at the beginning of the file_image_memory buffer, and I can see the HDF5 signature. One difference from earlier versions (of the superblock) is the checksum, which this is complaining about. My hunch is that despite the H5Fflush the checksum is either not updated or incorrect because the end-of-file-address will change when the file is closed (because of truncation). Maybe the logic changed/is flawed in 1.10.x and later? The reason why this doesn’t happen in earlier superblock versions (0,1) is that there was no checksum. I’ll check w/ my colleagues to get some clarity.

Best, G.

23

thank you for your comments and for looking into to this.

24

Hi,

Have you had the chance of looking into it? Should this issue be labeled as a bug? If you agree this is a bug, is there a place to fill up a proper ticket for it?

Best, Eduardo

25

Please file a bug report on GitHub! G.

26

Thanks. Bug created.

27

From one of your previous comments I assume the checksum got in the 1.10.x and later versions, is this right? I haven’t test with 1.10 but I think it would safe to assume the same problem would happen, right?

28

@gheber Could you provide a workaround for this issue? Maybe using a temporary file on disk would work, is there a way to set an image into a disk file?

29

@derobins and @gheber do you have any updates on this? Maybe the fix has been incorporated into one of the new versions? If so, could you point to a specific commit?