[HDFFV-10465] Spurious problem in H5Ocopy

Hello HDF5 Community!

Reproducible on HDF5 1.10.2+ and Windows, 32- and 64-bit, any Microsoft compiler version.
Not reproducible: HDF5 1.10.1 or earlier, GNU/Linux.

To reproduce:

  1. Install HDF5 binaries from
    https://www.hdfgroup.org/package/windows-10-64-bit-vs2015

  2. Install Visual Studio 2015 compiler.

  3. Put attached sources into a dir:
    lz4.h (23.8 KB)
    lz4.c (60.6 KB)
    lz4_h5plugin.h (435 Bytes)
    lz4_h5plugin.c (5.4 KB)
    test.c (3.6 KB)
    make.bat (428 Bytes)

  4. Run make.bat there:

set HDF5_PATH=%ProgramFiles%\HDF_Group\HDF5\1.10.2
call "%ProgramFiles(x86)%\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" x64

copy "%HDF5_PATH%\bin\hdf5.dll" .

mkdir plugins
cl /LD /MD /Ox lz4_h5plugin.c lz4.c ^
 /I"%HDF5_PATH%\include" ^
 /Fe:"plugins\h5lz4.dll" ^
 /link "%HDF5_PATH%\lib\hdf5.lib"
cl /MD /Ox test.c ^
 /I"%HDF5_PATH%\include" ^
 /DH5_BUILT_AS_DYNAMIC_LIB ^
 /link "%HDF5_PATH%\lib\hdf5.lib"
  1. Run test.exe
    Constructing file…done.
    Flushing…done.
    Copying…
    and then the program crashes without back-trace.

Reproducible also on debug version
(HDF5 sources needed):

  1. Run make-debug.bat:
    make-debug.bat (1.3 KB)
    (modify to specify HDF5 sources location and compiler)

  2. Run test.exe


I believe the code in lz4_plugin.c is written correctly, but could you please review?
The issue becomes critical as it was not addressed since 1.10.2 release.

Best wishes,
Andrey Paramonov

Hello!

18.09.2018 16:19, Andrey Paramonov пишет:

Reproducible on HDF5 1.10.2+ and Windows, 32- and 64-bit, any Microsoft
compiler version.
Not reproducible: HDF5 1.10.1 or earlier, GNU/Linux.

Running “git bisect” gives:

[980d5b4266133490192ceb10ea13e320532c6f03] Fix for HDFFV-7853 H5Ocopy
doesn’t work with open identifiers
Changes made so that raw data for dataset objects are copied from cached
info when possible instead of flushing objects to file and read them
back in again.

with cb_struct being
{func=0xcccccccccccccccc, op_data=0xcccccccccccccccc}
in
H5D__chunk_copy_cb(const H5D_chunk_rec_t * chunk_rec, void * _udata),
line 5823
in that revision.

In HEAD, the crash happens, equally reproducibly, in
H5D__chunk_copy_cb(const H5D_chunk_rec_t * chunk_rec, void * _udata),
line 5747

This bug is critical because it’s a regression.

Best wishes,
Andrey Paramonov

Hi Andrey,

Did you have a chance to try pre-released HDF5 1.10.4 (see hdf5_1_10_4 branch https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse?at=refs%2Fheads%2Fhdf5_1_10_4) ? Was the problem resolved?

Thank you!

Elena

Hi Elena!

08.10.2018 2:59, epourmal пишет:

Did you have a chance to try pre-released HDF5 1.10.4 (see hdf5_1_10_4
branch
https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse?at=refs%2Fheads%2Fhdf5_1_10_4)
? Was the problem resolved?

I’ve just checked and the problem is reproducible on hdf5_1_10_4 HEAD,
too :frowning:

Best wishes,
Andrey Paramonov

Andrey,

Thank you for checking!

It is embarrassing and we need to do some soul searching here - how did we allow digress in the code, but I am afraid, the fix will need to wait until 1.10.5 later in the year.

Thanks again!

Elena

Hi Elena!

08.10.2018 16:18, epourmal пишет:

It is embarrassing and we need to do some soul searching here - how did
we allow digress in the code, but I am afraid, the fix will need to wait
until 1.10.5 later in the year.

I think it might have helped if you checked earlier, when I reported the
issue for the first time (at times of 1.10.2 when the bug first
manifested itself) :wink:

But please keep me updated once the issue is localized and eliminated it
the master branch!

Best wishes,
Andrey Paramonov

Hello!

Another round of digging has revealed the problem source, finally!
Somehow, LZ4_compress() had sneaked into my version of LZ4 plugin,
instead of proper LZ4_compress_default(). After fixing that similarly
to
https://github.com/nexusformat/HDF5-External-Filter-Plugins/commit/12d3444aac72f22c2cdefe0150de0cd7b2cf8298#diff-0168af662096288809587999356c7c11,
I no longer witness the breakage. Yay!

Unfortunately, I do not have permission to comment on or close
https://jira.hdfgroup.org/browse/HDFFV-10465, so I leave that to my HDF
Group colleagues.

Best wishes in the New Year,
Andrey Paramonov