paramon
September 11, 2018, 8:32am
1
Hello!
Some time ago I reported a critical bug in H5Ocopy to help@hdfgroup.org . It was labeled [HDFFV-10466] but no investigation took place since. I now report it publicly to hdf-forum@lists.hdfgroup.org , in search for collaboration from HDF5 user community to try craft a patch.
To reproduce:
Download
https://zendto.acdlabs.com/download.php?claimID=PAECtrZAvv3vaJcg&claimPasscode=2GTeuhMRJPbdZ82w&auth=2da721baa9d05867fe94a8fc0f19e741&fid=18754
h5ls -r sulfadrug.abf
Works fine!
h5copy -i sulfadrug.abf -o new.abf -s /msms/info -d /test -p -v
Copying file <sulfadrug.abf> and object </msms/info> to file <new.abf> and object
h5copy: Creating parent groups
Segmentation fault
cat new.abf
%HDF
(file exists)
h5ls new.abf
new.abf: unable to open file
Reproducible on vanilla HDF5 binaries, Debian and Windows, 1.10.0…1.10.3.
Running under debugger, I see that src->shared->u.vlen has
loc == H5T_LOC_BADLOC
and all callbacks incl. isnull are NULLs here:
https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse/src/H5Tconv.c#3183
It’s not quite clear how src->shared could come with H5T_LOC_BADLOC and if it’s expected (according to the comment at
https://bitbucket.hdfgroup.org/projects/HDFFV/repos/hdf5/browse/src/H5Odtype.c#512
it might be).
Best wishes,
Andrey Paramonov
paramon
September 11, 2018, 12:03pm
2
I managed to create a minimally reproducible example with h5py.
This works fine:
import h5py
with h5py.File('test.h5', libver='latest') as f:
for i in range(8):
f.attrs[str(i + 1)] = u'test'
f.copy(f, 'copy')
But this breaks:
import h5py
with h5py.File('test.h5', libver='latest') as f:
for i in range(9): # <- note
f.attrs[str(i + 1)] = u'test'
f.copy(f, 'copy')
Best wishes,
Andrey Paramonov
paramon
September 17, 2018, 4:27pm
4
What about the following patch?
--- H5Aint.c
+++ CMake-hdf5-1.10.3\hdf5-1.10.3\src\H5Aint.c
@@ -2522,6 +2522,10 @@
HDassert(udata->file);
HDassert(udata->cpy_info);
+ /* Mark datatype as being on disk now (analogously to H5O__attr_copy_file). */
+ if(H5T_set_loc(attr_src->shared->dt, udata->oloc_src->file, H5T_LOC_DISK) < 0)
+ HGOTO_ERROR(H5E_ATTR, H5E_CANTINIT, NULL, "invalid datatype location")
+
if(NULL == (attr_dst = H5A__attr_copy_file(attr_src, udata->file, udata->recompute_size, udata->cpy_info)))
HGOTO_ERROR(H5E_ATTR, H5E_CANTCOPY, H5_ITER_ERROR, "can't copy attribute")
Best wishes,
Andrey Paramonov
Thank you for the patch! Added to the issue and will review.
Elena
paramon
February 14, 2019, 3:56pm
6
Hello Elena, Barbara, all!
Messages on the forum start mentioning “upcoming 1.10.5 release” more and more frequently, but I missed any updates on the status of
https://jira.hdfgroup.org/browse/HDFFV-10466
so far.
This report has a minimally-reproducible example, a simple patch, and was submitted several months ago. Let’s please make sure the fix gets into the 1.10.5 release.
Best wishes,
Andrey Paramonov
Andrey,
We plan to release your patch. Thank you for reminder!
Elena
Andrey,
First of all, I should say in my previous message that we planned to review your patch.
Last Friday HDF5 developers did review the patch and concluded that we couldn’t accept it. The patch just hides the problem. We need to debug more and find the root of the problem and then decide how it should be fixed.
Thank you!
Elena
paramon
February 21, 2019, 1:44pm
10
Hello!
I no longer reproduce with HDF5 1.10.5-pre1. Good job!
For the reference, a minimal C test program:
#include <stdio.h>
#include <stdlib.h>
#include "hdf5.h"
int main() {
hid_t fapl, file, datatype, dataspace, attr;
char *key = "0";
char *value = "value";
herr_t status;
fapl = H5Pcreate(H5P_FILE_ACCESS);
status = H5Pset_libver_bounds(fapl, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST);
file = H5Fcreate("test.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl);
datatype = H5Tcreate(H5T_STRING, 5);
dataspace = H5Screate(H5S_SCALAR);
for(int i=0; i<9; i++) {
attr = H5Acreate2(file, itoa(i, key, 10), datatype, dataspace,
H5P_DEFAULT, H5P_DEFAULT);
status = H5Awrite(attr, datatype, value);
status = H5Aclose(attr);
};
status = H5Sclose(dataspace);
status = H5Tclose(datatype);
status = H5Ocopy(file, "/", file, "copy", H5P_DEFAULT, H5P_DEFAULT);
status = H5Fclose(file);
}
Best wishes,
Andrey Paramonov
Hi folks!
This may have risen from the dead as I seem to have run into this issue as well, with both hdf5 1.10.5 and 1.10.6.
For example, the minimal h5py example provided above by @paramon results in a segmentation fault, and my observations match the others in this thread.
My environment (conda):
h5py 2.10.0 nompi_py38hfb01d0b_104 conda-forge
hdf5 1.10.6 nompi_h3c11f04_101 conda-forge
python 3.8.5 h1103e12_5_cpython conda-forge
Can anyone confirm?
Thanks!
Bruce Wallin