Hi all (new to the list),
I am currently experiencing/investigating an issue where our usage of HDF5 with compression is leading to some defects reported by valgrind (apologies for not having debug symbols for line numbers). These defects are also reported when I run valgrind on
http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_8/FORTRAN/H5D/h5ex_d_gzip.f90
The valgrind output I get from the above example is:
==54559== Conditional jump or move depends on uninitialised value(s)
==54559== at 0x63664E0: inflateReset2 (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==54559== by 0x63665D8: inflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==54559== by 0x5989E13: H5Z_filter_deflate (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x59891A0: H5Z_pipeline (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57C96A1: H5D__chunk_lock (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57CA5D0: H5D__chunk_read (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57DA482: H5D__read (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57DB7B3: H5Dread (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x4E5B143: h5dread_c_ (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5_fortran.so.8.0.0)
==54559== by 0x4E69593: __h5d_provisional_MOD_h5dread_integer_2 (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5_fortran.so.8.0.0)
==54559== by 0x40198E: MAIN__ (h5ex_d_gzip.f90:134)
==54559== by 0x401B81: main (h5ex_d_gzip.f90:16)
In my situation, when I am performing parallel reads I get anomalous behavior, where sometimes the executable works, and sometimes it does not. When it does not work I get the following diagnostic messages from HDF5.
HDF5-DIAG: Error detected in HDF5 (1.8.13) MPI-process 4223:
#000: H5Dio.c line 173 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: H5Dio.c line 545 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#002: H5Dchunk.c line 1861 in H5D__chunk_read(): unable to read raw data chunk
major: Low-level I/O
minor: Read failed
#003: H5Dchunk.c line 2891 in H5D__chunk_lock(): data pipeline read failed
major: Data filters
minor: Filter operation failed
#004: H5Z.c line 1383 in H5Z_pipeline(): filter returned failure during read
major: Data filters
minor: Read failed
#005: H5Zdeflate.c line 136 in H5Z_filter_deflate(): memory allocation failed for deflate uncompression
major: Resource unavailable
minor: No space available for allocation
Which has a similar call stack, and I wonder if it's trying to allocate using an undefined value. My gut tells me these may be related.
Is the valgrind issue I am reporting a known issue? Has anyone else run into this?
Can anyone provide some advice on how might continue debugging this?
Any thoughts on whether this is an issue with HDF5, zlib or some combination?
In my use case, it is extremely desirable to use compression as I am dealing with files that are 2.2GB compressed (4.5GB uncompressed). I can probably post some code if needed, but since the example has the same issue perhaps it is better to work from that.
My hope is that the root cause of this issue is:
A. Something in my usage of the library
B. Something wrong with the configure/build/install of the library (which I cannot necessarily change)
C. Something wrong with the HDF5 library
Any feedback is welcome.
Thanks,
-Brendan
Hi Brendan,
Could you please try more recent version of zlib? If you see the same problem, please send the example that demonstrates the problem to our Helpdesk.
Thank you!
Elena
···
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Jan 27, 2015, at 9:57 AM, Brendan Kochunas <bkochuna@umich.edu> wrote:
Hi all (new to the list),
I am currently experiencing/investigating an issue where our usage of HDF5 with compression is leading to some defects reported by valgrind (apologies for not having debug symbols for line numbers). These defects are also reported when I run valgrind on
http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_8/FORTRAN/H5D/h5ex_d_gzip.f90
The valgrind output I get from the above example is:
==54559== Conditional jump or move depends on uninitialised value(s)
==54559== at 0x63664E0: inflateReset2 (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==54559== by 0x63665D8: inflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==54559== by 0x5989E13: H5Z_filter_deflate (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x59891A0: H5Z_pipeline (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57C96A1: H5D__chunk_lock (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57CA5D0: H5D__chunk_read (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57DA482: H5D__read (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57DB7B3: H5Dread (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x4E5B143: h5dread_c_ (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5_fortran.so.8.0.0)
==54559== by 0x4E69593: __h5d_provisional_MOD_h5dread_integer_2 (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5_fortran.so.8.0.0)
==54559== by 0x40198E: MAIN__ (h5ex_d_gzip.f90:134)
==54559== by 0x401B81: main (h5ex_d_gzip.f90:16)
In my situation, when I am performing parallel reads I get anomalous behavior, where sometimes the executable works, and sometimes it does not. When it does not work I get the following diagnostic messages from HDF5.
HDF5-DIAG: Error detected in HDF5 (1.8.13) MPI-process 4223:
#000: H5Dio.c line 173 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: H5Dio.c line 545 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#002: H5Dchunk.c line 1861 in H5D__chunk_read(): unable to read raw data chunk
major: Low-level I/O
minor: Read failed
#003: H5Dchunk.c line 2891 in H5D__chunk_lock(): data pipeline read failed
major: Data filters
minor: Filter operation failed
#004: H5Z.c line 1383 in H5Z_pipeline(): filter returned failure during read
major: Data filters
minor: Read failed
#005: H5Zdeflate.c line 136 in H5Z_filter_deflate(): memory allocation failed for deflate uncompression
major: Resource unavailable
minor: No space available for allocation
Which has a similar call stack, and I wonder if it's trying to allocate using an undefined value. My gut tells me these may be related.
Is the valgrind issue I am reporting a known issue? Has anyone else run into this?
Can anyone provide some advice on how might continue debugging this?
Any thoughts on whether this is an issue with HDF5, zlib or some combination?
In my use case, it is extremely desirable to use compression as I am dealing with files that are 2.2GB compressed (4.5GB uncompressed). I can probably post some code if needed, but since the example has the same issue perhaps it is better to work from that.
My hope is that the root cause of this issue is:
A. Something in my usage of the library
B. Something wrong with the configure/build/install of the library (which I cannot necessarily change)
C. Something wrong with the HDF5 library
Any feedback is welcome.
Thanks,
-Brendan
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
Hi Elena,
I continued to dig into this today and it seems the valgrind issue was due to our zlib install. I'm still working through my suspected similar issue on the other machine. I am hoping it is an issue with their installs. I am waiting to hear back from their sysadmins to get more information on how to uncover the issue.
Thanks!
-Brendan
···
On 1/27/2015 3:32 PM, Elena Pourmal wrote:
Hi Brendan,
Could you please try more recent version of zlib? If you see the same problem, please send the example that demonstrates the problem to our Helpdesk.
Thank you!
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Jan 27, 2015, at 9:57 AM, Brendan Kochunas <bkochuna@umich.edu> wrote:
Hi all (new to the list),
I am currently experiencing/investigating an issue where our usage of HDF5 with compression is leading to some defects reported by valgrind (apologies for not having debug symbols for line numbers). These defects are also reported when I run valgrind on
http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/hdf5-examples/1_8/FORTRAN/H5D/h5ex_d_gzip.f90
The valgrind output I get from the above example is:
==54559== Conditional jump or move depends on uninitialised value(s)
==54559== at 0x63664E0: inflateReset2 (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==54559== by 0x63665D8: inflateInit2_ (in /lib/x86_64-linux-gnu/libz.so.1.2.3.4)
==54559== by 0x5989E13: H5Z_filter_deflate (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x59891A0: H5Z_pipeline (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57C96A1: H5D__chunk_lock (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57CA5D0: H5D__chunk_read (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57DA482: H5D__read (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x57DB7B3: H5Dread (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5.so.8.0.0)
==54559== by 0x4E5B143: h5dread_c_ (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5_fortran.so.8.0.0)
==54559== by 0x4E69593: __h5d_provisional_MOD_h5dread_integer_2 (in /opt/hdf5/1.8.11/gcc_4.6.3/lib/libhdf5_fortran.so.8.0.0)
==54559== by 0x40198E: MAIN__ (h5ex_d_gzip.f90:134)
==54559== by 0x401B81: main (h5ex_d_gzip.f90:16)
In my situation, when I am performing parallel reads I get anomalous behavior, where sometimes the executable works, and sometimes it does not. When it does not work I get the following diagnostic messages from HDF5.
HDF5-DIAG: Error detected in HDF5 (1.8.13) MPI-process 4223:
#000: H5Dio.c line 173 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: H5Dio.c line 545 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#002: H5Dchunk.c line 1861 in H5D__chunk_read(): unable to read raw data chunk
major: Low-level I/O
minor: Read failed
#003: H5Dchunk.c line 2891 in H5D__chunk_lock(): data pipeline read failed
major: Data filters
minor: Filter operation failed
#004: H5Z.c line 1383 in H5Z_pipeline(): filter returned failure during read
major: Data filters
minor: Read failed
#005: H5Zdeflate.c line 136 in H5Z_filter_deflate(): memory allocation failed for deflate uncompression
major: Resource unavailable
minor: No space available for allocation
Which has a similar call stack, and I wonder if it's trying to allocate using an undefined value. My gut tells me these may be related.
Is the valgrind issue I am reporting a known issue? Has anyone else run into this?
Can anyone provide some advice on how might continue debugging this?
Any thoughts on whether this is an issue with HDF5, zlib or some combination?
In my use case, it is extremely desirable to use compression as I am dealing with files that are 2.2GB compressed (4.5GB uncompressed). I can probably post some code if needed, but since the example has the same issue perhaps it is better to work from that.
My hope is that the root cause of this issue is:
A. Something in my usage of the library
B. Something wrong with the configure/build/install of the library (which I cannot necessarily change)
C. Something wrong with the HDF5 library
Any feedback is welcome.
Thanks,
-Brendan
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
