I have a program that accesses multiple HDF5 files for reading on Linux. When I use 1 thread to read from HDF5 files, the program works without a problem. However, when I run with multiple threads, the program crashes with a segmentation fault particularly when the number of threads is high (>16 threads). I am using the latest HDF5 1.10.5 version compiled with the thread-safety option enabled. To inspect where the segmentation fault is caused, I compiled HDF5 library with the gcc’s address sanitiser and it is identified as a heap-use-after-free by the the H5F_addr_decode function (called by H5Aread) - see the attached output below.
=================================================================
==26046==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0004daa68 at pc 0x55ea95a4606b bp 0x7f37b81657f0 sp 0x7f37b81657e0
READ of size 8 at 0x60d0004daa68 thread T58
#0 0x55ea95a4606a in H5F_addr_decode /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2537
#1 0x55ea962b1a71 in H5T_vlen_disk_isnull /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Tvlen.c:810
#2 0x55ea96079888 in H5T__conv_vlen /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Tconv.c:3183
#3 0x55ea9603218f in H5T_convert /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5T.c:5024
#4 0x55ea956acfda in H5A__read /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Aint.c:652
#5 0x55ea95697207 in H5Aread /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5A.c:669
#6 0x55ea955f33d1 in fast5_get_string_attribute(fast5_file_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/nanopolish_fast5_io.c:358
#7 0x55ea955b4348 in fast5_open src/fast5lite.h:95
#8 0x55ea955b4348 in read_from_fast5_files2 src/f5c.c:646
#9 0x55ea955b4348 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
#10 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
#11 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#12 0x7f37d844988e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)
0x60d0004daa68 is located 72 bytes inside of 136-byte region [0x60d0004daa20,0x60d0004daaa8)
freed by thread T15 here:
#0 0x7f37d97767b8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xde7b8)
#1 0x55ea95cd5dce in H5MM_xfree /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:555
#2 0x55ea95a3a709 in H5F__dest /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1353
#3 0x55ea95a43095 in H5F_try_close /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2180
#4 0x55ea95a41b7c in H5F__close_cb /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2009
#5 0x55ea95c66af9 in H5I_dec_ref /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5I.c:1254
#6 0x55ea95c66ebd in H5I_dec_app_ref /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5I.c:1299
#7 0x55ea95a4100d in H5F__close /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1951
#8 0x55ea95a0feaf in H5Fclose /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5F.c:674
#9 0x55ea955b471f in fast5_close src/fast5lite.h:119
#10 0x55ea955b471f in read_from_fast5_files2 src/f5c.c:671
#11 0x55ea955b471f in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
#12 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
#13 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
previously allocated by thread T15 here:
#0 0x7f37d9776b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
#1 0x55ea95cd4c6e in H5MM_malloc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:292
#2 0x55ea95cd518b in H5MM_calloc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:363
#3 0x55ea95a33b9f in H5F__new /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:867
#4 0x55ea95a3b77b in H5F_open /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1543
#5 0x55ea95a0e089 in H5Fopen /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5F.c:508
#6 0x55ea955b4133 in fast5_open src/fast5lite.h:91
#7 0x55ea955b4133 in read_from_fast5_files2 src/f5c.c:646
#8 0x55ea955b4133 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
#9 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
#10 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
Thread T58 created by T0 here:
#0 0x7f37d96cfd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
#1 0x55ea955b7b5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
#2 0x55ea955be4bc in load_db(core_t*, db_t*) src/f5c.c:1322
#3 0x55ea955a97fb in meth_main(int, char**, signed char) src/meth_main.c:448
#4 0x55ea955a6a41 in main src/main.c:85
#5 0x7f37d8349b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
Thread T15 created by T0 here:
#0 0x7f37d96cfd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
#1 0x55ea955b7b5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
#2 0x55ea955be4bc in load_db(core_t*, db_t*) src/f5c.c:1322
#3 0x55ea955a97fb in meth_main(int, char**, signed char) src/meth_main.c:448
#4 0x55ea955a6a41 in main src/main.c:85
#5 0x7f37d8349b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
SUMMARY: AddressSanitizer: heap-use-after-free /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2537 in H5F_addr_decode
Shadow bytes around the buggy address:
0x0c1a800934f0: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
0x0c1a80093500: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
0x0c1a80093510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c1a80093520: 00 fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
0x0c1a80093530: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
=>0x0c1a80093540: fa fa fa fa fd fd fd fd fd fd fd fd fd[fd]fd fd
0x0c1a80093550: fd fd fd fd fd fa fa fa fa fa fa fa fa fa 00 00
0x0c1a80093560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
0x0c1a80093570: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
0x0c1a80093580: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
0x0c1a80093590: fa fa 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==26046==ABORTING
This problematic H5Aread is associated with a variable-length string attribute where I access it as follows.
std::string out
if(H5Tis_variable_str(attribute_type) > 0) {
// variable length string
char* buffer;
ret = H5Aread(attribute, native_type, &buffer);
if(ret < 0) {
fprintf(stderr, "error reading attribute %s\n", attribute_name.c_str());
exit(EXIT_FAILURE);
}
out = buffer;
free(buffer);
buffer = NULL;
}
Note that I tried using H5free_memory(buffer) instead of free(), but still the program crashes and now with the address sanitiser pointing out to a heap-buffer-overflow inside H5free_memory - see below.
=================================================================
==30384==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000009020 at pc 0x7f03fadaaf54 bp 0x7f03f5cd3de0 sp 0x7f03f5cd3588
READ of size 4 at 0x602000009020 thread T2
#0 0x7f03fadaaf53 (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xaff53)
#1 0x5604bc52b763 in H5MM__is_our_block /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:141
#2 0x5604bc52cbf2 in H5MM_xfree /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:534
#3 0x5604bbed6117 in H5free_memory /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5.c:979
#4 0x5604bbe4a46a in fast5_get_string_attribute(fast5_file_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/nanopolish_fast5_io.c:364
#5 0x5604bbe0b348 in fast5_open src/fast5lite.h:95
#6 0x5604bbe0b348 in read_from_fast5_files2 src/f5c.c:646
#7 0x5604bbe0b348 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
#8 0x5604bbe046c9 in pthread_single2(void*) src/f5c.c:728
#9 0x7f03fa6c26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
#10 0x7f03f9aac88e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)
0x602000009020 is located 8 bytes to the right of 8-byte region [0x602000009010,0x602000009018)
allocated by thread T2 here:
#0 0x7f03fadd9d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
#1 0x5604bcb0d12a in H5TS_cancel_count_inc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5TS.c:266
#2 0x5604bbed47a1 in H5check_version /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5.c:738
#3 0x5604bbe0b11e in fast5_open src/fast5lite.h:91
#4 0x5604bbe0b11e in read_from_fast5_files2 src/f5c.c:646
#5 0x5604bbe0b11e in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
#6 0x5604bbe046c9 in pthread_single2(void*) src/f5c.c:728
#7 0x7f03fa6c26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
Thread T2 created by T0 here:
#0 0x7f03fad32d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
#1 0x5604bbe0eb5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
#2 0x5604bbe154bc in load_db(core_t*, db_t*) src/f5c.c:1322
#3 0x5604bbe007fb in meth_main(int, char**, signed char) src/meth_main.c:448
#4 0x5604bbdfda41 in main src/main.c:85
#5 0x7f03f99acb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xaff53)
Shadow bytes around the buggy address:
0x0c047fff91b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff91c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff91d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff91e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff91f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c047fff9200: fa fa 00 fa[fa]fa 00 fa fa fa 04 fa fa fa fa fa
0x0c047fff9210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==30384==ABORTING
Am I doing something wrong somewhere?