Heap-use-after-free by the call H5Aread when running with multiple threads in HDF5 thread safe version


#1

I have a program that accesses multiple HDF5 files for reading on Linux. When I use 1 thread to read from HDF5 files, the program works without a problem. However, when I run with multiple threads, the program crashes with a segmentation fault particularly when the number of threads is high (>16 threads). I am using the latest HDF5 1.10.5 version compiled with the thread-safety option enabled. To inspect where the segmentation fault is caused, I compiled HDF5 library with the gcc’s address sanitiser and it is identified as a heap-use-after-free by the the H5F_addr_decode function (called by H5Aread) - see the attached output below.

=================================================================
==26046==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0004daa68 at pc 0x55ea95a4606b bp 0x7f37b81657f0 sp 0x7f37b81657e0
READ of size 8 at 0x60d0004daa68 thread T58
    #0 0x55ea95a4606a in H5F_addr_decode /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2537
    #1 0x55ea962b1a71 in H5T_vlen_disk_isnull /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Tvlen.c:810
    #2 0x55ea96079888 in H5T__conv_vlen /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Tconv.c:3183
    #3 0x55ea9603218f in H5T_convert /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5T.c:5024
    #4 0x55ea956acfda in H5A__read /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Aint.c:652
    #5 0x55ea95697207 in H5Aread /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5A.c:669
    #6 0x55ea955f33d1 in fast5_get_string_attribute(fast5_file_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/nanopolish_fast5_io.c:358
    #7 0x55ea955b4348 in fast5_open src/fast5lite.h:95
    #8 0x55ea955b4348 in read_from_fast5_files2 src/f5c.c:646
    #9 0x55ea955b4348 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #10 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
    #11 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #12 0x7f37d844988e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

0x60d0004daa68 is located 72 bytes inside of 136-byte region [0x60d0004daa20,0x60d0004daaa8)
freed by thread T15 here:
    #0 0x7f37d97767b8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xde7b8)
    #1 0x55ea95cd5dce in H5MM_xfree /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:555
    #2 0x55ea95a3a709 in H5F__dest /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1353
    #3 0x55ea95a43095 in H5F_try_close /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2180
    #4 0x55ea95a41b7c in H5F__close_cb /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2009
    #5 0x55ea95c66af9 in H5I_dec_ref /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5I.c:1254
    #6 0x55ea95c66ebd in H5I_dec_app_ref /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5I.c:1299
    #7 0x55ea95a4100d in H5F__close /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1951
    #8 0x55ea95a0feaf in H5Fclose /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5F.c:674
    #9 0x55ea955b471f in fast5_close src/fast5lite.h:119
    #10 0x55ea955b471f in read_from_fast5_files2 src/f5c.c:671
    #11 0x55ea955b471f in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #12 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
    #13 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

previously allocated by thread T15 here:
    #0 0x7f37d9776b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x55ea95cd4c6e in H5MM_malloc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:292
    #2 0x55ea95cd518b in H5MM_calloc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:363
    #3 0x55ea95a33b9f in H5F__new /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:867
    #4 0x55ea95a3b77b in H5F_open /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1543
    #5 0x55ea95a0e089 in H5Fopen /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5F.c:508
    #6 0x55ea955b4133 in fast5_open src/fast5lite.h:91
    #7 0x55ea955b4133 in read_from_fast5_files2 src/f5c.c:646
    #8 0x55ea955b4133 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #9 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
    #10 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T58 created by T0 here:
    #0 0x7f37d96cfd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x55ea955b7b5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
    #2 0x55ea955be4bc in load_db(core_t*, db_t*) src/f5c.c:1322
    #3 0x55ea955a97fb in meth_main(int, char**, signed char) src/meth_main.c:448
    #4 0x55ea955a6a41 in main src/main.c:85
    #5 0x7f37d8349b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

Thread T15 created by T0 here:
    #0 0x7f37d96cfd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x55ea955b7b5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
    #2 0x55ea955be4bc in load_db(core_t*, db_t*) src/f5c.c:1322
    #3 0x55ea955a97fb in meth_main(int, char**, signed char) src/meth_main.c:448
    #4 0x55ea955a6a41 in main src/main.c:85
    #5 0x7f37d8349b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: heap-use-after-free /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2537 in H5F_addr_decode
Shadow bytes around the buggy address:
  0x0c1a800934f0: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x0c1a80093500: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
  0x0c1a80093510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a80093520: 00 fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
  0x0c1a80093530: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
=>0x0c1a80093540: fa fa fa fa fd fd fd fd fd fd fd fd fd[fd]fd fd
  0x0c1a80093550: fd fd fd fd fd fa fa fa fa fa fa fa fa fa 00 00
  0x0c1a80093560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c1a80093570: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1a80093580: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c1a80093590: fa fa 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==26046==ABORTING

This problematic H5Aread is associated with a variable-length string attribute where I access it as follows.

std::string out
if(H5Tis_variable_str(attribute_type) > 0) {
        // variable length string
        char* buffer;
        ret = H5Aread(attribute, native_type, &buffer);
        if(ret < 0) {
            fprintf(stderr, "error reading attribute %s\n", attribute_name.c_str());
            exit(EXIT_FAILURE);
        }
        out = buffer;
        free(buffer);
        buffer = NULL;

    } 

Note that I tried using H5free_memory(buffer) instead of free(), but still the program crashes and now with the address sanitiser pointing out to a heap-buffer-overflow inside H5free_memory - see below.

=================================================================
==30384==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000009020 at pc 0x7f03fadaaf54 bp 0x7f03f5cd3de0 sp 0x7f03f5cd3588
READ of size 4 at 0x602000009020 thread T2
    #0 0x7f03fadaaf53  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xaff53)
    #1 0x5604bc52b763 in H5MM__is_our_block /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:141
    #2 0x5604bc52cbf2 in H5MM_xfree /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:534
    #3 0x5604bbed6117 in H5free_memory /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5.c:979
    #4 0x5604bbe4a46a in fast5_get_string_attribute(fast5_file_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/nanopolish_fast5_io.c:364
    #5 0x5604bbe0b348 in fast5_open src/fast5lite.h:95
    #6 0x5604bbe0b348 in read_from_fast5_files2 src/f5c.c:646
    #7 0x5604bbe0b348 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #8 0x5604bbe046c9 in pthread_single2(void*) src/f5c.c:728
    #9 0x7f03fa6c26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #10 0x7f03f9aac88e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

0x602000009020 is located 8 bytes to the right of 8-byte region [0x602000009010,0x602000009018)
allocated by thread T2 here:
    #0 0x7f03fadd9d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
    #1 0x5604bcb0d12a in H5TS_cancel_count_inc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5TS.c:266
    #2 0x5604bbed47a1 in H5check_version /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5.c:738
    #3 0x5604bbe0b11e in fast5_open src/fast5lite.h:91
    #4 0x5604bbe0b11e in read_from_fast5_files2 src/f5c.c:646
    #5 0x5604bbe0b11e in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #6 0x5604bbe046c9 in pthread_single2(void*) src/f5c.c:728
    #7 0x7f03fa6c26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T2 created by T0 here:
    #0 0x7f03fad32d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x5604bbe0eb5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
    #2 0x5604bbe154bc in load_db(core_t*, db_t*) src/f5c.c:1322
    #3 0x5604bbe007fb in meth_main(int, char**, signed char) src/meth_main.c:448
    #4 0x5604bbdfda41 in main src/main.c:85
    #5 0x7f03f99acb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xaff53)
Shadow bytes around the buggy address:
  0x0c047fff91b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c047fff9200: fa fa 00 fa[fa]fa 00 fa fa fa 04 fa fa fa fa fa
  0x0c047fff9210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==30384==ABORTING

Am I doing something wrong somewhere?