Heap-use-after-free by the call H5Aread when running with multiple threads in HDF5 thread safe version


#1

I have a program that accesses multiple HDF5 files for reading on Linux. When I use 1 thread to read from HDF5 files, the program works without a problem. However, when I run with multiple threads, the program crashes with a segmentation fault particularly when the number of threads is high (>16 threads). I am using the latest HDF5 1.10.5 version compiled with the thread-safety option enabled. To inspect where the segmentation fault is caused, I compiled HDF5 library with the gcc’s address sanitiser and it is identified as a heap-use-after-free by the the H5F_addr_decode function (called by H5Aread) - see the attached output below.

=================================================================
==26046==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0004daa68 at pc 0x55ea95a4606b bp 0x7f37b81657f0 sp 0x7f37b81657e0
READ of size 8 at 0x60d0004daa68 thread T58
    #0 0x55ea95a4606a in H5F_addr_decode /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2537
    #1 0x55ea962b1a71 in H5T_vlen_disk_isnull /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Tvlen.c:810
    #2 0x55ea96079888 in H5T__conv_vlen /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Tconv.c:3183
    #3 0x55ea9603218f in H5T_convert /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5T.c:5024
    #4 0x55ea956acfda in H5A__read /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Aint.c:652
    #5 0x55ea95697207 in H5Aread /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5A.c:669
    #6 0x55ea955f33d1 in fast5_get_string_attribute(fast5_file_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/nanopolish_fast5_io.c:358
    #7 0x55ea955b4348 in fast5_open src/fast5lite.h:95
    #8 0x55ea955b4348 in read_from_fast5_files2 src/f5c.c:646
    #9 0x55ea955b4348 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #10 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
    #11 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #12 0x7f37d844988e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

0x60d0004daa68 is located 72 bytes inside of 136-byte region [0x60d0004daa20,0x60d0004daaa8)
freed by thread T15 here:
    #0 0x7f37d97767b8 in __interceptor_free (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xde7b8)
    #1 0x55ea95cd5dce in H5MM_xfree /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:555
    #2 0x55ea95a3a709 in H5F__dest /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1353
    #3 0x55ea95a43095 in H5F_try_close /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2180
    #4 0x55ea95a41b7c in H5F__close_cb /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2009
    #5 0x55ea95c66af9 in H5I_dec_ref /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5I.c:1254
    #6 0x55ea95c66ebd in H5I_dec_app_ref /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5I.c:1299
    #7 0x55ea95a4100d in H5F__close /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1951
    #8 0x55ea95a0feaf in H5Fclose /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5F.c:674
    #9 0x55ea955b471f in fast5_close src/fast5lite.h:119
    #10 0x55ea955b471f in read_from_fast5_files2 src/f5c.c:671
    #11 0x55ea955b471f in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #12 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
    #13 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

previously allocated by thread T15 here:
    #0 0x7f37d9776b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x55ea95cd4c6e in H5MM_malloc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:292
    #2 0x55ea95cd518b in H5MM_calloc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:363
    #3 0x55ea95a33b9f in H5F__new /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:867
    #4 0x55ea95a3b77b in H5F_open /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:1543
    #5 0x55ea95a0e089 in H5Fopen /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5F.c:508
    #6 0x55ea955b4133 in fast5_open src/fast5lite.h:91
    #7 0x55ea955b4133 in read_from_fast5_files2 src/f5c.c:646
    #8 0x55ea955b4133 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #9 0x55ea955ad6c9 in pthread_single2(void*) src/f5c.c:728
    #10 0x7f37d905f6da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T58 created by T0 here:
    #0 0x7f37d96cfd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x55ea955b7b5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
    #2 0x55ea955be4bc in load_db(core_t*, db_t*) src/f5c.c:1322
    #3 0x55ea955a97fb in meth_main(int, char**, signed char) src/meth_main.c:448
    #4 0x55ea955a6a41 in main src/main.c:85
    #5 0x7f37d8349b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

Thread T15 created by T0 here:
    #0 0x7f37d96cfd2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x55ea955b7b5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
    #2 0x55ea955be4bc in load_db(core_t*, db_t*) src/f5c.c:1322
    #3 0x55ea955a97fb in meth_main(int, char**, signed char) src/meth_main.c:448
    #4 0x55ea955a6a41 in main src/main.c:85
    #5 0x7f37d8349b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: heap-use-after-free /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5Fint.c:2537 in H5F_addr_decode
Shadow bytes around the buggy address:
  0x0c1a800934f0: fa fa fa fa fa fa fd fd fd fd fd fd fd fd fd fd
  0x0c1a80093500: fd fd fd fd fd fd fd fa fa fa fa fa fa fa fa fa
  0x0c1a80093510: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c1a80093520: 00 fa fa fa fa fa fa fa fa fa fd fd fd fd fd fd
  0x0c1a80093530: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
=>0x0c1a80093540: fa fa fa fa fd fd fd fd fd fd fd fd fd[fd]fd fd
  0x0c1a80093550: fd fd fd fd fd fa fa fa fa fa fa fa fa fa 00 00
  0x0c1a80093560: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 fa
  0x0c1a80093570: fa fa fa fa fa fa fa fa 00 00 00 00 00 00 00 00
  0x0c1a80093580: 00 00 00 00 00 00 00 00 00 fa fa fa fa fa fa fa
  0x0c1a80093590: fa fa 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==26046==ABORTING

This problematic H5Aread is associated with a variable-length string attribute where I access it as follows.

std::string out
if(H5Tis_variable_str(attribute_type) > 0) {
        // variable length string
        char* buffer;
        ret = H5Aread(attribute, native_type, &buffer);
        if(ret < 0) {
            fprintf(stderr, "error reading attribute %s\n", attribute_name.c_str());
            exit(EXIT_FAILURE);
        }
        out = buffer;
        free(buffer);
        buffer = NULL;

    } 

Note that I tried using H5free_memory(buffer) instead of free(), but still the program crashes and now with the address sanitiser pointing out to a heap-buffer-overflow inside H5free_memory - see below.

=================================================================
==30384==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x602000009020 at pc 0x7f03fadaaf54 bp 0x7f03f5cd3de0 sp 0x7f03f5cd3588
READ of size 4 at 0x602000009020 thread T2
    #0 0x7f03fadaaf53  (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xaff53)
    #1 0x5604bc52b763 in H5MM__is_our_block /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:141
    #2 0x5604bc52cbf2 in H5MM_xfree /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5MM.c:534
    #3 0x5604bbed6117 in H5free_memory /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5.c:979
    #4 0x5604bbe4a46a in fast5_get_string_attribute(fast5_file_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) src/nanopolish_fast5_io.c:364
    #5 0x5604bbe0b348 in fast5_open src/fast5lite.h:95
    #6 0x5604bbe0b348 in read_from_fast5_files2 src/f5c.c:646
    #7 0x5604bbe0b348 in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #8 0x5604bbe046c9 in pthread_single2(void*) src/f5c.c:728
    #9 0x7f03fa6c26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)
    #10 0x7f03f9aac88e in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x12188e)

0x602000009020 is located 8 bytes to the right of 8-byte region [0x602000009010,0x602000009018)
allocated by thread T2 here:
    #0 0x7f03fadd9d38 in __interceptor_calloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xded38)
    #1 0x5604bcb0d12a in H5TS_cancel_count_inc /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5TS.c:266
    #2 0x5604bbed47a1 in H5check_version /home/hasindu/hasindu2008.git/f5c-fastt2/hdf5/src/H5.c:738
    #3 0x5604bbe0b11e in fast5_open src/fast5lite.h:91
    #4 0x5604bbe0b11e in read_from_fast5_files2 src/f5c.c:646
    #5 0x5604bbe0b11e in read_fast5_single(core_t*, db_t*, int) src/f5c.c:807
    #6 0x5604bbe046c9 in pthread_single2(void*) src/f5c.c:728
    #7 0x7f03fa6c26da in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76da)

Thread T2 created by T0 here:
    #0 0x7f03fad32d2f in __interceptor_pthread_create (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x37d2f)
    #1 0x5604bbe0eb5d in pthread_db2(core_t*, db_t*, void (*)(core_t*, db_t*, int)) src/f5c.c:768
    #2 0x5604bbe154bc in load_db(core_t*, db_t*) src/f5c.c:1322
    #3 0x5604bbe007fb in meth_main(int, char**, signed char) src/meth_main.c:448
    #4 0x5604bbdfda41 in main src/main.c:85
    #5 0x7f03f99acb96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)

SUMMARY: AddressSanitizer: heap-buffer-overflow (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xaff53)
Shadow bytes around the buggy address:
  0x0c047fff91b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff91f0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c047fff9200: fa fa 00 fa[fa]fa 00 fa fa fa 04 fa fa fa fa fa
  0x0c047fff9210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9240: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c047fff9250: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==30384==ABORTING

Am I doing something wrong somewhere?


#2

I seem to be getting this same error as well on 1.10.6, the commonality being threads, thread safe (mt enabled) builds of hdf5, attributes with vlen data. Sometimes it crashes, sometimes it doesn’t. Only writing basic vlen strings to these attributes, nothing fancy.

Maybe a little different but very similar:

Segfaulting occasionally on :
PC: @ 0x555fd00f7ddd (unknown) H5F_addr_decode|
@ 0x7fd0b67d5390 1456 (unknown)|
@ 0x555fd0140be4 352 H5T__conv_vlen|
@ 0x555fd002c19f 96 H5T_convert|
@ 0x555fcffe45f5 144 H5A__read|
@ 0x555fcffde08f 64 H5Aread|

@epourmal you loop in anyone with good introspection here?


#3

Hello everyone,

Well… I think we have an issue. Could you please send us a program and a file to reproduce the issue. I am going to file an JIRA ticket and we will take a look.

Thank you for reporting!

Elena


#4

Here is the issue for tacking HDFFV-11080


#5

Hello!

We cannot reproduce the issue. Would it be possible for you to create one? I will keep the ticket open for a while.
Thank you!

Elena


#6

Sorry for the late reply. Was very busy. The dataset involved is a bit large. Is there a particular place that I can upload the source and the data?


#7

Hello!

I sent you instructions through the HDF Helpdesk on how you can send us the file.

Thanks!
-Barbara


#8

It has been uploaded. To reproduce the issue, the tarball must be extracted and there is a script named reproduce.sh that should be run. If there is any issues let me know.