Crash when freeing user-provided buffer on filter callback

HDF5 Library
1

Hello there!

I’ve noticed that, on recent development snapshots of HDF5, data reads from chunked datasets are subject to different optimization that were not seen on 1.10.0. In special, the filter callback may no longer receive a pointer to the start of a heap-allocated buffer, but rather a pointer to an offset from the beginning of that buffer. This seems to be coming from H5D__chunk_read(), which allocates an internal buffer to hold the chunk data filled up by the filter callback routine.

The problem is that when we get a pointer to the middle of a malloc()d buffer, filters are no longer allowed to free it and replace it with something else – the code crashes immediately otherwise. My code does something like this:

size_t filter_callback(flags, cd_nelmts, *cd_values, nbytes, *buf_size, **buf) {
    if (! (flags & H5Z_FLAG_REVERSE)) {
        // transform the input data
        *buf_size = new_size;
        return new_size;
    } else {
        // read data from disk
        free(*buf);
        *buf = data_read;
        *buf_size = data_read_size;
    }
}

What I have observed is that if new_size is small enough (say, 200 bytes, while the file is compressed with a chunk size of 1MB), then the call to free(*buf) crashes the program. I’m not sure if this is relevant, yet, as I couldn’t dig any further.

Is this new behavior intentional?

Thanks,
Lucas

1 Like
2

Lucas,

Thank you for bringing this issue to our attention. I entered a JIRA task https://jira.hdfgroup.org/browse/HDFFV-10936 to investigate. We shouldn’t be crashing users programs :wink:

Thanks again!

Elena

1 Like
3

Elena, I have just updated the git snapshot and the filter is working with no crashes this time. Maybe the repository was in an inconsistent state when I cloned it? Anyway, please feel free to close the ticket on Jira, and thanks a lot for your prompt help!

Best regards,
Lucas

4

Lucas,

We rarely have repository in an inconsistent state. All changes are thoroughly tested before they are checked in. I will leave the ticket open for now since we have too many reports for memory issues when using filters :thinking:

Thank you!
Elena

1 Like
5

Hi Elena!

It looks like it’s still happening – but oddly enough, only one of my machines is exhibiting the behavior. I wonder if libc is playing a crucial role here. Here are some details I could get from the problem:

  • H5D__chunk_lock is called, and the function determines that the chunk exists on disk, but isn’t the same size as the final chunk in memory. As a result, the function decides to allocate a new chunk to provide to the filter callback [https://github.com/live-clones/hdf5/blob/develop/src/H5Dchunk.c#L3956]
  • H5D__chunk_mem_alloc is called to allocate memory. Since pline->used == 1, allocation goes through H5MM_malloc()
  • The filter callback is called and attempts to free(*buf). Since that buffer is managed by HDF5, the application cannot free it, as the pointer passed back to the application is really just an offset within an internal HDF5 structure.

I’ve looked into the user buffer provided to the callback and it’s possible to see the magic id in the bytes that precede that buffer:

(gdb) xxd *buf-8 16
00000000: 4445 4144 4245 4546 7b22 6461 7461 7365  DEADBEEF{"datase

It looks like a raw buffer should be provided to the callback instead.

Here is the relevant stack trace:

#0  0x00007ffff74eb202 in free () from /lib64/libc.so.6
#1  0x00007ffff738d341 in readDataset<int> (names=std::vector of length 0, capacity 0, file_id=72057594037927937, n_elements=10000, hdf5_datatype=216172782113783872, buf_size=0x7fffffffd4c8, 
    buf=0x7fffffffd4e8, nbytes=@0x7fffffffd130: 771, payload=0x4c11a0 "\033LJ\002\nl", payload_size=675, dtype=0x7ffff73ba141 "int32_t*") at hdf5-filter.cpp:261
#2  0x00007ffff7389eda in filter_callback (flags=256, cd_nelmts=0, cd_values=0x0, nbytes=771, buf_size=0x7fffffffd4c8, buf=0x7fffffffd4e8) at hdf5-filter.cpp:307
#3  0x00007ffff7e22a00 in H5Z_pipeline (pline=pline@entry=0x4a9408, flags=flags@entry=256, filter_mask=filter_mask@entry=0x7fffffffd620, edc_read=H5Z_ENABLE_EDC, cb_struct=..., 
    nbytes=0x7fffffffd4c0, buf_size=0x7fffffffd4c8, buf=0x7fffffffd4e8) at H5Z.c:1322
#4  0x00007ffff7ab7d05 in H5D__chunk_lock (io_info=io_info@entry=0x7fffffffd9b0, udata=udata@entry=0x7fffffffd5f0, relax=relax@entry=false, prev_unfilt_chunk=prev_unfilt_chunk@entry=false)
    at H5Dchunk.c:3971
#5  0x00007ffff7ab9d71 in H5D__chunk_read (io_info=0x7fffffffd9b0, type_info=0x7fffffffd930, nelmts=<optimized out>, file_space=<optimized out>, mem_space=<optimized out>, fm=0x4bfb40)
    at H5Dchunk.c:2608
#6  0x00007ffff7ae4261 in H5D__read (dataset=dataset@entry=0x4a92b0, mem_type_id=mem_type_id@entry=216172782113784248, mem_space=0x4ac600, file_space=0x4ac600, buf=<optimized out>, 
    std::vector<std::string> dataset_names;
    for (auto name : names)
    {
        auto dataset = filterCallback<T>(file_id, name, n_elements, hdf5_datatype);
    buf@entry=0x4b5e70) at H5Dio.c:569
#7  0x00007ffff7e10cf9 in H5VL__native_dataset_read (obj=0x4a92b0, mem_type_id=216172782113784248, mem_space_id=0, file_space_id=0, dxpl_id=<optimized out>, buf=0x4b5e70, req=0x0)
    at H5VLnative_dataset.c:166
#8  0x00007ffff7df0afe in H5VL__dataset_read (obj=0x4a92b0, cls=0x438840, mem_type_id=mem_type_id@entry=216172782113784248, mem_space_id=mem_space_id@entry=0, 
    file_space_id=file_space_id@entry=0, plist_id=plist_id@entry=792633534417207304, buf=0x4b5e70, req=0x0) at H5VLcallback.c:2028
#9  0x00007ffff7dfa7b5 in H5VL_dataset_read (vol_obj=vol_obj@entry=0x4b0420, mem_type_id=mem_type_id@entry=216172782113784248, mem_space_id=mem_space_id@entry=0, 
    file_space_id=file_space_id@entry=0, plist_id=plist_id@entry=792633534417207304, buf=buf@entry=0x4b5e70, req=0x0) at H5VLcallback.c:2062
#10 0x00007ffff7ae2564 in H5Dread (dset_id=<optimized out>, mem_type_id=216172782113784248, mem_space_id=0, file_space_id=0, dxpl_id=792633534417207304, buf=0x4b5e70) at H5Dio.c:191
6

Hi Lucas,

Developers discussed the issue and concluded that we would need to revert the usage of H5MM_malloc(free) when working with the chunks. I entered a JIRA issue HDFFV-10948 and we will try to address it in our next maintenance releases in Spring 2020 (or earlier, but I cannot promise).

Thank you for reporting!

Elena

7

Thank you for the update! It’s great to know that the team has come up with a plan to fix the issue.

Best regards,
Lucas

8

Hi there,

I have encountered the same problem with HDF 1.12.0, but not with 1.10.6. I added to the minimal C program from Crash when writing parallel compressed chunks to reproduce the problem. I hope this helps:

#include <stdlib.h>

#include <stdio.h>

#include <string.h>

​

#include "hdf5.h"

​

#define _MPI

//#define _DSET2

//#define _COMPRESS

#define _FILTER

​

#define NPROC 4

​

#define NDIM 2

#define CHUNK0 4

​

// Equivalent to original gist

// Works on 1.10.5 with patch, crashes on 1.10.5 vanilla and hangs on 1.10.6

#define CHUNK1 32768

#define NCHUNK1 32

​

// Works on 1.10.5 with and without patch and 1.10.6

//#define CHUNK1 256

//#define NCHUNK1 8192

​

// Works on 1.10.5 with and without patch and 1.10.6

//#define CHUNK1 512

//#define NCHUNK1 8192

​

// Crashes on 1.10.5 with and without patch, 1.10.6 and 1.12.0

//#define CHUNK1 256

//#define NCHUNK1 16384

​

// 3 dimensional example

//#define NDIM 3

//#define CHUNK0 16

//#define NCHUNK1 28

//#define CHUNK1 256

//#define NCHUNK2 4096/128

//#define CHUNK2 128

​

​

#ifdef _FILTER

#define TEST_FILTER 305

​

size_t

minimal_filter(unsigned int flags, size_t cd_nelmts,

           const unsigned int cd_values[], size_t nbytes,

           size_t *buf_size, void **buf)

{

    // memcpy the input buffer and free it

    void* out_buf = malloc(*buf_size);

    memcpy((char*)(*buf), out_buf, nbytes);

    free(*buf);

    *buf = out_buf;

    out_buf = NULL;

    return nbytes;

}

​

H5Z_class_t minimal_H5Filter[1] = {{

    H5Z_CLASS_T_VERS,

    (H5Z_filter_t)(TEST_FILTER),

    1, 1,

    "minimal test filter",

    NULL,

    NULL,

    (H5Z_func_t)(minimal_filter)

}};

#endif

​

​

int main(int argc, char **argv) {

​

    int mpi_size = 1, mpi_rank = 0;

    hid_t fapl_id, file_id, dset_space, dcpl_id, ds, ds2, propid, mem_dspace, sel_dspace;

​

    fapl_id = H5Pcreate(H5P_FILE_ACCESS);

#ifdef _MPI

    MPI_Comm comm  = MPI_COMM_WORLD;

    MPI_Info info  = MPI_INFO_NULL;

​

    MPI_Init(&argc, &argv);

    MPI_Comm_size(comm, &mpi_size);

    MPI_Comm_rank(comm, &mpi_rank);

​

    H5Pset_fapl_mpio(fapl_id, comm, info);

#endif

​

    printf("MPI rank [%i/%i]\n", mpi_rank, mpi_size);

​

    printf("rank=%i creating file\n", mpi_rank);

    file_id = H5Fcreate("test1.h5", H5F_ACC_TRUNC, H5P_DEFAULT, fapl_id);

    H5Pclose(fapl_id);

​

    // Define total dataset size

#if NDIM==3

    hsize_t dset_dims[NDIM] = {NPROC * CHUNK0, CHUNK1 * NCHUNK1, CHUNK2 * NCHUNK2};

#elif NDIM==2

    hsize_t dset_dims[NDIM] = {NPROC * CHUNK0, CHUNK1 * NCHUNK1};

#else

   return 0;

#endif

    dset_space = H5Screate_simple (NDIM, dset_dims, NULL);

    dcpl_id = H5Pcreate(H5P_DATASET_CREATE);

​

#ifdef _FILTER

    // register and set minimal test filter

    herr_t status = H5Zregister(minimal_H5Filter);

    if (status < 0) {

        printf("Failed to register filter. code %i\n", status);

        return 1;

    }

    status = H5Pset_filter(dcpl_id, TEST_FILTER, 0, 0, NULL);

    if (status < 0) {

        printf("Failed to set filter. code %i\n", status);

        return 1;

    }

#endif

​

    // Set chunking and compression params

#if NDIM==3

    hsize_t chunk_dims[NDIM] = {CHUNK0, CHUNK1, CHUNK2};

#elif NDIM==2

    hsize_t chunk_dims[NDIM] = {CHUNK0, CHUNK1};

#else

    return 0;

#endif

    H5Pset_chunk(dcpl_id, NDIM, chunk_dims);

​

#ifdef _COMPRESS

    H5Pset_deflate(dcpl_id, 9);

#endif

​

    // Define selection

#if NDIM==3

    hsize_t sel_dims[NDIM] = {CHUNK0, CHUNK1 * NCHUNK1, CHUNK2 * NCHUNK2};

    hsize_t offset[NDIM] = {mpi_rank * sel_dims[0], 0, 0};

    printf("rank=%i creating selection [%llu:%llu, %llu:%llu, %llu:%llu]\n",

           mpi_rank, offset[0], offset[0] + sel_dims[0], offset[1], offset[1] + sel_dims[1], offset[2], offset[2] + sel_dims[2]);

#elif NDIM==2

    hsize_t sel_dims[NDIM] = {CHUNK0, CHUNK1 * NCHUNK1};

    hsize_t offset[NDIM] = {mpi_rank * sel_dims[0], 0};

    printf("rank=%i creating selection [%llu:%llu, %llu:%llu]\n",

           mpi_rank, offset[0], offset[0] + sel_dims[0], offset[1], offset[1] + sel_dims[1]);

#else

    return 0;

#endif

    sel_dspace = H5Screate_simple(NDIM, dset_dims, NULL);

    H5Sselect_hyperslab(sel_dspace, H5S_SELECT_SET,

                        offset, NULL, sel_dims, NULL);

​

    // Set the dspace for the input data

    mem_dspace = H5Screate_simple (NDIM, sel_dims, NULL);

​

    propid = H5Pcreate(H5P_DATASET_XFER);

#ifdef _MPI

    H5Pset_dxpl_mpio(propid, H5FD_MPIO_COLLECTIVE);

#endif

​

    // Create the dataset

    printf("rank=%i creating dataset1\n", mpi_rank);

    ds = H5Dcreate (file_id, "dset1", H5T_NATIVE_FLOAT, dset_space, H5P_DEFAULT, dcpl_id, H5P_DEFAULT);

​

    // Create array of data to write

    // Initialise with random data.

#if NDIM==3

    int totalsize = sel_dims[0] * sel_dims[1] * sel_dims[2];

#elif NDIM==2

    int totalsize = sel_dims[0] * sel_dims[1];

#else

    return 0;

#endif

    float *data = (float *)malloc(sizeof(float) * totalsize);

    for(int i = 0; i < totalsize; i++) {

        data[i] = (float)drand48();

    }

​

    printf("rank=%i writing dataset1\n", mpi_rank);

    H5Dwrite(ds, H5T_NATIVE_FLOAT, mem_dspace, sel_dspace, propid, data);

    printf("rank=%i finished writing dataset1\n", mpi_rank);

​

#ifdef _DSET2

    // Create the dataset

    printf("rank=%i creating dataset2\n", mpi_rank);

    ds2 = H5Dcreate (file_id, "dset2", H5T_NATIVE_FLOAT, dset_space, H5P_DEFAULT, dcpl_id, H5P_DEFAULT);

    

    // Generate new data and write

    printf("rank=%i writing dataset2\n", mpi_rank);

    for(int i = 0; i < totalsize; i++) {

        data[i] = drand48();

    }

    H5Dwrite(ds2, H5T_NATIVE_FLOAT, mem_dspace, sel_dspace, propid, data);

    

    H5Dclose(ds2);

#endif

​

    // Close down everything

    printf("rank=%i closing everything\n", mpi_rank);

    H5Dclose(ds);

    H5Sclose(dset_space);

    H5Sclose(sel_dspace);

    H5Sclose(mem_dspace);

    H5Pclose(dcpl_id);

    H5Pclose(propid);

    H5Fclose(file_id);

​

    free(data);

​

    MPI_Finalize();

    return 0;

 }

This is the output with the address sanitizer enabled:

MPI rank [0/4]

rank=0 creating file

MPI rank [1/4]

rank=1 creating file

MPI rank [2/4]

rank=2 creating file

MPI rank [3/4]

rank=3 creating file

rank=0 creating selection [0:16, 0:7168, 0:4096]

rank=0 creating dataset1

rank=1 creating selection [16:32, 0:7168, 0:4096]

rank=2 creating selection [32:48, 0:7168, 0:4096]

rank=3 creating selection [48:64, 0:7168, 0:4096]

rank=2 creating dataset1

rank=1 creating dataset1

rank=3 creating dataset1

=================================================================

==135802==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x2afbfd586830 in thread T0

=================================================================

==135803==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x2b5022fa9830 in thread T0

=================================================================

=================================================================

==135805==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x2b9116cf3830 in thread T0

==135804==ERROR: AddressSanitizer: attempting free on address which was not malloc()-ed: 0x2b2d6adf3830 in thread T0

    #0 0x2afbf0f9d487 in free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487)

    #1 0x428c4e in minimal_filter /scratch/rickn/test_hdf5/test_filter.c:35

    #2 0x739ef3 in H5Z_pipeline /scratch/rickn/hdf5/src/H5Z.c:1366

    #0 0x2b2d5e80c487 in free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487)

    #0 0x2b910a707487 in free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487)

    #0 0x2b50169bc487 in free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487)

    #1 0x428c4e in minimal_filter /scratch/rickn/test_hdf5/test_filter.c:35

    #1 0x428c4e in minimal_filter /scratch/rickn/test_hdf5/test_filter.c:35

    #1 0x428c4e in minimal_filter /scratch/rickn/test_hdf5/test_filter.c:35

    #2 0x739ef3 in H5Z_pipeline /scratch/rickn/hdf5/src/H5Z.c:1366

    #2 0x739ef3 in H5Z_pipeline /scratch/rickn/hdf5/src/H5Z.c:1366

    #2 0x739ef3 in H5Z_pipeline /scratch/rickn/hdf5/src/H5Z.c:1366

    #3 0x789f89 in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4473

    #3 0x789f89 in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4473

    #3 0x789f89 in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4473

    #3 0x789f89 in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4473

    #4 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #5 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #6 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #7 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #8 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #9 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #4 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #4 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #4 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #5 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #5 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #5 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #6 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #6 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #6 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #7 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #8 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #7 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #8 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #7 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #10 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #8 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #9 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #9 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #9 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #10 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #11 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #10 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #10 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #12 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #13 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #14 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #15 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #16 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #17 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #11 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #11 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #11 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #12 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #12 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #13 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #14 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #13 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #12 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #15 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #14 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #15 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #16 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #13 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #16 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #14 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #15 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #16 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #17 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #17 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #17 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #18 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #19 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #20 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #21 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #22 0x2afbf28ac2df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

    #23 0x428b09 in _start (/scratch/rickn/test_hdf5/testh5+0x428b09)

0x2afbfd586830 is located 48 bytes inside of 2097208-byte region [0x2afbfd586800,0x2afbfd786838)

allocated by thread T0 here:

    #0 0x2afbf0f9d838 in malloc (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106838)

    #18 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #1 0x5903a1 in H5MM_malloc /scratch/rickn/hdf5/src/H5MM.c:292

    #19 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #18 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #2 0x77bc3f in H5D__chunk_mem_alloc /scratch/rickn/hdf5/src/H5Dchunk.c:1404

    #19 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #18 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #19 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #3 0x79de6d in H5D__fill_init /scratch/rickn/hdf5/src/H5Dfill.c:525

    #4 0x789e9a in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4437

    #5 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #6 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #7 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #8 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #9 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #10 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #20 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #11 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #12 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #13 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #20 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #21 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #14 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #15 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #21 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #22 0x2b50182cb2df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

    #22 0x2b2d6011b2df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

    #23 0x428b09 in _start (/scratch/rickn/test_hdf5/testh5+0x428b09)

0x2b2d6adf3830 is located 48 bytes inside of 2097208-byte region [0x2b2d6adf3800,0x2b2d6aff3838)

    #23 0x428b09 in _start (/scratch/rickn/test_hdf5/testh5+0x428b09)

    #16 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #17 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #18 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #20 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #21 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #22 0x2b910c0162df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

allocated by thread T0 here:

    #0 0x2b2d5e80c838 in malloc (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106838)

0x2b5022fa9830 is located 48 bytes inside of 2097208-byte region [0x2b5022fa9800,0x2b50231a9838)

allocated by thread T0 here:

    #19 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #20 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #21 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #22 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #0 0x2b50169bc838 in malloc (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106838)

    #23 0x428b09 in _start (/scratch/rickn/test_hdf5/testh5+0x428b09)

0x2b9116cf3830 is located 48 bytes inside of 2097208-byte region [0x2b9116cf3800,0x2b9116ef3838)

    #1 0x5903a1 in H5MM_malloc /scratch/rickn/hdf5/src/H5MM.c:292

    #2 0x77bc3f in H5D__chunk_mem_alloc /scratch/rickn/hdf5/src/H5Dchunk.c:1404

    #23 0x2afbf28ac2df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

SUMMARY: AddressSanitizer: bad-free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487) in free

allocated by thread T0 here:

    #1 0x5903a1 in H5MM_malloc /scratch/rickn/hdf5/src/H5MM.c:292

    #2 0x77bc3f in H5D__chunk_mem_alloc /scratch/rickn/hdf5/src/H5Dchunk.c:1404

==135802==ABORTING

    #0 0x2b910a707838 in malloc (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106838)

    #1 0x5903a1 in H5MM_malloc /scratch/rickn/hdf5/src/H5MM.c:292

    #2 0x77bc3f in H5D__chunk_mem_alloc /scratch/rickn/hdf5/src/H5Dchunk.c:1404

    #3 0x79de6d in H5D__fill_init /scratch/rickn/hdf5/src/H5Dfill.c:525

    #4 0x789e9a in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4437

    #5 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #3 0x79de6d in H5D__fill_init /scratch/rickn/hdf5/src/H5Dfill.c:525

    #6 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #4 0x789e9a in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4437

    #7 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #5 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #8 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #9 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #6 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #7 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #8 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #9 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #10 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #11 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #12 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #3 0x79de6d in H5D__fill_init /scratch/rickn/hdf5/src/H5Dfill.c:525

    #4 0x789e9a in H5D__chunk_allocate /scratch/rickn/hdf5/src/H5Dchunk.c:4437

    #5 0x4ae27f in H5D__init_storage /scratch/rickn/hdf5/src/H5Dint.c:2473

    #10 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #11 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #12 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #13 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #13 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #14 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #15 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #16 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #6 0x4b3d31 in H5D__alloc_storage /scratch/rickn/hdf5/src/H5Dint.c:2386

    #14 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #15 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #16 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #17 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #18 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #19 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #17 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #18 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #19 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #20 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #21 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #22 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #7 0x4bc06e in H5D__layout_oh_create /scratch/rickn/hdf5/src/H5Dlayout.c:507

    #8 0x4ad1c4 in H5D__update_oh_info /scratch/rickn/hdf5/src/H5Dint.c:1070

    #9 0x4afa39 in H5D__create /scratch/rickn/hdf5/src/H5Dint.c:1373

    #10 0x79e984 in H5O__dset_create /scratch/rickn/hdf5/src/H5Doh.c:300

    #11 0x5b62df in H5O_obj_create /scratch/rickn/hdf5/src/H5Oint.c:2521

    #12 0x57ef66 in H5L__link_cb /scratch/rickn/hdf5/src/H5L.c:1850

    #23 0x2b2d6011b2df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

    #20 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

    #13 0x540a49 in H5G__traverse_real /scratch/rickn/hdf5/src/H5Gtraverse.c:629

    #14 0x5416c8 in H5G_traverse /scratch/rickn/hdf5/src/H5Gtraverse.c:854

    #15 0x5781d7 in H5L__create_real /scratch/rickn/hdf5/src/H5L.c:2044

    #16 0x580cfd in H5L_link_object /scratch/rickn/hdf5/src/H5L.c:1803

    #17 0x4aed95 in H5D__create_named /scratch/rickn/hdf5/src/H5Dint.c:410

    #18 0x728b3d in H5VL__native_dataset_create /scratch/rickn/hdf5/src/H5VLnative_dataset.c:74

    #19 0x708ac5 in H5VL__dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1834

    #20 0x7115ae in H5VL_dataset_create /scratch/rickn/hdf5/src/H5VLcallback.c:1868

SUMMARY: AddressSanitizer: bad-free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487) in free

==135804==ABORTING

    #21 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

    #22 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #23 0x2b50182cb2df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

SUMMARY: AddressSanitizer: bad-free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487) in free

    #21 0x4a3322 in H5Dcreate2 /scratch/rickn/hdf5/src/H5D.c:150

==135803==ABORTING

    #22 0x4295bd in main /scratch/rickn/test_hdf5/test_filter.c:122

    #23 0x2b910c0162df in __libc_start_main (/cvmfs/soft.computecanada.ca/nix/store/63pk88rnmkzjblpxydvrmskkc8ci7cx6-glibc-2.24/lib/libc.so.6+0x202df)

SUMMARY: AddressSanitizer: bad-free (/cvmfs/soft.computecanada.ca/nix/var/nix/profiles/16.09/lib/libasan.so.5+0x106487) in free

==135805==ABORTING

--------------------------------------------------------------------------

Primary job  terminated normally, but 1 process returned

a non-zero exit code. Per user-direction, the job has been aborted.

--------------------------------------------------------------------------

--------------------------------------------------------------------------

mpirun detected that one or more processes exited with non-zero status, thus causing

the job to be terminated. The first process to do so was:

  Process name: [[9035,1],0]

  Exit code:    1

--------------------------------------------------------------------------

Cheers,
Rick