Writing Variable Length Data To Compound Dataset

HDF5 Library C++ Users' Group
1

Hello,

I am writing a program that converts google protocol buffer messages into the HDF5 file format via the CPP interface, and I seem to be hitting an issue when attempting to copy strings + other variable length fields.

I have a compound type compType that I add a member to as follows

// Note: I am calculating the offset by hand since I do not have the necessary structures at compile time
H5::DataType v = H5::StrType(H5::PredType::C_S1, H5T_VARIABLE);
compType.insertMember(fieldName, currentOffset, v);
currentOffset += sizeof(char *);

I write the structured data to a buffer containg the rest of my compound type as follows

char * value = calloc(1, src.length()+1);
memcpy(value, src, src.length());
buffer.replace(offset, sizeof(char *), &value, sizeof(char *));

Then I write the buffers containing my compound data to a dataset

dataSet.write(buffer, compType);

This process works for all of the fixed length data types I want to support, but I seem to get either bad data or segfaults when writing no matter what approach I use for strings / var length types. Wondering what the correct approach to do something like this would be / what type to use + what goes in the binary buffer?

NOTE: please don’t suggest HDFql, it’s not available to me

EDIT: the approach above segfaults inside some vlen conversion function that I don’t have debug symbols for. Another approach I have tried is as follows:

Create the compound type

// Note: I am calculating the offset by hand since I do not have the necessary structures at compile time
H5::VarLenType v = H5::VarLenType(H5::PredType::C_S1);
compType.insertMember(fieldName, currentOffset, v);
currentOffset += sizeof(hvl_t);

Copy the data from std::string

hvl_t vlInfo;
vlInfo.len = value.length() + 1;
vlInfo.p = calloc(1, value.length() + 1);
memcpy(vlInfo.p, value.c_str(), value.length());

buffer.replace(offset, sizeof(hvl_t), (char *)&vlInfo, sizeof(hvl_t));

Write to file

dataSet.write(buffer, compType);

This approach does not segfault but all of my strings show up as ERROR in HDFView

image

image

2

Hi Nicholas,

I don’t think you want sizeof(char *), that’s only the size of the pointer.
Also, the last argument of insertMember is the datatype of the new member.

4

Hi,

It was my understanding for variable length types you either use a pointer or an item of hvl_t. Regardless I don’t really find this answer to be all that helpful since I know what I was doing was wrong.

I was really hoping the C++ lib has the ability to put variable length types inside a compound type, and that someone could give advice on how to do that not just tell me the code I said isn’t working doesn’t work.

5

I understand that not having a minimal example to work off of can make this process more difficult so I’ve cobbled together a basic example of what I’m trying to do:

#include <H5Cpp.h>
#include <vector>
#include <string>

// NOTE: THIS CODE SEGFAULTS SO LONG AS THIS IS TRUE
bool recreateIssue = true;

constexpr uint32_t dataspaceRank = 1;
constexpr uint32_t datapointCount = 10;
constexpr hsize_t dataspaceDims[] = { 10 };

std::vector<std::string> stringData = 
{
    "1",
    "22",
    "333",
    "4444",
    "55555",
    "666666",
    "7777777",
    "88888888",
    "999999999",
    "0000000000"
};

std::string empty = "";

std::vector<uint32_t> uintData = 
{
    1,2,3,4,5,6,7,8,9,0
};

int main()
{
    // Open file
    H5::H5File file("test.hdf", H5F_ACC_TRUNC);

    // Create compound type
    // PLEASE NOTE: I am not using HOFFSET since in my real program I don't know the structure of the data until runtime
    H5::CompType compType( (size_t)100 );

    compType.insertMember("uint1", 0, H5::PredType::NATIVE_UINT32);
    compType.insertMember("str", sizeof(uint32_t), H5::StrType(0, H5T_VARIABLE));
    compType.insertMember("uint2", sizeof(uint32_t) + sizeof(char *), H5::PredType::NATIVE_UINT32);
    compType.setSize(sizeof(uint32_t) + sizeof(char *) + sizeof(uint32_t));

    // Create dataset/space
    H5::DataSpace dataSpace(dataspaceRank, dataspaceDims);

    H5::DataSet dataSet = file.createDataSet("test", compType, dataSpace);

    // Organize data into buffer

    std::string databuffer(compType.getSize() * datapointCount, ' ');

    uint32_t offset = 0;
    for(uint32_t i = 0; i < datapointCount; i++)
    {
        databuffer.replace(offset, sizeof(uint32_t), (char *)&uintData[i], sizeof(uint32_t));
        offset += sizeof(uint32_t);

        if(recreateIssue)
            databuffer.replace(offset, sizeof(char *), stringData[i].c_str(), sizeof(char *));
        else
            databuffer.replace(offset, sizeof(char *), empty.c_str(), sizeof(char *));
        
        offset += sizeof(char *);

        databuffer.replace(offset, sizeof(uint32_t), (char *)&uintData[i], sizeof(uint32_t));
        offset += sizeof(uint32_t);
    }

    // Write buffer to file

    dataSet.write(databuffer, compType);

    file.close();

    return 0;
}

This was the only approach I could find that people said worked online, but it segfaults so long as the strings actually have data in them. I’ve also attached a copy of the backtrace for additional information.

#0  0x00007ffff699ce57 in __strlen_avx2 () from /lib64/libc.so.6
#1  0x00007ffff7a85812 in H5T__conv_vlen () from /lib64/libhdf5.so.103
#2  0x00007ffff7a7a29b in H5T_convert () from /lib64/libhdf5.so.103
#3  0x00007ffff7a840df in H5T__conv_struct_opt () from /lib64/libhdf5.so.103
#4  0x00007ffff7a7a29b in H5T_convert () from /lib64/libhdf5.so.103
#5  0x00007ffff790e931 in H5D__scatgath_write () from /lib64/libhdf5.so.103
#6  0x00007ffff78f6426 in H5D__contig_write () from /lib64/libhdf5.so.103
#7  0x00007ffff790a260 in H5D__write () from /lib64/libhdf5.so.103
#8  0x00007ffff790a9aa in H5Dwrite () from /lib64/libhdf5.so.103
#9  0x00007ffff7621673 in H5::DataSet::write(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, H5::DataType const&, H5::DataSpace const&, H5::DataSpace const&, H5::DSetMemXferPropList const&) const () from /lib64/libhdf5_cpp.so.103
#10 0x0000000000402bc4 in main () at /home/nicholas.desmarais/Documents/hdf_example/src/main.cpp:75
6

Thanks for the example! I’ll try to take a look.