# Creating a compound datatype with no predefined struct

**URL:** <https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098>\
**Category:** HDF5 Library\
**Created:** [March 12, 2025, 12:07pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098 "2025-03-12T12:07:17Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![gsabhishek1ags](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/g/ad7895/32.png) [@gsabhishek1ags](https://forum.hdfgroup.org/u/gsabhishek1ags)\
**Post date:** [March 12, 2025, 12:07pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/1 "2025-03-12T12:07:17Z")

</div>

I am trying to write a set of fields to a H5Table, not knowing what the fields are apriori.  
Calculating the offsets here has been an issue, since HOFFSET/offsetof cannot be used due to absence of a predefined struct. I am calculating the offsets assuming contiguous allocation and no padding which may be erroneous.  
Is there an alternative way to handle such a situation.

```auto
hid_t out_stream = H5Fcreate("vlen",H5F_ACC_TRUNC,H5P_DEFAULT,H5P_DEFAULT);
//vlen_string = H5Tcopy(H5T_C_S1);
//H5Tset_size(vlen_string,H5T_VARIABLE);
vlen_string = H5Tvlen_create(H5T_C_S1);
std::vector<std::string> field_names = {"FIELD_1","FIELD_2","FIELD_3"};
std::vector<hid_t> field_types = {getH5Type(CVGTYPE::INT),getH5Type(CVGTYPE::COMMENT),getH5Type(CVGTYPE::DOUBLE)};
std::vector<hsize_t> field_offsets = {0,0,0};
//Create Compound Datatype
hsize_t tot_size = 0;
hsize_t offset = 0;
for(int i=0;i<field_names.size();i++){
    hsize_t type_size = H5Tget_size(field_types[i]);
    field_offsets[i]=offset;
    tot_size += type_size;
    offset += type_size;
}hid_t compound_dtype = H5Tcreate(H5T_COMPOUND,tot_size);
for(int i=0;i<field_names.size();i++){
    H5Tinsert(compound_dtype,field_names[i].c_str(),field_offsets[i],field_types[i]);
}
//Create Table
int nrecords = 0;
int chunk_size = 0;
hsize_t compound_dtype_size = sizeof(compound_dtype);
const char* field_names_c[field_names.size()];
for (size_t i = 0; i < field_names.size(); ++i) {
   field_names_c[i] = field_names[i].c_str();
}
H5TBmake_table("some_table",out_stream,"some_dataset_name",hsize_t(field_names.size()),nrecords,compound_dtype_size,field_names_c,field_offsets.data(),field_types.data(),chunk_size,NULL,0,NULL);

```

This throws the following error

```auto
HDF5-DIAG: Error detected in HDF5 (1.14.5):
  #000: /home/gsabhishek/Downloads/hdf5-1.14.5/src/H5Tcompound.c line 340 in H5Tinsert(): unable to insert member
    major: Datatype
    minor: Unable to insert object
  #001: /home/gsabhishek/Downloads/hdf5-1.14.5/src/H5Tcompound.c line 421 in H5T__insert(): member extends past end of compound type
    major: Datatype
    minor: Unable to insert object

```

Similar to [this post](https://forum.hdfgroup.org/t/reading-attributes-with-compound-datatypes-without-a-priori-creating-structs-using-c/11509), but I cannot use H5QL as of now.

---

<div class="post-metadata">

**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\
**Post date:** [March 12, 2025, 6:47pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/2 "2025-03-12T18:47:26Z")

</div>

The error message is clear: `H5T__insert(): member extends past end of compound type`. How about you check the invariant `field_offsets[i] + H5Tget_size(field_types[i]) <= tot_size` in the `H5Tinsert` loop?

G.

---

<div class="post-metadata">

**Author:** ![gsabhishek1ags](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/g/ad7895/32.png) [@gsabhishek1ags](https://forum.hdfgroup.org/u/gsabhishek1ags)\
**Post date:** [March 13, 2025, 4:21am UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/3 "2025-03-13T04:21:39Z")

</div>

Thanks.  
I checked the insert size

```auto
std::cout<<i<<"th Element Inserted :"<<field_offsets[i]+H5Tget_size(field_types[i])<<" Tot_size : "<<tot_size<<std::endl;
0 Element Inserted :4 Tot_size : 28
1 Element Inserted :20 Tot_size : 28
2 Element Inserted :28 Tot_size : 28

```

H5Tinsert does not throw an error but `H5TBmake_table` is the one throwing the error,

---

<div class="post-metadata">

**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\
**Post date:** [March 13, 2025, 11:14am UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/4 "2025-03-13T11:14:08Z")

</div>

The problematic line is:  
`hsize_t compound_dtype_size = sizeof(compound_dtype);`

[The documentation](https://support.hdfgroup.org/documentation/hdf5/latest/group___h5_t_b.html#title14) states:

> [in] type\_size The size in bytes of the structure associated with the table; This value is obtained with `sizeof()`.

Notice that it says “… the structure associated with the table,” which you said you don’t have.

Clearly, `sizeof(compound_dtype) == sizeof(hsize_t) == 8 < 28`, which is the problem.

`sizeof()` can work only with a predefined struct. The documentation should be clearer on this.

What would work is `hsize_t compound_dtype_size = H5Tget_size(compound_dtype);`

OK?

G.

---

<div class="post-metadata">

**Author:** ![gsabhishek1ags](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/g/ad7895/32.png) [@gsabhishek1ags](https://forum.hdfgroup.org/u/gsabhishek1ags)\
**Post date:** [March 17, 2025, 11:22am UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/5 "2025-03-17T11:22:32Z")

</div>

@gheber  
Thanks. This fixed the issue.

A follow up issue I have is regarding the creation of a data buffer we pass to `H5TBmake_table`

I am trying to create a void pointer for the data. Since we have a variable length string, the size of the void pointer may not be equal to the `compound_dtype_size` and the offsets calculated during the creation of `compound_dtype` would also not be valid anymore.

So here I tried to calculate the new offsets and field sizes. The code runs fine, but when I open the table with hdfview I get an error and the fields are not populated.

```c++
std::vector<int> data1 = {1,2,4};
std::vector<std::string> data2 = {"do","do re","do re mi"};
std::vector<double> data3 = {1.,4.,16.};
for(int n=0;n<3;n++){
  hsize_t start = n;
  void *data = ::operator new( sizeof(int)+data2[n].size()+sizeof(double));
  memcpy(data, &data1[n], sizeof(int));
  memcpy((char *)data + sizeof(int), data2[n].c_str(), data2[n].size());
  memcpy((char *)data + sizeof(int) + data2[n].size(), &data3[n], sizeof(double));
  std::vector<size_t> new_field_sizes={sizeof(int),data2[n].size(),sizeof(double)};
  std::vector<size_t> new_field_offsets={0,new_field_sizes[0],new_field_sizes[0]+new_field_sizes[1]};
 H5TBinsert_record(out_stream,"some_dataset_name",start,nrecords,compound_dtype,new_field_offsets.data(),new_field_sizes.data(),data);
  ::operator delete(data);
}

```

 ![Screenshot from 2025-03-17 16-51-52](https://dbkt.hdfgroup.org/original/2X/0/00e74831416d911a8dc433101a8a3d8bb30b3f5e.png)

---

<div class="post-metadata">

**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\
**Post date:** [March 18, 2025, 12:09pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/6 "2025-03-18T12:09:19Z")

</div>

What’s the datatype of `FIELD_2`?

---

<div class="post-metadata">

**Author:** ![gsabhishek1ags](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/g/ad7895/32.png) [@gsabhishek1ags](https://forum.hdfgroup.org/u/gsabhishek1ags)\
**Post date:** [March 18, 2025, 5:48pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/7 "2025-03-18T17:48:29Z")

</div>

It’s a variable length string.

I created the type as  
hid\_t vlen\_string = H5Tvlen\_create(H5T\_C\_S1);

---

<div class="post-metadata">

**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\
**Post date:** [March 19, 2025, 11:45am UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/8 "2025-03-19T11:45:08Z")

</div>

> [@gsabhishek1ags](#):
>
> ```auto
> memcpy((char *)data + sizeof(int), data2[n].c_str(), data2[n].size());
> 
> ```

I recommend you read the [section on string handling in the HDF5 user guide](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_t__u_g.html#subsubsec_datatype_other_strings) and look at [this example](https://github.com/HDFGroup/hdf5/blob/hdf5_1_14_6/HDF5Examples/C/H5T/h5ex_t_vlstring.c). The gist is that the API for variable-length strings works with pointers. In a structure/compound datatype, the corresponding field of the nominal structure is a pointer and not an array of characters. In your snippet, the field would potentially have a different size, `data2[n].size()`, for each string, which is not how structures work. OK?

G.

---

<div class="post-metadata">

**Author:** ![gsabhishek1ags](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/g/ad7895/32.png) [@gsabhishek1ags](https://forum.hdfgroup.org/u/gsabhishek1ags)\
**Post date:** [March 20, 2025, 9:23am UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/10 "2025-03-20T09:23:40Z")

</div>

Thanks.  
I got most of it working, but I seem to be missing something here and I am still unable to write the strings.

```auto
hid_t vlen_string;

hid_t getH5Type(const CVGTYPE &type){
    if(type==CVGTYPE::INT) return H5T_NATIVE_INT;
    else if(type==CVGTYPE::DOUBLE) return H5T_NATIVE_DOUBLE;
    else if(type==CVGTYPE::COMMENT) return vlen_string;
    else throw std::runtime_error("Invalid CVGType");
}

hid_t getH5TypeOffset(const CVGTYPE &type){
    if(type==CVGTYPE::INT) return H5Tget_size(H5T_NATIVE_INT);
    else if(type==CVGTYPE::DOUBLE) return H5Tget_size(H5T_NATIVE_DOUBLE);
    else if(type==CVGTYPE::COMMENT) return sizeof(hvl_t);
    else throw std::runtime_error("Invalid CVGType");
}

void test_vlen_compound_dtype(){
    hid_t out_stream = H5Fcreate("vlen_table.h5",H5F_ACC_TRUNC,H5P_DEFAULT,H5P_DEFAULT);
    vlen_string = H5Tvlen_create(H5T_C_S1);
    std::vector<std::string> field_names = {"FIELD_1","FIELD_2","FIELD_3"};
    std::vector<hid_t> field_types = {getH5Type(CVGTYPE::INT),getH5Type(CVGTYPE::COMMENT),getH5Type(CVGTYPE::DOUBLE)};
    std::vector<size_t> field_sizes = {0,0,0};
    std::vector<size_t> field_offsets = {0,0,0};
    //Create Compound Datatype
    hsize_t tot_size = 0;
    hsize_t offset = 0;
    for(int i=0;i<field_names.size();i++){
        hsize_t type_size = H5Tget_size(field_types[i]);
        field_offsets[i]=offset;
        field_sizes[i] = type_size;
        tot_size += type_size;
        offset += type_size;
    }
    hid_t compound_dtype = H5Tcreate(H5T_COMPOUND,tot_size);
    for(size_t i=0;i<field_names.size();i++){
        H5Tinsert(compound_dtype,field_names[i].c_str(),field_offsets[i],field_types[i]);
    }
    //Create Table
    hsize_t nfields = field_names.size();
    hsize_t nrecords = 0;
    hsize_t chunk_size = 1;
    const char* field_names_c[field_names.size()];
    for (size_t i = 0; i < field_names.size(); ++i) {
        field_names_c[i] = field_names[i].c_str();
    }
    H5TBmake_table("some_table",out_stream,"some_dataset_name",nfields,nrecords,tot_size,field_names_c,field_offsets.data(),field_types.data(),chunk_size,NULL,0,NULL);

    std::vector<int> data1 = {1,2,4};
    std::vector<std::string> data2 = {"do","do re","do re mi"};
    std::vector<double> data3 = {1.,4.,16.};
    for(int n=0;n<data1.size();n++){
        void* buffer = ::operator new(tot_size);
        memcpy((char*)buffer + field_offsets[0], &data1[n], field_sizes[0]);
        hvl_t vlen_data;
        vlen_data.len = data2[n].length();
        vlen_data.p = new char[vlen_data.len];
        std::strcpy(static_cast<char*>(vlen_data.p), data2[n].c_str());
        memcpy((char*)buffer + field_offsets[1], &vlen_data, field_sizes[1]);
        memcpy((char*)buffer + field_offsets[2], &data3[n], field_sizes[2]);
        std::cout<<static_cast<char*>(reinterpret_cast<hvl_t*>((char*)buffer + field_offsets[1])->p)<<std::endl;
        hsize_t start = n;
        hsize_t nrecs_to_insert = 1;
        H5TBappend_records(out_stream,"some_dataset_name",nrecs_to_insert,tot_size,field_offsets.data(),field_sizes.data(),buffer);
        delete[] static_cast<char*>(vlen_data.p);
        ::operator delete(buffer);
    }
    H5Tclose(compound_dtype);
    H5Fclose(out_stream);
    return
}

```

When I print out from the buffer using

```auto
std::cout<<static_cast<char*>(reinterpret_cast<hvl_t*>((char*)buffer + field_offsets[1])->p)<<std::endl;

```

I am able to output the strings.  
But in the hdfview, I see _Error_

 ![Screenshot from 2025-03-20 14-24-17](https://dbkt.hdfgroup.org/original/2X/3/35bf94b77f4a4b682a7f7cf0bbb4774fe52bb0c3.png)

---

<div class="post-metadata">

**Author:** ![gheber](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/gheber/32/1919_2.png) [@gheber](https://forum.hdfgroup.org/u/gheber)\
**Post date:** [March 20, 2025, 12:01pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/11 "2025-03-20T12:01:21Z")

</div>

OK, I think you overshot a little: variable-length strings can be seen as special cases of variable-length sequences. However, the HDF5 library API treats them differently. Compare [this vlen of integers example](https://github.com/HDFGroup/hdf5/blob/hdf5_1_14_6/HDF5Examples/C/H5T/h5ex_t_vlen.c) with the [vlen string example](https://github.com/HDFGroup/hdf5/blob/hdf5_1_14_6/HDF5Examples/C/H5T/h5ex_t_vlstring.c). The main difference is that in the former example, you deal with `hvl_t,` whereas it is `char*` in the latter. OK?

G.

---

<div class="post-metadata">

**Author:** ![gsabhishek1ags](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/g/ad7895/32.png) [@gsabhishek1ags](https://forum.hdfgroup.org/u/gsabhishek1ags)\
**Post date:** [March 20, 2025, 5:41pm UTC](https://forum.hdfgroup.org/t/creating-a-compound-datatype-with-no-predefined-struct/13098/12 "2025-03-20T17:41:06Z")

</div>

Thanks.  
This finally worked!! Thanks a lot.  
Here is the final code

```auto
void test_vlen_compound_dtype(){
    hid_t out_stream = H5Fcreate("vlen_table.h5",H5F_ACC_TRUNC,H5P_DEFAULT,H5P_DEFAULT);
    hid_t vlen_string = H5Tcopy(H5T_C_S1);
    H5Tset_size(vlen_string,H5T_VARIABLE);
    std::vector<std::string> field_names = {"FIELD_1","FIELD_2","FIELD_3"};
    std::vector<hid_t> field_types = {H5T_NATIVE_INT,vlen_string,H5T_NATIVE_DOUBLE};
    std::vector<size_t> field_sizes = {H5Tget_size(H5T_NATIVE_INT),sizeof(char*),H5Tget_size(H5T_NATIVE_DOUBLE)};
    std::vector<size_t> field_offsets = {0,0,0};
    //Create Compound Datatype
    hsize_t tot_size = 0;
    hsize_t offset = 0;
    for(int i=0;i<field_names.size();i++){
        hsize_t type_size = field_sizes[i];
        field_offsets[i]=offset;
        tot_size += type_size;
        offset += type_size;
    }
    hid_t compound_dtype = H5Tcreate(H5T_COMPOUND,tot_size);
    for(size_t i=0;i<field_names.size();i++){
        H5Tinsert(compound_dtype,field_names[i].c_str(),field_offsets[i],field_types[i]);
    }
    //Create Table
    hsize_t nfields = field_names.size();
    hsize_t nrecords = 0;
    hsize_t chunk_size = 1;
    const char* field_names_c[field_names.size()];
    for (size_t i = 0; i < field_names.size(); ++i) {
        field_names_c[i] = field_names[i].c_str();
    }
    H5TBmake_table("some_table",out_stream,"some_dataset_name",nfields,nrecords,tot_size,field_names_c,field_offsets.data(),field_types.data(),chunk_size,NULL,0,NULL);
    std::vector<int> data1 = {1,2,4};
    std::vector<std::string> data2 = {"do","do re","do re mi"};
    std::vector<double> data3 = {1.,4.,16.};
    void* buffer = ::operator new(tot_size);
    for(int n=0;n<data1.size();n++){
        memcpy((char*)buffer + field_offsets[0], &data1[n], field_sizes[0]);
        char* vlen_data = new char[data2[n].length()];
        std::strcpy(vlen_data, data2[n].c_str());
        //vlen_data[data2[n].length()] = '\0';
        memcpy((char*)buffer + field_offsets[1], &vlen_data, field_sizes[1]);
        memcpy((char*)buffer + field_offsets[2], &data3[n], field_sizes[2]);
        hsize_t start = n;
        hsize_t nrecs_to_insert = 1;
        H5TBappend_records(out_stream,"some_dataset_name",nrecs_to_insert,tot_size,field_offsets.data(),field_sizes.data(),buffer);
        delete[] vlen_data;
    }::operator delete(buffer);
    H5Tclose(compound_dtype);
    H5Tclose(vlen_string);
    H5Fclose(out_stream);
    return;
}

```

 ![Screenshot from 2025-03-20 23-11-29](https://dbkt.hdfgroup.org/original/2X/b/ba51962ced6af8efb481dd5dd692b16d990cc307.png)