Virtual datasets from columns in Compound type


This is my first post on this forum. So I hope it is in the correct group.

The thing I try to accomplish is the following.

During a measurment, I get a lot of data per timestamp. This is dynamic, and of different types.

I use a set of packet tables to store this data, one table for a time-sequence of each measured value.
This however is quite slow. If I write the whole set to one packet table, using a Compound datatype, it goes much faster.

Follow-up tools within the company however expect the data in per measured value.

To accomplish this, I was experimenting with dataslabs and virtual datasets. I got this working, more or less. The only issue is that the virtual dataset does not contain a sequence of values of the datatype of the measured item, but contain a sequence of a compound datatype, with only one column, which is of the datatype of the measured item. To say it in c++ terms: I do not get an integer, but a struct containing only one integer.

Can I use Virtual datasets to create a dataset of the type of a single colnum frm a compount type.

This is the code I have made so far.

#define FILE            "D:\\tmp\\sfile_compound.hdf5"
#define DATASET         "DS1"
#define PARTDS			"DS1"
#define DIM0            4
#define RANK			1

#define VDS_FILE            "D:\\tmp\\sfile_compound_virtual.hdf5"
#define VDS_RANK	1
#define VDS_DATASET		"VDS1"

    hid_t compound_type;	
        typedef struct {
            int     serial_no;
            char    *location;
            double  temperature;
            double  pressure;
        } sensor_t;                                 /* Compound type */

        hid_t       file, filetype, strtype, space, dset;
        /* Handles */
        herr_t      status;
        hsize_t     dims[RANK] = { DIM0 };
        sensor_t    wdata[DIM0],                /* Write buffer */

        int         ndims;
        hsize_t     i;

        * Initialize data.
        wdata[0].serial_no = 1153;
        wdata[0].location = "Exterior (static)";
        wdata[0].temperature = 53.23;
        wdata[0].pressure = 24.57;
        wdata[1].serial_no = 1184;
        wdata[1].location = "Intake";
        wdata[1].temperature = 55.12;
        wdata[1].pressure = 22.95;
        wdata[2].serial_no = 1027;
        wdata[2].location = "Intake manifold";
        wdata[2].temperature = 103.55;
        wdata[2].pressure = 31.23;
        wdata[3].serial_no = 1313;
        wdata[3].location = "Exhaust manifold";
        wdata[3].temperature = 1252.89;
        wdata[3].pressure = 84.11;
        * Create a new file using the default properties.
        file = H5Fcreate(FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

        * Create variable-length string datatype.
        strtype = H5Tcopy(H5T_C_S1);
        status = H5Tset_size(strtype, H5T_VARIABLE);

        * Create the compound datatype for memory.
        compound_type = H5Tcreate(H5T_COMPOUND, sizeof(sensor_t));
        status = H5Tinsert(compound_type, "Serial number",
            HOFFSET(sensor_t, serial_no), H5T_NATIVE_INT);
        status = H5Tinsert(compound_type, "Location", HOFFSET(sensor_t, location),
        status = H5Tinsert(compound_type, "Temperature (F)",
            HOFFSET(sensor_t, temperature), H5T_NATIVE_DOUBLE);
        status = H5Tinsert(compound_type, "Pressure (inHg)",
            HOFFSET(sensor_t, pressure), H5T_NATIVE_DOUBLE);

        * Create dataspace.  Setting maximum size to NULL sets the maximum
        * size to be the current size.
        space = H5Screate_simple(1, dims, NULL);

        * Create the dataset and write the compound data to it.
        dset = H5Dcreate(file, DATASET, compound_type, space, H5P_DEFAULT, H5P_DEFAULT,
        status = H5Dwrite(dset, compound_type, H5S_ALL, H5S_ALL, H5P_DEFAULT, wdata);

        * Close and release resources.
        status = H5Tclose(compound_type);
        status = H5Sclose(space);
        status = H5Dclose(dset);
        status = H5Fclose(file);
    // OK, let's try to get a column from a coumpound type
    // First create the file for the VDS
    hid_t vfile = H5Fcreate(VDS_FILE, H5F_ACC_TRUNC, H5P_DEFAULT, H5P_DEFAULT);

    // Now a VDS dataspace, only for the first column
    hsize_t vdsdims[VDS_RANK] = { 0 }; //<
    hsize_t vdsmax_dims[VDS_RANK] = { H5S_UNLIMITED };
    hid_t vspace = H5Screate_simple(VDS_RANK, vdsdims, vdsmax_dims);

    // A source dataspace
    hsize_t dims[COMPOUND_RANK] = { 0 }; //< 
    hsize_t max_dims[COMPOUND_RANK] = { H5S_UNLIMITED };
    hid_t src_space = H5Screate_simple(COMPOUND_RANK, dims, max_dims);

    // Creation property
    hid_t dcpl = H5Pcreate(H5P_DATASET_CREATE);

    // Create  VDS compound type
    hid_t vds_compound_type = H5Tcreate(H5T_COMPOUND, sizeof(H5T_NATIVE_INT));
    herr_t status = H5Tinsert(vds_compound_type, "Serial number",
        0, H5T_NATIVE_INT);

    // Creat mapping 
    const hsize_t src_start[COMPOUND_RANK] = { 0, 0};
    const hsize_t src_count[COMPOUND_RANK] = { H5S_UNLIMITED, 1};
    const hsize_t vds_start[VDS_RANK] = { 0 };
    const hsize_t vds_count[VDS_RANK] = { H5S_UNLIMITED };

    // First get the source dataslab: select what we want from the source file
    status = H5Sselect_hyperslab(src_space, H5S_SELECT_SET, src_start, NULL, src_count, NULL);
    // Now get the VDS dataslab, how we want to store the data in teh target file
    status = H5Sselect_hyperslab(vspace, H5S_SELECT_SET, vds_start, NULL, vds_count, NULL);
    // Connect the two
    status = H5Pset_virtual(dcpl, vspace, FILE, PARTDS, src_space);

    // ???

    // Stap 4: write the virtual dataset definition to the file
    hid_t vdset = H5Dcreate2(vfile, VDS_DATASET, vds_compound_type, vspace, H5P_DEFAULT, dcpl, H5P_DEFAULT);
    status = H5Sclose(vspace);
    status = H5Sclose(src_space);
    status = H5Pclose(dcpl);
    status = H5Dclose(vdset);
    status = H5Fclose(vfile);


No. The current implementation of VDS “virtualizes” only the data-spatial extent of a dataset
and does not “virtualize” the user-visible field dimension of datatypes. Although, currently, you cannot
persist it in this columnar form (as a dataset), you can still do partitial I/O from/to a compound datatype VDS with a single-column in-memory datatype. G.