Memory allocation using HDFql C++

Hello, I’m using HDFql C++, and I’m having trouble registering a dynamic array

I tried the following code which works perfectly:

customType vect[10];



    std::string query = "SELECT FROM /PATH/TO/DATASET(" + strNum + ":::" + strChunk + ") INTO MEMORY 0";
    std::cout << vect[0].ID << std::endl; // works perfectly


But my problem is since I don’t know the array size beforehand I either have to use a dynamic array or a vector
Problem is, the same code doesn’t work with dynamic array or vector, registering them and then using the query doesn’t work

Example :

customType* vect = new customType[10];



    std::string query = "SELECT FROM /PATH/TO/DATASET(" + strNum + ":::" + strChunk + ") INTO MEMORY 0";
    std::cout << vect[0].ID << std::endl; // it crashes

I tried the same with vectors , but it also crash

What is the solution to use array with HDFql when you know size only at execution ?


Hi @angelbillyguyon,

Currently, there are only two ways to solve the use-case you have described in HDFql, namely:

  1. retrieve information about the dataset (using, e.g., HDFql operations SHOW DIMENSION and SHOW DATA TYPE) and allocate enough memory based on this before reading the dataset and populate the variable with it

  2. read the dataset and populate HDFql cursor with it instead of memory (afterwards, one can use methods HDFql::cursor* to traverse the cursor and retrieve data from it)

In the next official release of HDFql, there will be an additional way to solve the same use-case, which is to indicate to HDFql to dynamically allocate memory (i.e. on-the-fly and in function of the characteristics of the dataset to read) through the keyword ALLOCATE. Example:


HDFql will dynamically allocate enough memory to variable assigned to number 0 to successfully store the data read from dataset dset.

Hope it helps!

Isn’t it what I’m doing with customType* vect = new customType[10]; ?
I wrote it like that but in reality I’m getting the size from a field in a dataset (like int var = dataset.nbId)
and then customType* vect = new customType[var];

I can’t use static array since I don’t know the size during compilation, so I have to allocate memory with new as shown above, but then that doesn’t work, my dynamic array doesn’t get populated with “select from” as shown in the example above. Therefore I don’t know what you mean by “allocate enough memory before reading and populate it”, could you give an example please?

for the cursor, I’ll look into it, I’ve never used it so far so I don’t really know how to select X element from dataset and store it in cursor to then store them in custom type, but I’ll look into documentation for it

Thanks for your response

Hi @angelbillyguyon,

Based on the code snippet you have posted, it seems that the issue you are facing has to do with the way member ID of variable vect[0] is accessed (and not HDFql per se).

By default (i.e. when no redirecting option is specified), the result of reading a dataset is (implicitly) stored in a cursor:

HDFql::execute("SELECT FROM dset"); 

In case of need to be explicit in the redirecting of the reading of a dataset into a cursor do the following instead (functionally equivalent to the previous code snippet though):

HDFql::execute("SELECT FROM dset INTO CURSOR"); 

Hope it helps!

Nono, I think you understood my first messages wrong
When using static array in C++, the code above works perfectly, data are stored perfectly inside the array

When using dynamic array ( with new keyword in C++), the exact same code (just with new because I don’t know the size before compilation) doesn’t work

So it means that HDFql doesn’t accept to register my dynamic array and store value inside it, but accepts to do it with my static array

Vectors doesn’t work either

So my question is if there’s a way of using dynamic array to store variable as I did with static array in my first example

Hi @angelbillyguyon,

Would you mind to print the result of registering variable vect (by doing std::cout << HDFql::variableTransientRegister(&vect);) and post the result here?


I already did that and it correctly prints 0

In my second example I forgot to put that I do “HDFql::execute(query.c_str());” also obviously

With customType vect[10]; code works till the end perfectly
with customType* vect = new customType[10] , code crashes when I try to access vect, but even worst

If I just execute the query ( cout << HDFql::execute(query.c_str()) << endl ; prints 0 also btw ), and don’t try to print vect, function is gonna crash at the end when trying to clean stack with the following error message “Stack was corrupted around query variable”

So trying to stock dataset into dynamic array doesn’t return an HDFql error message, but does corrupt the stack
By the way you can try thing yourself, because I tried with just a simple 6 line program (basicly what I copied here), so it’s not caused by previous function call

Thanks for your help

Hi @angelbillyguyon,

Thanks for the additional details - would you mind to post the declaration/specification of customType?

As a side note, HDFql method execute accepts a C++ std::string (besides a pointer of type char). Therefore, your code snippet can be simplified as follows (i.e. no need to call c_str()):


#pragma once
#ifndef customType_H
#define customType_H
using namespace std;

struct customType {
int64_t ID;
double T;
double T2;
int64_t S;

#endif // customType_H

I wanna mention that since when I use static array I can access all those variable correctly (codes run from start to end displaying the right result), I think the problem really comes down to how HDFql threats dynamic array vs static array

Hi @angelbillyguyon,

Thanks for this! One last thing: would you mind to post the HD5 file containing the dataset that you want to read with HDFql or, alternatively, the dump of running the tool h5dump against this file?

With all this information, it should be easy to track the issue down afterwards.

H5T_IEEE_F64LE “T2”;
H5T_STD_I64LE “S”;

I also tried with putting int instead of int64_t,
If my definition is in cause for that problem, what does HDFql does different with static array so that it would still work with static array?


Hi @angelbillyguyon,

Would you mind to register variable vect as follows (i.e. do not pass the address of the variable given that it’s a pointer):


This change should fix the issue you have been experiencing.

I was sure I tried that… I probably tried it while I had an other error in my code and therefore thought it didn’t do anything.

Well it was the solution, a pretty simple one indeed, sorry for the loss of time and thanks for your help


Great to know that it works fine now!