Hello, I’m using HDFql C++, and I’m having trouble registering a dynamic array
I tried the following code which works perfectly:
customType vect[10];
if (HDFql::execute("SHOW DATASET /PATH/TO/DATASET") == HDFQL_SUCCESS) {
HDFql::variableTransientRegister(&vect);
std::string query = "SELECT FROM /PATH/TO/DATASET(" + strNum + ":::" + strChunk + ") INTO MEMORY 0";
HDFql::execute(query.c_str());
std::cout << vect[0].ID << std::endl; // works perfectly
}
But my problem is since I don’t know the array size beforehand I either have to use a dynamic array or a vector
Problem is, the same code doesn’t work with dynamic array or vector, registering them and then using the query doesn’t work
Example :
customType* vect = new customType[10];
if (HDFql::execute("SHOW DATASET /PATH/TO/DATASET") == HDFQL_SUCCESS) {
HDFql::variableTransientRegister(&vect);
std::string query = "SELECT FROM /PATH/TO/DATASET(" + strNum + ":::" + strChunk + ") INTO MEMORY 0";
std::cout << vect[0].ID << std::endl; // it crashes
I tried the same with vectors , but it also crash
What is the solution to use array with HDFql when you know size only at execution ?
Currently, there are only two ways to solve the use-case you have described in HDFql, namely:
retrieve information about the dataset (using, e.g., HDFql operations SHOW DIMENSION and SHOW DATA TYPE) and allocate enough memory based on this before reading the dataset and populate the variable with it
read the dataset and populate HDFql cursor with it instead of memory (afterwards, one can use methods HDFql::cursor* to traverse the cursor and retrieve data from it)
In the next official release of HDFql, there will be an additional way to solve the same use-case, which is to indicate to HDFql to dynamically allocate memory (i.e. on-the-fly and in function of the characteristics of the dataset to read) through the keyword ALLOCATE. Example:
HDFql::execute("SELECT FROM dset INTO ALLOCATE MEMORY 0");
HDFql will dynamically allocate enough memory to variable assigned to number 0 to successfully store the data read from dataset dset.
Isn’t it what I’m doing with customType* vect = new customType[10]; ?
I wrote it like that but in reality I’m getting the size from a field in a dataset (like int var = dataset.nbId)
and then customType* vect = new customType[var];
I can’t use static array since I don’t know the size during compilation, so I have to allocate memory with new as shown above, but then that doesn’t work, my dynamic array doesn’t get populated with “select from” as shown in the example above. Therefore I don’t know what you mean by “allocate enough memory before reading and populate it”, could you give an example please?
for the cursor, I’ll look into it, I’ve never used it so far so I don’t really know how to select X element from dataset and store it in cursor to then store them in custom type, but I’ll look into documentation for it
Based on the code snippet you have posted, it seems that the issue you are facing has to do with the way member ID of variable vect[0] is accessed (and not HDFqlper se).
By default (i.e. when no redirecting option is specified), the result of reading a dataset is (implicitly) stored in a cursor:
HDFql::execute("SELECT FROM dset");
In case of need to be explicit in the redirecting of the reading of a dataset into a cursor do the following instead (functionally equivalent to the previous code snippet though):
Nono, I think you understood my first messages wrong
When using static array in C++, the code above works perfectly, data are stored perfectly inside the array
When using dynamic array ( with new keyword in C++), the exact same code (just with new because I don’t know the size before compilation) doesn’t work
So it means that HDFql doesn’t accept to register my dynamic array and store value inside it, but accepts to do it with my static array
Vectors doesn’t work either
So my question is if there’s a way of using dynamic array to store variable as I did with static array in my first example
Would you mind to print the result of registering variable vect (by doing std::cout << HDFql::variableTransientRegister(&vect);) and post the result here?
In my second example I forgot to put that I do “HDFql::execute(query.c_str());” also obviously
With customType vect[10]; code works till the end perfectly
with customType* vect = new customType[10] , code crashes when I try to access vect, but even worst
If I just execute the query ( cout << HDFql::execute(query.c_str()) << endl ; prints 0 also btw ), and don’t try to print vect, function is gonna crash at the end when trying to clean stack with the following error message “Stack was corrupted around query variable”
So trying to stock dataset into dynamic array doesn’t return an HDFql error message, but does corrupt the stack
By the way you can try thing yourself, because I tried with just a simple 6 line program (basicly what I copied here), so it’s not caused by previous function call
Thanks for the additional details - would you mind to post the declaration/specification of customType?
As a side note, HDFql method execute accepts a C++ std::string (besides a pointer of type char). Therefore, your code snippet can be simplified as follows (i.e. no need to call c_str()):
I wanna mention that since when I use static array I can access all those variable correctly (codes run from start to end displaying the right result), I think the problem really comes down to how HDFql threats dynamic array vs static array
Thanks for this! One last thing: would you mind to post the HD5 file containing the dataset that you want to read with HDFql or, alternatively, the dump of running the tool h5dump against this file?
With all this information, it should be easy to track the issue down afterwards.
I also tried with putting int instead of int64_t,
If my definition is in cause for that problem, what does HDFql does different with static array so that it would still work with static array?
