# Memory allocation using HDFql C++

**URL:** <https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219>\
**Category:** HDFql\
**Created:** [June 22, 2023, 3:12pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219 "2023-06-22T15:12:15Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 22, 2023, 3:12pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/1 "2023-06-22T15:12:15Z")

</div>

Hello, I’m using HDFql C++, and I’m having trouble registering a dynamic array

I tried the following code which works perfectly:

customType vect[10];

```
if (HDFql::execute("SHOW DATASET /PATH/TO/DATASET") == HDFQL_SUCCESS) {

    HDFql::variableTransientRegister(&vect);

    std::string query = "SELECT FROM /PATH/TO/DATASET(" + strNum + ":::" + strChunk + ") INTO MEMORY 0";
    HDFql::execute(query.c_str());
    std::cout << vect[0].ID << std::endl; // works perfectly

```

}

But my problem is since I don’t know the array size beforehand I either have to use a dynamic array or a vector  
Problem is, the same code doesn’t work with dynamic array or vector, registering them and then using the query doesn’t work

Example :

customType\* vect = new customType[10];

```
if (HDFql::execute("SHOW DATASET /PATH/TO/DATASET") == HDFQL_SUCCESS) {

    HDFql::variableTransientRegister(&vect);

    std::string query = "SELECT FROM /PATH/TO/DATASET(" + strNum + ":::" + strChunk + ") INTO MEMORY 0";
    std::cout << vect[0].ID << std::endl; // it crashes

```

I tried the same with vectors , but it also crash

What is the solution to use array with HDFql when you know size only at execution ?

Thanks

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 22, 2023, 8:30pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/2 "2023-06-22T20:30:39Z")

</div>

Hi @angelbillyguyon,

Currently, there are only two ways to solve the use-case you have described in [HDFql](https://www.hdfql.com), namely:

1. retrieve information about the dataset (using, e.g., HDFql operations `SHOW DIMENSION` and `SHOW DATA TYPE`) and allocate enough memory based on this before reading the dataset and populate the variable with it

2. read the dataset and populate HDFql cursor with it instead of memory (afterwards, one can use methods `HDFql::cursor*` to traverse the cursor and retrieve data from it)

In the next official release of HDFql, there will be an additional way to solve the same use-case, which is to indicate to HDFql to dynamically allocate memory (i.e. on-the-fly and in function of the characteristics of the dataset to read) through the keyword `ALLOCATE`. Example:

```auto
HDFql::execute("SELECT FROM dset INTO ALLOCATE MEMORY 0"); 

```

HDFql will dynamically allocate enough memory to variable assigned to number `0` to successfully store the data read from dataset `dset`.

Hope it helps!

---

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 23, 2023, 5:31am UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/4 "2023-06-23T05:31:22Z")

</div>

> [@contact](#):
>
> - retrieve information about the dataset (using, e.g., HDFql operations `SHOW DIMENSION` and `SHOW DATA TYPE`) and allocate enough memory based on this before reading the dataset and populate the variable with it

Isn’t it what I’m doing with customType\* vect = new customType[10]; ?  
I wrote it like that but in reality I’m getting the size from a field in a dataset (like int var = dataset.nbId)  
and then customType\* vect = new customType[var];

I can’t use static array since I don’t know the size during compilation, so I have to allocate memory with new as shown above, but then that doesn’t work, my dynamic array doesn’t get populated with “select from” as shown in the example above. Therefore I don’t know what you mean by “allocate enough memory before reading and populate it”, could you give an example please?

for the cursor, I’ll look into it, I’ve never used it so far so I don’t really know how to select X element from dataset and store it in cursor to then store them in custom type, but I’ll look into documentation for it

Thanks for your response

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 23, 2023, 3:47pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/5 "2023-06-23T15:47:15Z")

</div>

Hi @angelbillyguyon,

Based on the code snippet you have posted, it seems that the issue you are facing has to do with the way member `ID` of variable `vect[0]` is accessed (and not [HDFql](https://www.hdfql.com) _per se_).

By default (i.e. when no redirecting option is specified), the result of reading a dataset is (implicitly) stored in a cursor:

```auto
HDFql::execute("SELECT FROM dset"); 

```

In case of need to be explicit in the redirecting of the reading of a dataset into a cursor do the following instead (functionally equivalent to the previous code snippet though):

```auto
HDFql::execute("SELECT FROM dset INTO CURSOR"); 

```

Hope it helps!

---

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 23, 2023, 5:51pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/6 "2023-06-23T17:51:04Z")

</div>

Nono, I think you understood my first messages wrong  
When using static array in C++, the code above works perfectly, data are stored perfectly inside the array

When using dynamic array ( with new keyword in C++), the exact same code (just with new because I don’t know the size before compilation) doesn’t work

So it means that HDFql doesn’t accept to register my dynamic array and store value inside it, but accepts to do it with my static array

Vectors doesn’t work either

So my question is if there’s a way of using dynamic array to store variable as I did with static array in my first example

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 25, 2023, 7:03pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/7 "2023-06-25T19:03:10Z")

</div>

Hi @angelbillyguyon,

Would you mind to print the result of registering variable `vect` (by doing `std::cout << HDFql::variableTransientRegister(&vect);`) and post the result here?

Thanks!

---

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 26, 2023, 6:51am UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/8 "2023-06-26T06:51:38Z")

</div>

I already did that and it correctly prints 0

In my second example I forgot to put that I do “HDFql::execute(query.c\_str());” also obviously

With customType vect[10]; code works till the end perfectly  
with customType\* vect = new customType[10] , code crashes when I try to access vect, but even worst

If I just execute the query ( cout \<\< HDFql::execute(query.c\_str()) \<\< endl ; prints 0 also btw ), and don’t try to print vect, function is gonna crash at the end when trying to clean stack with the following error message “Stack was corrupted around query variable”

So trying to stock dataset into dynamic array doesn’t return an HDFql error message, but does corrupt the stack  
By the way you can try thing yourself, because I tried with just a simple 6 line program (basicly what I copied here), so it’s not caused by previous function call

Thanks for your help

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 26, 2023, 6:59pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/9 "2023-06-26T18:59:31Z")

</div>

Hi @angelbillyguyon,

Thanks for the additional details - would you mind to post the declaration/specification of `customType`?

As a side note, [HDFql](https://www.hdfql.com) method `execute` accepts a C++ `std::string` (besides a pointer of type `char`). Therefore, your code snippet can be simplified as follows (i.e. no need to call `c_str()`):

```
HDFql::execute(query);

```

---

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 27, 2023, 8:35am UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/10 "2023-06-27T08:35:44Z")

</div>

#pragma once  
#ifndef customType\_H  
#define customType\_H  
#include   
using namespace std;

struct customType {  
int64\_t ID;  
double T;  
double T2;  
int64\_t S;  
};

#endif // customType\_H

I wanna mention that since when I use static array I can access all those variable correctly (codes run from start to end displaying the right result), I think the problem really comes down to how HDFql threats dynamic array vs static array

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 27, 2023, 8:41am UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/11 "2023-06-27T08:41:31Z")

</div>

Hi @angelbillyguyon,

Thanks for this! One last thing: would you mind to post the HD5 file containing the dataset that you want to read with [HDFql](https://www.hdfql.com) or, alternatively, the dump of running the tool `h5dump` against this file?

With all this information, it should be easy to track the issue down afterwards.

---

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 27, 2023, 8:47am UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/12 "2023-06-27T08:47:51Z")

</div>

DATATYPE H5T\_COMPOUND {  
H5T\_STD\_I64LE “ID”;  
H5T\_IEEE\_F64LE “T”;  
H5T\_IEEE\_F64LE “T2”;  
H5T\_STD\_I64LE “S”;  
}

I also tried with putting int instead of int64\_t,  
If my definition is in cause for that problem, what does HDFql does different with static array so that it would still work with static array?

Thanks

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 27, 2023, 11:29am UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/13 "2023-06-27T11:29:05Z")

</div>

Hi @angelbillyguyon,

Would you mind to register variable `vect` as follows (i.e. do not pass the address of the variable given that it’s a pointer):

```auto
HDFql::variableTransientRegister(vect);

```

This change should fix the issue you have been experiencing.

---

<div class="post-metadata">

**Author:** ![angelbillyguyon](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/50afbb/32.png) [@angelbillyguyon](https://forum.hdfgroup.org/u/angelbillyguyon)\
**Post date:** [June 27, 2023, 1:14pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/14 "2023-06-27T13:14:51Z")

</div>

I was sure I tried that… I probably tried it while I had an other error in my code and therefore thought it didn’t do anything.

Well it was the solution, a pretty simple one indeed, sorry for the loss of time and thanks for your help

---

<div class="post-metadata">

**Author:** ![contact](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/contact/32/1067_2.png) [@contact](https://forum.hdfgroup.org/u/contact)\
**Post date:** [June 27, 2023, 3:16pm UTC](https://forum.hdfgroup.org/t/memory-allocation-using-hdfql-c/11219/15 "2023-06-27T15:16:15Z")

</div>

Great to know that it works fine now!