Hi, I use the following code to read a distance matrix in Java:
String openFileCommand = String.format("USE FILE \"%s\"", distanceMatrixFilepath);
HDFql.execute(openFileCommand);
HDFql.execute("SHOW DIMENSION " + datasetName);
HDFql.cursorFirst();
int size = HDFql.cursorGetInt();
int[][] data = new int[size][size];
int variableId = HDFql.variableRegister(data);
HDFql.execute("SELECT FROM " + datasetName + " INTO MEMORY " + variableId);
HDFql.execute("CLOSE FILE");
However, it was unsuccessful (the array was not filled with values from the dataset). I also tested the return value of the execute command, and it is zero. What am I doing wrong?
Hi @david.fido.fiedler,
According to section 6.7.16 of HDFql reference manual, the operation SHOW DIMENSION returns data of type BIGINT (i.e. a 64 bit integer). Therefore, change the following line in your code:
int size = HDFql.cursorGetInt();
to:
long size = HDFql.cursorGetBigint();
Hope it helps!
Well, it does not help. It is no surprise to me, actually, cause the size and the zero-initialization of the array go fine, even with HDFql.cursorGetInt();. The problem lies within the HDFql.execute("SELECT FROM " + datasetName + " INTO MEMORY " + variableId); command, which appears to have no effect.
Hi @david.fido.fiedler,
Would you mind checking if the dataset is of data type INT? Otherwise, if possible, please upload the HDF5 file in question to help better understand/debug the issue you are facing.
The data type is unsigned int. All details from HDFView:
Thanks for posting a screenshot of the dataset, @david.fido.fiedler!
Would it be possible to have access to the HDF5 file containing the dataset? AFAWCS, your code should be able to read it successfully.
Hi, when preparing test data for you, I realized that the dataset size is problematic. If I create a smaller dataset, it works fine. I haven’t had time to find the precise size from which it fails, but it may be 8GB. Here is the test file (for which it fails). Note that I set up -Xmx30g, so there should be no problem with the Java heap size.
1 Like
Hi @david.fido.fiedler,
Thanks for the additional information and the test file. We can confirm that there is an issue with the HDFql Java wrapper when reading/writing huge datasets into/from a variable.
We can provide a fix for this issue. Would you mind confirming which platform (Windows, Linux or macOS) are you using HDFql?
1 Like
I’m using Windows. Let me know if you need any additional info.
Hi @david.fido.fiedler,
Thanks for the info!
We will prepare an updated version of HDFql, addressing the issue described in this post. Once it’s ready, we will let you know so you can test it out.
