Hi @djnijeboer,
To continuously add (i.e. stream) data into an HDF5 file without knowing how many rows of data exist a priori, you need to use an extendible dataset.
If you are not bound to a particular library, take a look at HDFql. As an example, your use-case could be solved as follows using this library in C# (we assume that a function named acquire
exists which populates variable values
with a timestamp (UNIX Epoch Time) and an acquired value):
using System.Runtime.InteropServices;
using AS.HDFql;
[StructLayout(LayoutKind.Sequential, Pack = 0)]
struct Data
{
public int timestamp;
public float reading;
}
public class Example
{
public static void Main(string []args)
{
Data []values = new Data[1];
int number;
// create an HDF5 file named 'log.h5'
HDFql.Execute("CREATE FILE log.h5");
// use (i.e. open) HDF5 file 'log.h5'
HDFql.Execute("USE FILE log.h5");
// create a dataset named 'dset' of data type compound composed of two members named 'timestamp' (of data type int containing a UNIX Epoch Time)
// and 'reading' (of data type float containing an acquired value). The dataset starts with 0 rows and can grow (i.e. be extended) in an unlimited fashion
HDFql.Execute("CREATE DATASET dset AS COMPOUND(timestamp AS INT, reading AS FLOAT)(0 TO UNLIMITED)");
// register variable 'values' for subsequent use (by HDFql)
number = HDFql.VariableRegister(values);
// call hypothetical function 'acquire' that populates variable 'values' with a timestamp and an acquired value
while(acquire(values))
{
// alter (i.e. change) dimension of dataset 'dset' to +1 (i.e. add a new row at the end of 'dset')
HDFql.Execute("ALTER DIMENSION dset TO +1");
// insert (i.e. write) data from variable 'values' into the last row of dataset 'dset' (thanks to a point selection)
HDFql.Execute("INSERT INTO dset(-1) VALUES FROM MEMORY " + number);
}
}
}
Hope this helps!