Example on how to append 2D array to an existing dataset in .h5 file


#1

Hello everyone,

I would like to add data in chunks(2d arrays) to the dataset with unlimited size in Fortran.
Is there any way where I can simply open and extend the existing dataset without rewriting the whole data?

As I understood I have to open the file in RDWR mode:
CALL h5fopen_f(filename, H5F_ACC_RDWR_F, file, hdferr)

Access dataset
CALL h5dopen_f (file, dataset, dset, hdferr)

Extend dimensions, where dims are old dimensions and extdims = (/dims(1), dims(2)+5/) - add 5 more lines
CALL h5dset_extent_f(dset, extdims, hdferr)

Then, my question is how to write additional data without rewriting the old data? Can I somehow specify the initial index for writing?
CALL h5dwrite_f(dset, H5T_NATIVE_INTEGER, wdata2, extdims, hdferr)

Best wishes,
Volodymyr


#2

Hi @volodymyr.rodin,

Please find below an example in Fortran using HDFql (not sure how this is done with other APIs) that creates an HDF5 file named test.h5 and, within it, creates a dataset named dset. This dataset is extendible (on its first dimension) and is populated with data coming from an hypothetical function named acquire. The populating is done through an hyperslab selection so that previous written data is not overwritten.

PROGRAM Example
    USE HDFql

    CHARACTER :: variable_number

    INTEGER, DIMENSION(1024) :: values

    INTEGER :: state

    state = hdfql_execute("CREATE AND USE FILE test.h5")

    state = hdfql_execute("CREATE DATASET dset AS INT(0 TO UNLIMITED, 1024)")

    WRITE(variable_number, "(I0)") hdfql_variable_register(values)

    DO WHILE(acquire(values))  ! call hypothetical function that acquires data
        state = hdfql_execute("ALTER DIMENSION dset TO +1")

        state = hdfql_execute("INSERT INTO dset(-1:::) VALUES FROM MEMORY " // variable_number)
    END DO

    state = hdfql_variable_unregister(values)

    state = hdfql_execute("CLOSE FILE")
END PROGRAM

Hope it helps!


#3

Thanks for your reply

Unfortunately, I don’t have this API on a cluster. But I have found a solution with the standard HDF5 features
Extend by row
Extend by column

Best regards,
Volodymyr