Adding data dynamically to a dataset in hdf java

hi all,

I was wondering if¨s possible to add data dynamically to a dataset in hdf
java.

Normally, the dataset is populated with data then added after this. We want
to parse a text file with around 25 million lines, so having all this data
in memory is not so easy. One solution is to create 25 data sets with one
million entries in it. But would be better is we where able to add all the
data inside one dataset. So instead of making 25 different dataset I would
like to append the new data to an existing dataset.

I found this example [1], is this how in can be achieved? First write the
first chunk, then read the data back again and add chunk two and so on?

cheers, Håkon

[1]
http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/java/examples/datasets/H5Ex_D_UnlimitedAdd.java

···

--
Håkon Sagehaug, Scientific Programmer
Parallab, Uni BCCS/Uni Research
Hakon.Sagehaug@uni.no, phone +47 55584125

Yes, you are doing the right thing. You can also use H5Dextend().

You can write chunk by chunk. To write the second chunk, you do
need to read the first chunk. Using H5Sselect_hyperslab() to select
whatever chunk you want to write.

Thanks
--pc

Håkon Sagehaug wrote:

···

hi all,

I was wondering if¨s possible to add data dynamically to a dataset in hdf java.

Normally, the dataset is populated with data then added after this. We want to parse a text file with around 25 million lines, so having all this data in memory is not so easy. One solution is to create 25 data sets with one million entries in it. But would be better is we where able to add all the data inside one dataset. So instead of making 25 different dataset I would like to append the new data to an existing dataset.

I found this example [1], is this how in can be achieved? First write the first chunk, then read the data back again and add chunk two and so on?

cheers, Håkon

[1] http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/java/examples/datasets/H5Ex_D_UnlimitedAdd.java

--
Håkon Sagehaug, Scientific Programmer
Parallab, Uni BCCS/Uni Research
Hakon.Sagehaug@uni.no <mailto:Hakon.Sagehaug@uni.no>, phone +47 55584125
------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org