How to shift old dataset along one dimension?

HDF5 Library
1

Hi,
I have a h5 file which has a 3-D matrix (30x500x500 with unlimited length) dataset.
I can extend the dataset to 60x500x500 by reset the size to [60,500,500]. By this way the space is enlarged, so I can write more data immediately after the old data.

However, now I need to insert the new 30x500x500 to the beginning of dataset, not following the old dataset. This means I need to “push/shift” the old data to the extending end, then “overwrite” the new data from the origin.

A way I can think of is:
(1) Enlarge the dataset dimension as needed.
(2) copy-paste (shift) the 500X500 slices one-by-one from the end of old data to end of new container. Then empty space are available at the origin of dataset.
(3) write the new dataset from the origin side.

It seems thatstep (2) takes substantial time. Is there a native way to directly shift dataset along first dimension?

Thanks

3

Hi,

Interesting question… Have you considered using VDS? You can leave the initial dataset intact, create a new dataset with the new data and then create VDS that maps first 30x500x500 slices to the new dataset, and the next 30x500x500 to the old one. Creating VDS is cheap since you will not writing any data but just storing the mapping. VDS dataset will look like you "inserted’ new data.

4

Hi,
The VDS looks very interesting. I just wonder for the old software that has its own file format based on hdf5, if the new VDS feature was used to generate a h5 file, will that old software can recognize that file? In other words, does the source code of software need to be re-write to accomodate the new VDS feature?
Thanks