HDF5 data are stored in directory but not in files

HSDS
1

Hi,

I built a hsds server and could access (write and read) data into it. But when I checked the data folder, I find there are no files (.h5 or .hdf5) existing, but only directories with defined name. Inside the directories, I don’t see any files either. Nevertheless, I can still write and read data without problems. Could someone instruct me where the data is really stored? This info is important for my application because I’d like to backup these data into another pc.

Thanks,
Peiyun

2

Hi @pshi,

Since you stored data using HSDS there are no HDF5 files any more. It uses a different storage format.

Do you have home and db folders somewhere where HSDS is storing its data? If so, you should really backup every folder and file starting from those two since all of them are important for HSDS to return to the correct state after restoring backed up files.

As for “real data”, belonging to dataset chunks, those are in folders of the form db/<uuid1>/d/<uuid2>/.

-Aleksandar

3

Hi, @ajelenak,

Thanks for your prompt response! Yes, there is one db folder with real data inside. And I can use hsstat to find the mapping from shot to folder names.
One following up question in term of data back up to another pc: if I build one hsds server on the backup pc, and directly copy all files there. Will these files be accessed via the hsds server on the backup pc? Or what need to be copied to make it work?

Best,
Peiyun

4

Beside copying entire content of the db and home folders and their subfolders, you should also copy all HSDS configuration files and update them for any difference between the two computers. Certainly you need to put the new locations of the copied db and home folders.

-Aleksandar

5

Hi, Aleksandar,

Thanks for helping solve this issue!

Peiyun