Hiya,
We have a repository of HDF5 files stored in Amazon S3 buckets, and we'd
like to use h5serv to work with these files.
I tried googling but didn't find any implementations of this.
Has anyone has any luck standing up h5serv in front of S3 storage instead
of the local data directory?
Any chance of this becoming a future enhancement for the h5serv project?
Thanks,
RAY
Hi Ray,
H5serv doesn’t work directly with S3 storage since the HDF5 library is expecting a Posix file system. Using something like s3fuse (https://github.com/s3fs-fuse/s3fs-fuse) may work for read-only access to HDF5 files stored as S3 objects, but I haven’t tried this.
Anyway, as it happens, we’re currently working on a project for NASA to enable HDF Server to use object storage directly. The goal is to enable the HDF api across mutii-PB datasets hosted on S3. We’re about 6 months into the 2 year project, but it’s looking quite promising.
I gave a talk at the summer ESIP meeting regarding this. You can see the slides at: http://www.slideshare.net/HDFEOS/hdf-cloud-services. The first part is a review of HDF Server progress to-date. Slide 22 on talks about the object storage work.
Cheers,
John
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ray Courtney <ray@jawbone.com>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Date: Tuesday, February 14, 2017 at 12:39 PM
To: "hdf-forum@lists.hdfgroup.org" <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] h5serv and S3
Hiya,
We have a repository of HDF5 files stored in Amazon S3 buckets, and we'd like to use h5serv to work with these files.
I tried googling but didn't find any implementations of this.
Has anyone has any luck standing up h5serv in front of S3 storage instead of the local data directory?
Any chance of this becoming a future enhancement for the h5serv project?
Thanks,
RAY
