# File storage on AWS S3 **URL:** \ **Category:** HSDS\ **Created:** [July 9, 2021, 3:58pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715 "2021-07-09T15:58:47Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 9, 2021, 3:58pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/1 "2021-07-09T15:58:47Z")
Hi, I successfully deployed HSDS on my local machine and connected it with an S3 bucket. I tested reading/writing using h5pyd and both worked. However, one thing confused me is that on my S3 dashboard, I can only see folders (e.g. /home/admin/test.h5) and json files, without data files. Since HSDS stores each chunk in one object so I think I should see as many data files as the number of chunks. This is the first time I use S3 so I’m not sure if there is anything wrong here. Best, Ruochen ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [July 9, 2021, 9:25pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/2 "2021-07-09T21:25:30Z")
Hey, Do you see a folder in your bucket named “db”? All the data files will be there. In the HSDS schema, folders and domain names are stored by their path (e.g. /shared/mydata.h5 will be stored in s3://bucketname/shared/mydata.h5/domain.json). Inside domain.json there will be a key for the root group id which points to an object under the db/ path. The rational for this approach is that it makes it easy to move or rename domain names without having to move each data object. If you are curious, this doc: [https://github.com/HDFGroup/hsds/blob/master/docs/design/obj\_store\_schema/obj\_store\_schema\_v2.md](https://github.com/HDFGroup/hsds/blob/master/docs/design/obj_store_schema/obj_store_schema_v2.md), describes the storage model in some detail. BTW, if you run: hsinfo \, you’ll get information about the number of json objects, data objects, storage size, etc. for that domain. E.g.: ``` $ hsinfo /shared/bioconductor/tenx_full.h5 domain: /shared/bioconductor/tenx_full.h5 owner: admin id: g-eed60fdd-3e56eab3-665e-8755b6-de623b last modified: 2021-06-19 22:22:58 last scan: 2021-06-19 22:19:22 md5 sum: 4fafc30a05df174ca5cc8f05e4c6e659 total_size: 6112211528 allocated_bytes: 6112210230 metadata_bytes: 875 num objects: 2 num chunks: 105273 ``` ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 13, 2021, 12:51am UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/3 "2021-07-13T00:51:58Z")
Yeah I see the db folder and the data. Thanks a lot for the help! BTW, curious does HSDS support multiple datanodes with different configurations, e.g. one using S3 and others using POSIX. Best, Ruochen ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [July 13, 2021, 11:42pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/4 "2021-07-13T23:42:14Z")
No, each DN node in a deployment needs to have the same configuration. There’s nothing to stop you from having two different HSDS deployments on the same machine though. You’d have one endpoint that would server S3 data and another for Posix. Similarly, you can have two HSDS deployments on a Kuberentes cluster. As long as the deployments are to different namespaces, the HSDS pods will just talk to pods in their own deployment. ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 15, 2021, 7:44pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/5 "2021-07-15T19:44:14Z")
I see. Thanks a lot! ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 21, 2021, 7:11pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/6 "2021-07-21T19:11:54Z")
Hi, I tried to deploy two different HSDS deployments by docker-compose two files separately. I changed the project name (both COMPOSE\_PROJECT\_NAME and container\_name in yml file) to guarantee two deployments have different names. However, the service failed to start (503) after I changed those names, although all containers were created successfully. Is there anything wrong here? ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [July 24, 2021, 6:30pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/7 "2021-07-24T18:30:31Z")
Looks like the compose yml is using some hard-coded container names. I’ve updated the compose files in github and made a code fix. Please try it out and let me know if this resolves the issue. If you are using the runall.sh script, set the COMPOSE\_PROJECT\_NAME env var to the desired value (otherwise will default to “hsds”). You’ll also need set SN\_PORT so the public ports don’t clash. Grab the latest code from master, or pull the image from dockehub: hdfgroup/hsds:v0.7.0beta4 ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 27, 2021, 11:10pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/10 "2021-07-27T23:10:31Z")
It works perfectly now. Thank you! ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [July 28, 2021, 4:15pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/11 "2021-07-28T16:15:27Z")
Awesome! Glad to hear it. ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 28, 2021, 8:37pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/12 "2021-07-28T20:37:00Z")
Hi John, I just met the other problem: after starting two services, I tried `hsinfo` and `hstouch` to create folders. But it only works for one service. For the other one, it keeps returning `Error: [Errno 400] Invalid domain name`. I used `hsconfigure` to change endpoint before doing this and the connection was OK. For these two services, I changed all the ports (head, sn, dn, rangegate) to be different in yml files. Is there anything I did wrong here? ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [July 28, 2021, 10:48pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/13 "2021-07-28T22:48:40Z")
It also doesn’t work for AWS, even if it is the first service I started. ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [July 30, 2021, 7:37pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/14 "2021-07-30T19:37:11Z")
Hey, You shouldn’t need to mess with the head port, etc. It’s only if the port is exposed on the host that’s there’s a potential for conflict. If you look at the port lines in docker-compose, e.g. for the posix version: [https://github.com/HDFGroup/hsds/blob/master/admin/docker/docker-compose.posix.yml](https://github.com/HDFGroup/hsds/blob/master/admin/docker/docker-compose.posix.yml), you should see only the SN\_PORT has both external and internal mappings. (actually I goofed in my last update and forgot to remove the external port for the rangeget proxy. I’ve fixed this now.) In general, if the COMPOSE\_PROJECT\_NAME is different, two containers can have the same internal port, but they’ll be on different internal networks, so shouldn’t conflict. I setup two projects one using AWS the other using posix. Here’s what my docker ps looks like: ``` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 66e0e9140ef4 hdfgroup/hsds "/bin/bash -c 'sourc…" 12 minutes ago Up 12 minutes 5100-5999/tcp, 0.0.0.0:32777->6101/tcp posix_dn_1 5f2bc6a08d5b hdfgroup/hsds "/bin/bash -c 'sourc…" 13 minutes ago Up 12 minutes 5100-5999/tcp, 0.0.0.0:32776->6900/tcp posix_rangeget_1 74a28db22b8e hdfgroup/hsds "/bin/bash -c 'sourc…" 13 minutes ago Up 12 minutes 5100-5999/tcp, 0.0.0.0:8080->8080/tcp posix_sn_1 89782f803c1f hdfgroup/hsds "/bin/bash -c 'sourc…" 13 minutes ago Up 12 minutes 5101-5999/tcp, 0.0.0.0:32775->5100/tcp posix_head_1 9a57450213f5 hdfgroup/hsds "/bin/bash -c 'sourc…" 36 minutes ago Up 36 minutes 5100-5999/tcp, 0.0.0.0:32769->6101/tcp aws_dn_1 6871638c30ba hdfgroup/hsds "/bin/bash -c 'sourc…" 36 minutes ago Up 36 minutes 5100/tcp, 5102-5999/tcp, 0.0.0.0:5101->5101/tcp aws_sn_1 2d07d7c617b2 hdfgroup/hsds "/bin/bash -c 'sourc…" 36 minutes ago Up 36 minutes 5100-5999/tcp, 0.0.0.0:6900->6900/tcp aws_rangeget_1 6525b17c0f80 hdfgroup/hsds "/bin/bash -c 'sourc…" 36 minutes ago Up 36 minutes 5101-5999/tcp, 0.0.0.0:32768->5100/tcp aws_head_1 ``` By setting HS\_ENDPOINT to [http://localhost:5101](http://localhost:5101) or [http://localhost:8080](http://localhost:8080) I can read/write to AWS S3 or local posix respectively. Hope that helps! ---
**Author:** ![ruochenj123](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/r/67e7ee/32.png) [@ruochenj123](https://forum.hdfgroup.org/u/ruochenj123)\ **Post date:** [August 2, 2021, 6:11pm UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/15 "2021-08-02T18:11:59Z")
I still can’t run `hsinfo` or `hstouch` successfully even on a single AWS project. It still returns Error: [Errno 400] Invalid domain name. My docker ps looks like the following: ```auto CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 40cd68ff1173 hdfgroup/hsds "/bin/bash -c 'sourc…" 46 seconds ago Up 45 seconds 5100-5999/tcp, 0.0.0.0:49170->6101/tcp, :::49170->6101/tcp hsds_dn_1 8ab41e338d9e hdfgroup/hsds "/bin/bash -c 'sourc…" 48 seconds ago Up 47 seconds 5100/tcp, 5102-5999/tcp, 0.0.0.0:5101->5101/tcp, :::5101->5101/tcp hsds_sn_1 f87c4f497ff1 hdfgroup/hsds "/bin/bash -c 'sourc…" 48 seconds ago Up 46 seconds 5100-5999/tcp, 0.0.0.0:49169->6900/tcp, :::49169->6900/tcp hsds_rangeget_1 123ada60e5b1 hdfgroup/hsds "/bin/bash -c 'sourc…" 48 seconds ago Up 47 seconds 5101-5999/tcp, 0.0.0.0:49168->5100/tcp, :::49168->5100/tcp hsds_head_1 ``` Is HS\_ENDPOINT here the hsds\_endpoint in config file? I set it as [http://localhost](http://localhost) without any port while I set endpoint as [http://localhost:5101](http://localhost:5101) in hsconfigure. I’m still not sure if this error is caused by AWS connection or the local connection, because this error doesn’t occur when I setup single POSIX project. ---
**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Post date:** [August 3, 2021, 8:11am UTC](https://forum.hdfgroup.org/t/file-storage-on-aws-s3/8715/16 "2021-08-03T08:11:35Z")
If you set the env variable HS\_ENDPOINT it will override what’s in the config file. Your hsds\_sn\_1 container is exposed on port 5101, so that is what you should use. Try: `curl http://localhost:5101/about` as a sanity check.