# Authenticated AWS S3

**URL:** <https://forum.hdfgroup.org/t/authenticated-aws-s3/9924>\
**Category:** h5py\
**Created:** [June 30, 2022, 9:49am UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924 "2022-06-30T09:49:02Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![hdfgroup.web](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/eb9ed0/32.png) [@hdfgroup.web](https://forum.hdfgroup.org/u/hdfgroup.web)\
**Post date:** [June 30, 2022, 9:49am UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/1 "2022-06-30T09:49:02Z")

</div>

The ros3 driver is very slick. It works great for public h5 files.

I am trying to perform a read of an H5 file using ros3 that requires authentication credentials.  
I have been successful in using h5ls/h5dump in accessing an h5 file against an authenticated AWS S3 call using --vfd=ros3 and --s3-cred=(\<region,\<key\_id\>,\<key\_secret\>).

However, I cannot get this to work with h5py.  
I see that in [h5p.pyx](https://github.com/h5py/h5py/blob/master/h5py/h5p.pyx#L1085), set\_fapl\_ros3 appears to accept strings for aws\_region, secret\_id, and secret\_key. These appear to be the same three parameters that h5ls/h5dump accept for --s3-cred. However, I have not been able to get this to work.

My call looks like:  
`h5f = h5py.File(<url>, driver='ros3', aws_region='<region>'.encode('utf-8'), secret_id='<secret_id>'.encode('utf-8'), secret_key='<secret_key>'.encode('utf-8') )`

The error is not too informative: `OSError: Unable to open file (curl cannot perform request)` and it breaks on the `h5f.open()` call

Can anyone share a minimum working example? I can’t seem to find this documented anywhere. Any help would be much appreciated. Thank you.

---

<div class="post-metadata">

**Author:** ![ajelenak](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/3d9bf3/32.png) [@ajelenak](https://forum.hdfgroup.org/u/ajelenak)\
**Post date:** [June 30, 2022, 2:56pm UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/2 "2022-06-30T14:56:12Z")

</div>

Add this to your code:

```auto
h5py._errors.unsilence_errors()

```

and you should get the complete HDF5 library error stack. That may give some useful information about the problem.

-Aleksandar

---

<div class="post-metadata">

**Author:** ![hdfgroup.web](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/eb9ed0/32.png) [@hdfgroup.web](https://forum.hdfgroup.org/u/hdfgroup.web)\
**Post date:** [June 30, 2022, 5:26pm UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/3 "2022-06-30T17:26:13Z")

</div>

Thank you, that was very informative. I was able to get it working. My example syntax above was correct, I had an error in the URL. This works now!

---

<div class="post-metadata">

**Author:** ![hdfgroup.web](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/eb9ed0/32.png) [@hdfgroup.web](https://forum.hdfgroup.org/u/hdfgroup.web)\
**Post date:** [July 2, 2022, 7:47pm UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/4 "2022-07-02T19:47:35Z")

</div>

A couple follow-up notes/observations:

I was wrong, the authentication parameters are documented under the ros3 subsection of the [File Drivers](https://docs.h5py.org/en/stable/high/file.html#file-drivers) section of the [File Objects](https://docs.h5py.org/en/stable/high/file.html) page.

This S3 connector (both HDF5 and h5py) is not specific to just AWS’s S3 implementation. It appears to support [generic S3 providers](https://rclone.org/s3) in both public and authenticated modes. For example, I was able to get the h5py ros3 driver to work with GCP Cloud Storage (public and authenticated) using their [interoperability API](https://cloud.google.com/storage/docs/interoperability) that is S3 compatible. Authenticated calls are supported with [HMAC keys](https://cloud.google.com/storage/docs/authentication/hmackeys).

Thank you again!

---

<div class="post-metadata">

**Author:** ![chuck](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/c/8e7dd6/32.png) [@chuck](https://forum.hdfgroup.org/u/chuck)\
**Post date:** [October 14, 2022, 12:17pm UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/5 "2022-10-14T12:17:03Z")

</div>

I can’t get this to work by specifying only the `aws_region` keyword (in addition to the driver).

This works:

```auto
>>> url = "https://...h5" # Pointing to an S3 object
>>> h5f = h5py.File(url, driver="ros3")
>>> h5f
<HDF5 file "...h5" (mode r)>

```

but this fails:

```nohighlight
>>> h5f = h5py.File(url, driver="ros3", aws_region="us-west-2".encode("utf-8"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/gedi_subset/lib/python3.10/site-packages/h5py/_hl/files.py", line 502, in __init__
    fapl = make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0,
  File "/opt/conda/envs/gedi_subset/lib/python3.10/site-packages/h5py/_hl/files.py", line 166, in make_fapl
    set_fapl(plist, **kwds)
  File "/opt/conda/envs/gedi_subset/lib/python3.10/site-packages/h5py/_hl/files.py", line 78, in <lambda>
    _drivers['ros3'] = lambda plist, **kwargs: plist.set_fapl_ros3(**kwargs)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5p.pyx", line 1072, in h5py.h5p.PropFAID.set_fapl_ros3
ValueError: Invalid ros3 config (Inconsistent authentication information)

```

Should the `aws_region` be specified _only_ when I also need to supply `access_id` and `access_key` as well? In other words, is `aws_region` _not_ allowed for unauthenticated access?

I might suspect that `aws_region` is unnecessary when using the https url, so I tried using the corresponding `s3://` url instead, and that bombed with the same error as above.

Further, when I used the `s3://` url _without_ specifying `aws_region`, I got the following error instead:

```nohighlight
>>> h5 = h5py.File("s3://...h5", driver="ros3")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/envs/gedi_subset/lib/python3.10/site-packages/h5py/_hl/files.py", line 507, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
  File "/opt/conda/envs/gedi_subset/lib/python3.10/site-packages/h5py/_hl/files.py", line 220, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 106, in h5py.h5f.open
OSError: Unable to open file (invalid SCHEME construction)

```

Unfortunately, the [official docs](https://docs.h5py.org/en/stable/high/file.html#file-drivers) are not particularly informative regarding such details, so I’m having a hard time unraveling this mystery.

---

<div class="post-metadata">

**Author:** ![ajelenak](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/a/3d9bf3/32.png) [@ajelenak](https://forum.hdfgroup.org/u/ajelenak)\
**Post date:** [October 14, 2022, 3:21pm UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/6 "2022-10-14T15:21:36Z")

</div>

Hi @chuck,

This is in the process of getting fixed in h5py. See h5py issue [#2133](https://github.com/h5py/h5py/issues/2133) and the PR [#2140](https://github.com/h5py/h5py/pull/2140). My first PR that fixes your authentication error has already been merged and the requirements are (hopefully) better explained in the [_latest_](https://github.com/h5py/h5py/pull/2140) documentation version.

-Aleksandar

---

<div class="post-metadata">

**Author:** ![chuck](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/c/8e7dd6/32.png) [@chuck](https://forum.hdfgroup.org/u/chuck)\
**Post date:** [October 14, 2022, 6:06pm UTC](https://forum.hdfgroup.org/t/authenticated-aws-s3/9924/7 "2022-10-14T18:06:58Z")

</div>

Perfect! Thanks for the prompt reply.