H5pyd authentication issue with Keycloak

HDF5 Library
1

Hi,
I am encountering issues when I try to use Keycloak for authentication. I am using h5pyd library to interface with the HSDS server.
I created the .hscfg file in my home dir as indicated in the installation procedure hsds/docs/keycloak_setup.md at master · HDFGroup/hsds · GitHub

But when I try to use commands like hsls, I am getting the error below:

Traceback (most recent call last):
File “/mnt/c/Users/User/Desktop/scripts/venv/bin/hsls”, line 5, in
from h5pyd._apps.hsls import main
File “/mnt/c/Users/User/Desktop/scripts/venv/lib/python3.12/site-packages/h5pyd/_apps/hsls.py”, line 17, in
cfg = Config()
^^^^^^^^
File “/mnt/c/Users/User/Desktop/scripts/venv/lib/python3.12/site-packages/h5pyd/_apps/config.py”, line 130, in init
raise ValueError(f"undefined option: {name}")
ValueError: undefined option: ignore

For the HSDS configuration, I created the override.yml file in hsds/admin/config. The configuration seems correct since when I launch runall.sh command, the containers start as usual.
Also, when I am using the default authentication method which is to use a passwd.txt file, there is no issue.

Below are the versions of the packages I am using:

  • Python 3.12.3
  • HSDS v0.8.4
  • Docker version 26.1.3, build 26.1.3-0ubuntu1~24.04.1
  • docker-compose version 1.29.2, build unknown
  • keycloak-15.0.2
  • h5pyd 0.18.0

I launch the HSDS server from WSL2 on Windows 11 (dist: Ubuntu 24.04.1 LTS)
The storage is Azure Blob Storage.

Thank you for your help.

2

Thanks for the detailed issue report!
I haven’t tested with Keycloak recently, but I’ll give it a try and get back to you.

BTW, how do you like WSL2? I was thinking of switching my Linux laptop to Windows and using WSL2 to cover testing with Linux.

3

After a bit of time re-acquainting myself with Keycloak, I have some updates to HSDS and h5pyd that should fix the issue.

You’ll need to get the latest HSDS on master and rebuild (or fetch the latest hsds:master from DockerHub). Also you need to fetch and build h5pyd from master (or do a pip install git+https://github.com/HDFGroup/h5pyd).

Finally, note that the openidurl has been updated in: hsds/docs/keycloak_setup.md at master · HDFGroup/hsds · GitHub.

One thing to be aware of is that if you are running HSDS and Keycloak in Docker, HSDS won’t be able by default to access the Keycloak url. I got around this by running Keycloak in Docker and HSDS with the ./runall.sh --no-docker-tcp option.

Anyway, let us know if these updates work for you.

4

Hi, I have just tested to run HSDS without docker as you said and so far it seems to work fine! I will let you know if I come across other problems
Thank you!

5

Great!

FYI - if you do want to run HSDS in Docker, you’ll need to either use a KeyCloak endpoint that’s accessible via DNS, or modify your docker compose to include a KeyCloak container. It’s just Docker doesn’t allow to access other containers that aren’t part of the same docker-compose yaml.

6

Hi again, I think there is an issue with latest version of h5pyd (0.21.0). When I launch HSDS with keycloak and try to send requests to the server through h5pyd, I get the error below :

Traceback (most recent call last):
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/bin/hsls", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/lib/python3.12/site-packages/h5pyd/_apps/hsls.py", line 533, in main
    count = visitDomains(domain, depth=depth)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/lib/python3.12/site-packages/h5pyd/_apps/hsls.py", line 347, in visitDomains
    d = getFolder(domain + '/')
        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/lib/python3.12/site-packages/h5pyd/_apps/hsls.py", line 313, in getFolder
    d = h5py.Folder(domain, endpoint=endpoint, username=username, verbose=verbose,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/lib/python3.12/site-packages/h5pyd/_hl/folders.py", line 172, in __init__
    self._http_conn = HttpConn(
                      ^^^^^^^^^
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/lib/python3.12/site-packages/h5pyd/_hl/httpconn.py", line 312, in __init__
    self._api_key = openid.KeycloakOpenID(
                    ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/c/Users/TheoPLANTEFOL/Desktop/tests/venv/lib/python3.12/site-packages/h5pyd/_hl/openid.py", line 352, in __init__
    scopes = hs_config.get('hs_keycloak_scopes', 'openid').split()
             ^^^^^^^^^^^^^
AttributeError: 'Config' object has no attribute 'get'
7

I made some updates in 0.21.0 for compatibility with h5py.get_config(), but it looks like there was a regression.

I put a fix in master branch. If you could build fh5pyd rom master please let me know if that works with KeyCloak.