Http://hsdshdflab.hdfgroup.org gives 503 Service Unavailable

HSDS
1

Since two weeks or so I get the following response from HSDS:

image
image1268×440 21.7 KB

I have noticed it because my unit tests fail which test against this HSDS service as this is the only one I found to be publicly available. As a workaround I have disabled those tests but I would like to know if the server is really “at capacity” for a longer period of time or is there some failure in the service?

Thank you

2

We are in the processing of shutting down hsdshdflab (and the HDF Lab Jupyter Hub). Since now there are many free Jupyter environments (e.g. AWS SageMaker), it makes sense to utilize our resources in other areas.

Do your unit tests need to write data or just read? If the later you can use the NREL endpoint: https://developer.nrel.gov/api/hsds (you’ll need to go the NREL website to request an API key).

Also, you might want to look at github action for the HSDS tests: https://github.com/HDFGroup/hsds/blob/master/.github/workflows/python-package.yml. In line 67, it starts HSDS as a local process and then shuts it down after the test run.

1 Like
3

I need the service only for reading. I prefer to avoid being dependend on the NREL instance because my project is open source and although I can put a secret in my GitHub project to be used in GitHub action, I would not be able to run the unit tests locally anymore without carrying around the API key separately from my repository.

Therefore I will go with the second option and run HSDS myself. How do I ensure that the same datasets and attributes as for hsdshdflab are present in my local instance? E.g. the dataset “/shared/tall.h5”?

4

OK I found this: https://github.com/HDFGroup/hsds/blob/ae1e926502dcbd023b9b20ba981cad54086a64e7/docs/post_install.md

That should be a good starting point :slight_smile:

5

Yes, that should work. Let us know how it goes!

6

First thing I noticed is that the HSDS installation via pip may fail soon due to

DEPRECATION: hsds is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559

Installing the wheel package before installing hsds seems to solve that issue. Maybe it should be part of the hsds dependencies. I think I get this message because with my Ubuntu 23.04 setup I am forced to use virtual environments and a fresh virtual environment has only two packages installed:

> pip list
Package    Version
---------- -------
pip        23.0.1
setuptools 66.1.1
7

I noticed that the pip version does not yet support TCP sockets, at least specifying host and port on the command line does not have any effect and the HSDS startup code differs from that in the GitHub repository. I have seen that version 0.8 is being prepared and there it looks like TCP sockets will be supported. I would like to continue trying to run HSDS locally when version 0.8 is release. Is there already a release schedule?

8

version 0.8.0 of the HSDS package is on PyPI now.
To run without docker but with TCP, you can do: ./runall.sh --no-docker-tcp.

If you’d rather run the hsds directly you can do:
hsds --bucket_name ${BUCKET_NAME} --password_file ${PASSWORD_FILE} --logfile hs.log --loglevel ${LOG_LEVEL} --host localhost --port ${SN_PORT} --config_dir=${CONFIG_DIR} --count=${DN_CORES}

You just need to use appropriate values for the environment variables listed plus ROOT_DIR (if you will be using POSIX storage) or BUCKET_NAME and AWS_REGION, and AWS_S3_GATEWAY if using AWS.

9

Also, I created an issue for the setup tools deprecation: Update setup.py to remove setuptoos deprecation · Issue #243 · HDFGroup/hsds · GitHub.
Will try to get this resolved soon.

10

Thank you, I will test it soon.

11

It works very well now, thanks again for the 0.8.0 release.

There are two small minors things that you can maybe clarify / help me with:

  • When I load the sample tall.h5 file into HSDS, I get multiple fill value loading warnings/errors like this: WARNING:root:runtime error getting fillvalue for dataset: /g1/g1.1/dset1.1.1
  • When HSDS has been started in the background with the ampersand symbol (hsds <options> &), how to I stop it properly after my test? I tried pkill hsds but that does not stop everything and while running pkill python instead works, it seems to break the database because I cannot run the test a second time then without also deleting the database itself. Also pkill python kills other python processes, too.

Thank you