Errors installing HSDS via Python / PIP


#1

Hi there,

I’ve been trying to follow the readme on Github to install and run HSDS on a) a Debian 10 LXC Container (as root) , and b) Ubuntu Focal VM (as a user with sudo)

First I want to point to this problem Link, that I also encountered on the Ubuntu machine (= user install) but I could work around it by using symbolic links to point form the missing links/files in /usr/bin to /usr/local/bin

But on both ends I end up failing the first integrity tests:

    setup_test
    testHomeFolders
    req: http+unix://%2Ftmp%2Fhs%2Fsn_1.sock/
    domain: /home
    /home get status: 400
    got status code: 400
    F
    ======================================================================
    FAIL: testHomeFolders (__main__.SetupTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "setup_test.py", line 80, in testHomeFolders
    self.assertEqual(rsp.status_code, 200)
    AssertionError: 400 != 200

    ----------------------------------------------------------------------
    Ran 1 test in 0.004s

    FAILED (failures=1)
    Failed

I understand something is not reachable?

on the Ubuntu VM its actually worse, with Excpection-Inception:

    OK
    cwd /root/hsds/tests
    uptest
    EE/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py:745: ResourceWarning: unclosed <socket.socket fd=3, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
      keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]
    ResourceWarning: Enable tracemalloc to get the object allocation traceback
    /usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py:745: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
      keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]
    ResourceWarning: Enable tracemalloc to get the object allocation traceback
    E
    ======================================================================
    ERROR: testCorsGetAbout (__main__.UpTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
      File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
      File "/usr/local/lib/python3.9/site-packages/requests_unixsocket-0.3.0-py3.9.egg/requests_unixsocket/adapters.py", line 41, in connect
    sock.connect(socket_path)
    ConnectionRefusedError: [Errno 111] Connection refused

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
      File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
      File "/usr/local/lib/python3.9/site-packages/requests_unixsocket-0.3.0-py3.9.egg/requests_unixsocket/adapters.py", line 41, in connect
    sock.connect(socket_path)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/root/hsds/tests/integ/uptest.py", line 32, in testCorsGetAbout
    rsp = self.session.options(
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 553, in options
    return self.request('OPTIONS', url, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

    ======================================================================
    ERROR: testGetAbout (__main__.UpTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
      File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
      File "/usr/local/lib/python3.9/site-packages/requests_unixsocket-0.3.0-py3.9.egg/requests_unixsocket/adapters.py", line 41, in connect
    sock.connect(socket_path)
    ConnectionRefusedError: [Errno 111] Connection refused

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
      File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
      File "/usr/local/lib/python3.9/site-packages/requests_unixsocket-0.3.0-py3.9.egg/requests_unixsocket/adapters.py", line 41, in connect
    sock.connect(socket_path)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/root/hsds/tests/integ/uptest.py", line 51, in testGetAbout
    rsp = self.session.get(req)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

    ======================================================================
    ERROR: testGetInfo (__main__.UpTest)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
      File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
      File "/usr/local/lib/python3.9/site-packages/requests_unixsocket-0.3.0-py3.9.egg/requests_unixsocket/adapters.py", line 41, in connect
    sock.connect(socket_path)
    ConnectionRefusedError: [Errno 111] Connection refused

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
      File "/usr/local/lib/python3.9/site-packages/urllib3-1.26.8-py3.9.egg/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
      File "/usr/local/lib/python3.9/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
      File "/usr/local/lib/python3.9/http/client.py", line 1010, in _send_output
    self.send(msg)
      File "/usr/local/lib/python3.9/http/client.py", line 950, in send
    self.connect()
      File "/usr/local/lib/python3.9/site-packages/requests_unixsocket-0.3.0-py3.9.egg/requests_unixsocket/adapters.py", line 41, in connect
    sock.connect(socket_path)
    urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/root/hsds/tests/integ/uptest.py", line 71, in testGetInfo
    rsp = self.session.get(req)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 542, in get
    return self.request('GET', url, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
      File "/usr/local/lib/python3.9/site-packages/requests-2.27.1-py3.9.egg/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
    requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionRefusedError(111, 'Connection refused'))

    ----------------------------------------------------------------------
    Ran 3 tests in 0.007s

    FAILED (errors=3)
    Failed

I’ll try the docker as well, but these might be worth investigating?


#2

Hi Martin,

The quickstart setup (./runall.sh --no-docker) will run a server that can only be accessed from that machine. The endpoint “http+unix://%2Ftmp%2Fhs%2Fsn_1.sock/” is actually a url-encoded pointer to the unix domain socket at /tmp/hs/sn_1.sock.

On my machine I see the following in the /tmp/hs/ directory:

$ ls -l /tmp/hs
total 8
srwxr-xr-x  1 john  wheel     0 Feb  7 11:42 dn_1.sock
srwxr-xr-x  1 john  wheel     0 Feb  7 11:42 dn_2.sock
srwxr-xr-x  1 john  wheel     0 Feb  7 11:42 dn_3.sock
srwxr-xr-x  1 john  wheel     0 Feb  7 11:42 dn_4.sock
-rw-r--r--  1 john  wheel  3376 Feb  7 11:50 hs.log
srwxr-xr-x  1 john  wheel     0 Feb  7 11:42 rangeget.sock
srwxr-xr-x  1 john  wheel     0 Feb  7 11:42 sn_1.sock

The files with the ‘s’ in the permission bits are unix domain socket files. Using unix domain sockets is a bit faster than TCP/IP sockets and doesn’t need any special user permissions to run, but they are not accessible from a remote machine (or a different container running on the host). I.e. it’s handy for trying out things with HSDS, but not what you need if you require a service that can be accessed by other machines. If that’s a case, give the docker setup a go: https://github.com/HDFGroup/hsds/blob/master/docs/docker_install_posix.md.