I try to use HSDS as a data collector for multiple clients, i.e. multiple devices write data to HSDS.
Each client writes data every 1s to his own domain (e.g. /clients/client0.h5).
The clients use h5pyd to access the domains and write to the arrays.
First I hosted HSDS as a docker image on a Azure VM. The yml is configured to use one servicenode and 8 datanodes.
When one client starts writing data everything runs fast.
But as soon as new clients start accessing their domains and start writing, it slows down a lot.
With one client uploading it takes about 0.02s. 8 clients uploading variies between 5s to 8s for each.
Now I switched to Kubernetes. I followed the instructions to install HSDS on Azure Kubernetes and also added Azure AD and Front Door.
I scaled HSDS on Kubernetes to 30 pods. Everything works great for up to 5-8 clients(it varies for every test run I take).
But there are at least two that take 2s to write their data. You can also see the send/write time increasing when number of clients accessing rises.
Can somebody relate and give me tip? Where could this come from?
Am I missing something?