Hi everyone,
I’m working on a research project involving real-time performance monitoring across various mobile environments, and I’m running into a consistent bottleneck with my HSDS (Highly Scalable Data Service) setup. I’m attempting to stream high-frequency execution logs directly into an HDF5 cloud-backed store, but I keep hitting 504 Gateway Timeout errors when the client-side script complexity increases.
The specific issue arises when I use a delta executor for iphone to simulate rapid-fire event triggers for my data collection. It seems like the overhead of the REST API calls, combined with the way iOS handles background threading for the executor, is causing the HSDS service to struggle with the concurrent write requests.
Has anyone else dealt with performance tuning for HSDS when the data source is a mobile-based script environment? I’m trying to figure out if I should be batching these writes more aggressively on the client side or if there’s a specific hsds configuration (like max_request_size or io_timeout) that handles this type of high-velocity script data more gracefully. I’m worried that the intermittent connectivity of a mobile executor is fragmenting my datasets faster than the service can re-index them. Any advice on optimizing the handshake between mobile automation scripts and the HDF5 REST interface would be a huge help!
Hey,
Thanks for your question! This is the first time I’ve heard of anyone using HSDS with an iOS client.
There are some constraints the affect how many inflight requests HSDS can process. In hsds/admin/config.py, there’s a max_task_count entry that sets this parameter (100 by default). If a new request comes in and the number of requests the service is already processing is greater than max_task_count then the server will respond with a 503 Service Unavailable error. This is a signal to the client that it needs to slow down on the rate of requests. If you expect the requests won’t be especially memory intensive, you should be able to increase max_task_count with out ill effect.
But anyway, you are seeing 504s not 503s. That’s usually either an issue with a component between the client and the service (like a load balancer), or HSDS being down. Could you describe your network setup? That might provide some clues.
Check the HSDS logs. If you don’t see any warnings or errors in the hsds logs, it’s likely an issue that came up somewhere before the request hit the service.
Also if you run hsabout (included with the h5pyd package) it will show you the start time of the service. If you see something much later than when you started the service, that would indicate a service re-start that needs to be investigated. If HSDS restarts, there will be a 5-10 interval before the service is able to process requests.
Please update us on the project as it progresses. I’m quite curious what you find out.
