# HSDS **URL:** https://forum.hdfgroup.org/c/hsds/6.md [Latest](https://forum.hdfgroup.org/latest.md) · [Categories](https://forum.hdfgroup.org/categories.md) --- ## [About the HSDS category](https://forum.hdfgroup.org/t/about-the-hsds-category/19)
**Author:** [@info1](https://forum.hdfgroup.org/u/info1)\ **Replies:** 0\ **Last updated:** [February 7, 2018, 11:10pm UTC](https://forum.hdfgroup.org/t/about-the-hsds-category/19 "2018-02-07T23:10:51Z")
Questions about Highly Scalable Data Service (HSDS) usage and implementation. --- ## [HSDS + xarray + h5netcdf ](https://forum.hdfgroup.org/t/hsds-xarray-h5netcdf/13476)
**Author:** [@simhav](https://forum.hdfgroup.org/u/simhav)\ **Replies:** 5\ **Last updated:** [September 8, 2025, 5:18pm UTC](https://forum.hdfgroup.org/t/hsds-xarray-h5netcdf/13476 "2025-09-08T17:18:05Z")
Hey! I am currently trying to upgrade dependencies in our project as numpy 1.\* nears EOL. Currently we are on and want to migrate to: xarray==2022.12.0 → 2025.7.0 h5netcdf==1.3.0 → 1.6.4 h5py==3.10.0 → h5pyd==0.14.1… --- ## [Accelerating data ingest with h5pyd](https://forum.hdfgroup.org/t/accelerating-data-ingest-with-h5pyd/12937)
**Author:** [@leo](https://forum.hdfgroup.org/u/leo)\ **Replies:** 11\ **Last updated:** [June 11, 2025, 8:28am UTC](https://forum.hdfgroup.org/t/accelerating-data-ingest-with-h5pyd/12937 "2025-06-11T08:28:50Z")
Hi, I have the usecase that my datasets are highly compressible. I am often dealing with a compression factor of 99%. When uploading a .h5 file to hsds it seems to always decompress the data and upload them uncompresse… --- ## [NREL example repo](https://forum.hdfgroup.org/t/nrel-example-repo/13009)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 0\ **Last updated:** [January 29, 2025, 2:48pm UTC](https://forum.hdfgroup.org/t/nrel-example-repo/13009 "2025-01-29T14:48:44Z")
FYI - if anyone is curious to see HSDS used with really large datasets (\> 1TB), checkout this repo: GitHub - NREL/hsds-examples: Examples of using the HSDS Service to Access NREL WIND Toolkit data. It’s hosted by NREL,… --- ## [HSDS and h5pyd new releases](https://forum.hdfgroup.org/t/hsds-and-h5pyd-new-releases/12860)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 0\ **Last updated:** [December 4, 2024, 11:46pm UTC](https://forum.hdfgroup.org/t/hsds-and-h5pyd-new-releases/12860 "2024-12-04T23:46:45Z")
New releases of HSDS and h5pyd are out! The newest HSDS release is 0.9.1. You can get it with pip install hsds or get the Docker image using: docker pull hdfgroup/hsds:0.9.1. New features in this release: Improved s… --- ## [Numpystyle fancy indexing of datasets](https://forum.hdfgroup.org/t/numpystyle-fancy-indexing-of-datasets/12538)
**Author:** [@leo](https://forum.hdfgroup.org/u/leo)\ **Replies:** 19\ **Last updated:** [December 3, 2024, 7:43pm UTC](https://forum.hdfgroup.org/t/numpystyle-fancy-indexing-of-datasets/12538 "2024-12-03T19:43:44Z")
Hi, I have the case where I need to access may different scattered indices in a large dataset. I was hoping to use numpy style indexing to be able to do it in on request. There seems to be support for only one of those … --- ## [HSDS Map Objects](https://forum.hdfgroup.org/t/hsds-map-objects/12719)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 0\ **Last updated:** [October 14, 2024, 10:52am UTC](https://forum.hdfgroup.org/t/hsds-map-objects/12719 "2024-10-14T10:52:29Z")
Since the HDF5 library now has an interface for maps: H5M (even though currently only the DAOS VOL supports it), I thought it would be interesting to see how map support could fit into the HSDS architecture. So I create… --- ## [Read from multiple datasets at once](https://forum.hdfgroup.org/t/read-from-multiple-datasets-at-once/12492)
**Author:** [@leo](https://forum.hdfgroup.org/u/leo)\ **Replies:** 6\ **Last updated:** [July 11, 2024, 6:10pm UTC](https://forum.hdfgroup.org/t/read-from-multiple-datasets-at-once/12492 "2024-07-11T18:10:44Z")
Hello, I wanted to ask if there is a a functionality in h5pyd to retrieve data from multiple datasets/ groups in one call. Context: I am asking for performance reasons, especially for reading fairly small subsets. My … --- ## [Issues with Azure installation](https://forum.hdfgroup.org/t/issues-with-azure-installation/12450)
**Author:** [@r.trebu](https://forum.hdfgroup.org/u/r.trebu)\ **Replies:** 3\ **Last updated:** [June 26, 2024, 5:43pm UTC](https://forum.hdfgroup.org/t/issues-with-azure-installation/12450 "2024-06-26T17:43:51Z")
Hello, I am testing the installation of HSDS with azure using the windows wsl2, the launch of the service works, if I do a curl on it I manage to get an answer but I currently have a problem with the service. When I do … --- ## [Setup of HSDS from azure marketplace](https://forum.hdfgroup.org/t/setup-of-hsds-from-azure-marketplace/12455)
**Author:** [@leo](https://forum.hdfgroup.org/u/leo)\ **Replies:** 2\ **Last updated:** [June 26, 2024, 2:20pm UTC](https://forum.hdfgroup.org/t/setup-of-hsds-from-azure-marketplace/12455 "2024-06-26T14:20:55Z")
Hi everyone, I am trying to set up hsds in an azure VM that I host in the cloud. I used the VM that is offered on the Azure marketplace and ran the script. The docker containers seem to be running. I installed h5pyd on… --- ## [HSDS launching only on localhost](https://forum.hdfgroup.org/t/hsds-launching-only-on-localhost/12362)
**Author:** [@r.trebu](https://forum.hdfgroup.org/u/r.trebu)\ **Replies:** 4\ **Last updated:** [June 21, 2024, 1:27pm UTC](https://forum.hdfgroup.org/t/hsds-launching-only-on-localhost/12362 "2024-06-21T13:27:01Z")
Hello. I’m trying to run HSDS on an Azure virtual machine with podman and docker-compose. Currently I’ve managed to get the service running on the local host (i.e. I used h5pyd to store a file and then retrieved the file… --- ## [Performance & Cost benchmarks on AWS?](https://forum.hdfgroup.org/t/performance-cost-benchmarks-on-aws/12176)
**Author:** [@keith.knuth](https://forum.hdfgroup.org/u/keith.knuth)\ **Replies:** 3\ **Last updated:** [April 11, 2024, 6:13pm UTC](https://forum.hdfgroup.org/t/performance-cost-benchmarks-on-aws/12176 "2024-04-11T18:13:51Z")
Hello all, I am new to HSDS, but after its availability for a few years now I am wondering if there have been any published benchmarks for AWS use? I have seen the Local HSDS performance vs local HDF5 files - HSDS - HD… --- ## [HSDS Domain - bucket validation](https://forum.hdfgroup.org/t/hsds-domain-bucket-validation/12120)
**Author:** [@phoulee](https://forum.hdfgroup.org/u/phoulee)\ **Replies:** 5\ **Last updated:** [April 1, 2024, 2:42pm UTC](https://forum.hdfgroup.org/t/hsds-domain-bucket-validation/12120 "2024-04-01T14:42:13Z")
Hi, Attempting to use HSDS with nrel. I’m running into an issue regarding the changeset made from this MR: Restrict bucket names to alphanumeric characters and underscores by mattjala · Pull Request #319 · HDFGroup/hs… --- ## [Is it possible to make complex query on hsds](https://forum.hdfgroup.org/t/is-it-possible-to-make-complex-query-on-hsds/12040)
**Author:** [@r.trebu](https://forum.hdfgroup.org/u/r.trebu)\ **Replies:** 1\ **Last updated:** [March 1, 2024, 7:19pm UTC](https://forum.hdfgroup.org/t/is-it-possible-to-make-complex-query-on-hsds/12040 "2024-03-01T19:19:54Z")
Hello everyone, I’m learning to use hsds and I was wondering if with the different APIs, especially h5pyd, it would be possible to make complex queries similar to what we can do with pytables. I’m interested in any tuto… --- ## [Connecting HSDS to NREL data stored on S3](https://forum.hdfgroup.org/t/connecting-hsds-to-nrel-data-stored-on-s3/12009)
**Author:** [@matthew](https://forum.hdfgroup.org/u/matthew)\ **Replies:** 7\ **Last updated:** [February 16, 2024, 7:11am UTC](https://forum.hdfgroup.org/t/connecting-hsds-to-nrel-data-stored-on-s3/12009 "2024-02-16T07:11:31Z")
I would like to run some analysis on the NREL wind speed gridded dataset. The existing REST API, accessed from h5pyd at this end, is slow. I’m aiming to run my own HSDS server on an EC2 instance in the us-west-2 AWS re… --- ## [Implementing an associative matrix in HDF](https://forum.hdfgroup.org/t/implementing-an-associative-matrix-in-hdf/11942)
**Author:** [@scott1](https://forum.hdfgroup.org/u/scott1)\ **Replies:** 5\ **Last updated:** [January 29, 2024, 8:02pm UTC](https://forum.hdfgroup.org/t/implementing-an-associative-matrix-in-hdf/11942 "2024-01-29T20:02:20Z")
I have a use case that is common in information/data science and AI. I’d like to use HDF to store an associative array, which is essentially a (float or integer) matrix, with some sort of code for each row and column. In… --- ## [HSDS 0.9 release target](https://forum.hdfgroup.org/t/hsds-0-9-release-target/11917)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 1\ **Last updated:** [January 25, 2024, 10:10am UTC](https://forum.hdfgroup.org/t/hsds-0-9-release-target/11917 "2024-01-25T10:10:29Z")
I’m hoping to have an official release of HSDS 0.9 out early this year, so it would be a good idea to take a stab at finalizing what features to include. There’s already a bunch of stuff that is in the master branch, t… --- ## [Up to date openapi.yaml](https://forum.hdfgroup.org/t/up-to-date-openapi-yaml/11904)
**Author:** [@jonathan.graesser](https://forum.hdfgroup.org/u/jonathan.graesser)\ **Replies:** 3\ **Last updated:** [January 3, 2024, 12:57pm UTC](https://forum.hdfgroup.org/t/up-to-date-openapi-yaml/11904 "2024-01-03T12:57:50Z")
Hi HSDS-Team, I have been testing HSDS for a few days and it works really nice. My only difficulty was that the differences between domain and folders are never really explained in the docs and there was some trail and… --- ## [Conversion error when using hsget](https://forum.hdfgroup.org/t/conversion-error-when-using-hsget/11773)
**Author:** [@pshi](https://forum.hdfgroup.org/u/pshi)\ **Replies:** 3\ **Last updated:** [December 22, 2023, 8:10am UTC](https://forum.hdfgroup.org/t/conversion-error-when-using-hsget/11773 "2023-12-22T08:10:29Z")
Hi, I attempt to copy HSDS domain data to a local h5 file using “hsget /shots/001000/001038/data test.h5”. The error is returned seemingly regarding conversion issues. Could you help me diagnose what might be the cause?… --- ## [Error using HSDS to download NSRDB data](https://forum.hdfgroup.org/t/error-using-hsds-to-download-nsrdb-data/11840)
**Author:** [@acheyette](https://forum.hdfgroup.org/u/acheyette)\ **Replies:** 4\ **Last updated:** [December 21, 2023, 2:10am UTC](https://forum.hdfgroup.org/t/error-using-hsds-to-download-nsrdb-data/11840 "2023-12-21T02:10:41Z")
Hello, I recently posted this issue on github, but posting here as well in case this is monitored more frequently. I have set up a local HSDS server using a version of these steps. I am trying to run SAM using nsrdb dat… --- ## [UTF-8 string support](https://forum.hdfgroup.org/t/utf-8-string-support/11593)
**Author:** [@jan-willem.blokland](https://forum.hdfgroup.org/u/jan-willem.blokland)\ **Replies:** 8\ **Last updated:** [December 18, 2023, 3:18am UTC](https://forum.hdfgroup.org/t/utf-8-string-support/11593 "2023-12-18T03:18:58Z")
Hello, Still working in testing my library (build on top of HDF5) in combination with the HSDS functionality (see Fill value support - HSDS - HDF Forum (hdfgroup.org)). In at least one of my test I make use of UTF-8 str… --- ## [Bitshuffle support](https://forum.hdfgroup.org/t/bitshuffle-support/11774)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 0\ **Last updated:** [December 1, 2023, 1:59am UTC](https://forum.hdfgroup.org/t/bitshuffle-support/11774 "2023-12-01T01:59:53Z")
The bitshuffle filter is now supported in HSDS. Bitshuffle works much like the standard HDF5 library byteshuffle, except that it works at the bit level. Also, bitshuffle includes it’s own lz4 compression. See: bitshuf… --- ## [H5pyd 0.16.0 Release](https://forum.hdfgroup.org/t/h5pyd-0-16-0-release/11571)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 0\ **Last updated:** [September 28, 2023, 5:57pm UTC](https://forum.hdfgroup.org/t/h5pyd-0-16-0-release/11571 "2023-09-28T17:57:02Z")
A new version of h5pyd has been released. Updates: Fixes for numpy deprecation issues Support for creating domains in root folder (for admin users) Removed h5serv support from test cases Fix for issue: #152 --- ## [Broadcast and reduce extent features](https://forum.hdfgroup.org/t/broadcast-and-reduce-extent-features/11650)
**Author:** [@jreadey](https://forum.hdfgroup.org/u/jreadey)\ **Replies:** 0\ **Last updated:** [October 27, 2023, 7:52pm UTC](https://forum.hdfgroup.org/t/broadcast-and-reduce-extent-features/11650 "2023-10-27T19:52:01Z")
FYI, there are two new features for HSDS that are now in the master branch (GitHub - HDFGroup/hsds: Cloud-native, service based access to HDF data). Broadcast lets you do numpy style broadcasting. For example if you ne… --- ## [Unsupported compression: AEC/SZIP](https://forum.hdfgroup.org/t/unsupported-compression-aec-szip/11608)
**Author:** [@jan-willem.blokland](https://forum.hdfgroup.org/u/jan-willem.blokland)\ **Replies:** 2\ **Last updated:** [October 17, 2023, 6:29am UTC](https://forum.hdfgroup.org/t/unsupported-compression-aec-szip/11608 "2023-10-17T06:29:29Z")
Hello, In one of my tests I make use of AEC/SZIP compression. From the forum discussion ZFP compression supported? - HSDS - HDF Forum (hdfgroup.org) I understand that this compression type is not supported. That is fine… --- ## [Attribute size: C struct padding problem](https://forum.hdfgroup.org/t/attribute-size-c-struct-padding-problem/11602)
**Author:** [@jan-willem.blokland](https://forum.hdfgroup.org/u/jan-willem.blokland)\ **Replies:** 4\ **Last updated:** [October 13, 2023, 7:46am UTC](https://forum.hdfgroup.org/t/attribute-size-c-struct-padding-problem/11602 "2023-10-13T07:46:10Z")
Hello, While testing VOL-REST in combination with HSDS, I think I found a bug related the padding of a C struct. Suppose, one the following c struct typedef struct { char\[64\] str1; char\[64\] str2; int32\_t valu… --- ## [Fill value support](https://forum.hdfgroup.org/t/fill-value-support/11583)
**Author:** [@jan-willem.blokland](https://forum.hdfgroup.org/u/jan-willem.blokland)\ **Replies:** 4\ **Last updated:** [October 5, 2023, 6:24pm UTC](https://forum.hdfgroup.org/t/fill-value-support/11583 "2023-10-05T18:24:21Z")
Hello, Currently, I am working on supporting S3 in my library which is build on top of HDF5 using VOL-REST in combination with HSDS. In one of my test I noticed that fill value is not supported in HSDS 0.8.x. Are there… --- ## [ZFP compression supported?](https://forum.hdfgroup.org/t/zfp-compression-supported/11528)
**Author:** [@jan-willem.blokland](https://forum.hdfgroup.org/u/jan-willem.blokland)\ **Replies:** 3\ **Last updated:** [October 4, 2023, 6:12am UTC](https://forum.hdfgroup.org/t/zfp-compression-supported/11528 "2023-10-04T06:12:03Z")
Hello, In VOL-REST: memory-side hyperslab unsupported?, it was mentioned that the selected filters are applied inside HSDS. Now I am wondering which filters are currently supported? Is the H5Z-ZFP filter based on the ZF… --- ## [Http://hsdshdflab.hdfgroup.org gives 503 Service Unavailable](https://forum.hdfgroup.org/t/http-hsdshdflab-hdfgroup-org-gives-503-service-unavailable/11376)
**Author:** [@apollo3zehn-h5](https://forum.hdfgroup.org/u/apollo3zehn-h5)\ **Replies:** 10\ **Last updated:** [August 18, 2023, 8:59am UTC](https://forum.hdfgroup.org/t/http-hsdshdflab-hdfgroup-org-gives-503-service-unavailable/11376 "2023-08-18T08:59:27Z")
Since two weeks or so I get the following response from HSDS: I have noticed it because my unit tests fail which test against this HSDS service as this is the only one I found to be publicly available. As a workaroun… --- ## [How much is too much for the NREL API?](https://forum.hdfgroup.org/t/how-much-is-too-much-for-the-nrel-api/11400)
**Author:** [@isucolin](https://forum.hdfgroup.org/u/isucolin)\ **Replies:** 13\ **Last updated:** [August 17, 2023, 5:11pm UTC](https://forum.hdfgroup.org/t/how-much-is-too-much-for-the-nrel-api/11400 "2023-08-17T17:11:56Z")
Hi! We are trying to use the NREL sup3r data but having no luck. When I try to run the “using the data” example \[sup3r/examples/sup3rcc/using\_the\_data.ipynb at main · NREL/sup3r (github.com)\] I get the same error discu… [Next page](https://forum.hdfgroup.org/c/hsds/6.md?page=1)