Broadcast and reduce extent features

FYI, there are two new features for HSDS that are now in the master branch (GitHub - HDFGroup/hsds: Cloud-native, service based access to HDF data).

Broadcast lets you do numpy style broadcasting. For example if you need to set a large dataset selection to a specific value, you can just send the single value in the HTTP PUT request rather than sending a large request contains all the elements to write to the selection. See https://github.com/HDFGroup/hsds/blob/master/tests/integ/broadcast_test.py to see how this works.

The next h5pyd release will use this feature to support broadcasting efficiently. E.g. dset[...] = 99 would set every element to 99.

The other feature is that now dataset shapes can be reduced in size as well as increased. After a shape reduction, any allocated chunks that are no longer in the reduced shape will be deleted.

If you have comments or questions about these features, please post here or contact help@hdfgroup.org.

3 Likes