Conversion error when using hsget

HSDS
1

Hi,

I attempt to copy HSDS domain data to a local h5 file using “hsget /shots/001000/001038/data test.h5”. The error is returned seemingly regarding conversion issues. Could you help me diagnose what might be the cause? Or some other ways to copy to local h5 files?

Thanks!

2023-11-30 10:17:13,440 ERROR: failed to create attribute date of object / – No conversion path for dtype: dtype(‘<U26’)
ERROR: failed to create attribute date of object / – No conversion path for dtype: dtype(‘<U26’)
2023-11-30 10:17:13,460 ERROR: failed to create attribute flare.data version of object / – No conversion path for dtype: dtype(‘<U3’)
ERROR: failed to create attribute flare.data version of object / – No conversion path for dtype: dtype(‘<U3’)
Traceback (most recent call last):
File “c:\programdata\miniconda3\lib\runpy.py”, line 193, in _run_module_as_main
“main”, mod_spec)
File “c:\programdata\miniconda3\lib\runpy.py”, line 85, in run_code
exec(code, run_globals)
File "C:\ProgramData\Miniconda3\Scripts\hsget.exe_main.py", line 9, in
File “c:\programdata\miniconda3\lib\site-packages\h5pyd_apps\hsget.py”, line 200, in main
load_file(fin, fout, verbose=verbose, nodata=nodata)
File “c:\programdata\miniconda3\lib\site-packages\h5pyd_apps\utillib.py”, line 491, in load_file
fin.visititems(object_create_helper)
File “c:\programdata\miniconda3\lib\site-packages\h5pyd_hl\group.py”, line 773, in visititems
retval = func(parent.name, parent)
File “c:\programdata\miniconda3\lib\site-packages\h5pyd_apps\utillib.py”, line 465, in object_create_helper
create_dataset(obj, ctx)
File “c:\programdata\miniconda3\lib\site-packages\h5pyd_apps\utillib.py”, line 317, in create_dataset
scaleoffset=dobj.scaleoffset)
File “c:\programdata\miniconda3\lib\site-packages\h5py_hl\group.py”, line 136, in create_dataset
dsid = dataset.make_new_dset(self, shape, dtype, data, **kwds)
File “c:\programdata\miniconda3\lib\site-packages\h5py_hl\dataset.py”, line 167, in make_new_dset
dset_id = h5d.create(parent.id, None, tid, sid, dcpl=dcpl)
File “h5py_objects.pyx”, line 54, in h5py._objects.with_phil.wrapper
File “h5py_objects.pyx”, line 55, in h5py._objects.with_phil.wrapper
File “h5py\h5d.pyx”, line 79, in h5py.h5d.create
ValueError: Unable to create dataset (no appropriate function for conversion path)

2

Hi,
This is very curious! The numpy datatype “<U26” (or “<U3”) is a fixed-width UTF32 string. These aren’t currently supported in HDF5 or HSDS. Do you know how these got created? Could you provide some code to illustrate how this attribute was added?

We recently added support for fixed-width UTF8 strings to HSDS (since they are supported in the library, it’s a feature that was good to have for the REST VOL), but fixed-width UTF8 is a bit wonky – the number of bytes needed to represent a given unicode string is not always clear. Since any Unicode character can be encoded in 4 bytes or less, UTF32 would be a good option if you need fixed-width unicode support. We can consider adding this to HSDS and supporting in h5pyd, but even then translating to an HDF5 file with hsget would still be problematic.

3

I believe numpy internally stores all unicode strings, including UTF-8, as UTF-32.

4

Check data types in the source file /shots/001000/001038/ data for attributes date and flare.data version and also check these data types align with the destination file format test.h5 to prevent conversion errors.