# hsload NetCDF4 file

**URL:** <https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357>\
**Category:** HSDS\
**Created:** [July 30, 2020, 2:52pm UTC](https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357 "2020-07-30T14:52:54Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![harri.hytonen](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/e19b73/32.png) [@harri.hytonen](https://forum.hdfgroup.org/u/harri.hytonen)\
**Post date:** [July 30, 2020, 2:52pm UTC](https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357/1 "2020-07-30T14:52:55Z")

</div>

Is hsload capable of uploading NetCDF4 files? I have a [file](https://alexapublic.s3-eu-west-1.amazonaws.com/certs/WLS.nc) that fails to upload and hsload gives a core dump. Any ideas what goes wrong?

BR,  
Harri

sboxes@osboxes:~$ python --version  
Python 3.7.3  
osboxes@osboxes:~$ hsload -v -u -p WLS.nc /home/test\_user2/  
creating attribute title in /  
creating attribute Conventions in /  
creating attribute institution in /  
creating attribute references in /  
creating attribute source in /  
creating attribute history in /  
creating attribute comment in /  
creating attribute instrument\_name in /  
create\_links: /  
got link: Sweep\_119916  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: Sweep\_119916  
WARNING: could not find map item to src id: 3640696641629669709  
got link: altitude  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: altitude  
WARNING: could not find map item to src id: 3644576009099261484  
got link: georeference\_correction  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: georeference\_correction  
WARNING: could not find map item to src id: 3641622515394424284  
got link: instrument\_type  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: instrument\_type  
WARNING: could not find map item to src id: 3631887032700034734  
got link: latitude  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: latitude  
WARNING: could not find map item to src id: 3632029474817689284  
got link: longitude  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: longitude  
WARNING: could not find map item to src id: 3634328897574112734  
got link: sweep  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: sweep  
WARNING: could not find map item to src id: 3643754786686253109  
got link: sweep\_fixed\_angle  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: sweep\_fixed\_angle  
WARNING: could not find map item to src id: 3645238800993654084  
got link: sweep\_group\_name  
creating multilink \<HDF5 file “WLS.nc” (mode r+)\> with title: sweep\_group\_name  
WARNING: could not find map item to src id: 3644000426664657384  
creating group /Sweep\_119916  
create\_links: /Sweep\_119916  
got link: scan\_file\_dim  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: scan\_file\_dim  
WARNING: could not find map item to src id: 3650311484571351834  
got link: scan\_file  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: scan\_file  
WARNING: could not find map item to src id: 3650690845721431809  
got link: settings\_file\_dim  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: settings\_file\_dim  
WARNING: could not find map item to src id: 3651231544372120509  
got link: settings\_file  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: settings\_file  
WARNING: could not find map item to src id: 3651545498427359109  
got link: res\_file\_dim  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: res\_file\_dim  
WARNING: could not find map item to src id: 3611222751203149659  
got link: res\_file  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: res\_file  
WARNING: could not find map item to src id: 3613480022720675334  
got link: ray\_angle\_resolution  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: ray\_angle\_resolution  
WARNING: could not find map item to src id: 3628471328858318484  
got link: range\_gate\_length  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: range\_gate\_length  
WARNING: could not find map item to src id: 3627715513540151484  
got link: ray\_accumulation\_time  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: ray\_accumulation\_time  
WARNING: could not find map item to src id: 3624147193143805359  
got link: time\_reference  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: time\_reference  
WARNING: could not find map item to src id: 3680526655406074134  
got link: sweep\_mode  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: sweep\_mode  
WARNING: could not find map item to src id: 3679451072068682634  
got link: sweep\_index  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: sweep\_index  
WARNING: could not find map item to src id: 3681840611266887534  
got link: rotation\_direction  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: rotation\_direction  
WARNING: could not find map item to src id: 3682206890997999234  
got link: time  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: time  
WARNING: could not find map item to src id: 3681497587391719434  
got link: range  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: range  
WARNING: could not find map item to src id: 3686420561396780259  
got link: timestamp  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: timestamp  
WARNING: could not find map item to src id: 3683407474561087584  
got link: ray\_index  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: ray\_index  
WARNING: could not find map item to src id: 3692462723469126834  
got link: azimuth  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: azimuth  
WARNING: could not find map item to src id: 3694891506924236559  
got link: elevation  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: elevation  
WARNING: could not find map item to src id: 3657548416242800859  
got link: cnr  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: cnr  
WARNING: could not find map item to src id: 3653052768590703684  
got link: gate\_index  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: gate\_index  
WARNING: could not find map item to src id: 3596555573557720434  
got link: radial\_wind\_speed  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: radial\_wind\_speed  
WARNING: could not find map item to src id: 3598792496201295459  
got link: radial\_wind\_speed\_ci  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: radial\_wind\_speed\_ci  
WARNING: could not find map item to src id: 3815032712219870634  
got link: radial\_wind\_speed\_status  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: radial\_wind\_speed\_status  
WARNING: could not find map item to src id: 3711249094598566209  
got link: doppler\_spectrum\_width  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: doppler\_spectrum\_width  
WARNING: could not find map item to src id: 3722750568853672884  
got link: doppler\_spectrum\_mean\_error  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: doppler\_spectrum\_mean\_error  
WARNING: could not find map item to src id: 3226027288263324009  
got link: atmospherical\_structures\_type  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: atmospherical\_structures\_type  
WARNING: could not find map item to src id: 3718717131338454759  
got link: relative\_beta  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: relative\_beta  
WARNING: could not find map item to src id: 3233913930410197359  
got link: absolute\_beta  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: absolute\_beta  
WARNING: could not find map item to src id: 3165387643890387009  
got link: instrumental\_function\_x\_max  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: instrumental\_function\_x\_max  
WARNING: could not find map item to src id: 3211175517261342459  
got link: instrumental\_function\_y\_average  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: instrumental\_function\_y\_average  
WARNING: could not find map item to src id: 3324685896631057584  
got link: instrumental\_function\_amplitude  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: instrumental\_function\_amplitude  
WARNING: could not find map item to src id: 3324278919152044584  
got link: instrumental\_function\_half\_height\_width  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: instrumental\_function\_half\_height\_width  
WARNING: could not find map item to src id: 3320563796165054484  
got link: instrumental\_function\_status  
creating multilink \<HDF5 group “/Sweep\_119916” (0 members)\> with title: instrumental\_function\_status  
WARNING: could not find map item to src id: 3319476584899691184  
creating dataset /Sweep\_119916/absolute\_beta, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-6cba-691c01-52694f, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/atmospherical\_structures\_type, shape: (120, 57), type: int32  
dataset created, uuid: d-6770f047-b3306fa2-bd1a-933d9d-772875, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/azimuth, shape: (120,), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-4b63-dfbf47-18070e, chunk\_size: (120,)  
creating dataset /Sweep\_119916/cnr, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-bfa9-1b14ea-f50b64, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/doppler\_spectrum\_mean\_error, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-f903-d849bc-b6756b, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/doppler\_spectrum\_width, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-8a64-dd36d8-dc8f0b, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/elevation, shape: (120,), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-02b1-a350ad-fdd855, chunk\_size: (120,)  
creating dataset /Sweep\_119916/gate\_index, shape: (57,), type: int32  
dataset created, uuid: d-6770f047-b3306fa2-b460-cc5662-0dcb05, chunk\_size: (57,)  
creating dataset /Sweep\_119916/instrumental\_function\_amplitude, shape: (), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-b5d1-871acc-55650f, chunk\_size: (1,)  
creating dataset /Sweep\_119916/instrumental\_function\_half\_height\_width, shape: (), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-6dc7-cd7256-532ec6, chunk\_size: (1,)  
creating dataset /Sweep\_119916/instrumental\_function\_status, shape: (), type: uint8  
dataset created, uuid: d-6770f047-b3306fa2-82c2-27aa3a-fa3611, chunk\_size: (1,)  
creating dataset /Sweep\_119916/instrumental\_function\_x\_max, shape: (), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-69eb-f7ae23-1c4349, chunk\_size: (1,)  
creating dataset /Sweep\_119916/instrumental\_function\_y\_average, shape: (), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-4edc-b7a398-2b6593, chunk\_size: (1,)  
creating dataset /Sweep\_119916/radial\_wind\_speed, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-3ea8-59d562-09b1c0, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/radial\_wind\_speed\_ci, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-d5b4-c10cad-160798, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/radial\_wind\_speed\_status, shape: (120, 57), type: uint8  
dataset created, uuid: d-6770f047-b3306fa2-bb1c-8ac93c-067bf5, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/range, shape: (57,), type: int32  
dataset created, uuid: d-6770f047-b3306fa2-e2c3-c3e796-863dca, chunk\_size: (57,)  
creating dataset /Sweep\_119916/range\_gate\_length, shape: (), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-59ce-bb2c23-56d184, chunk\_size: (1,)  
creating dataset /Sweep\_119916/ray\_accumulation\_time, shape: (), type: int32  
dataset created, uuid: d-6770f047-b3306fa2-c86f-f91354-4a615c, chunk\_size: (1,)  
creating dataset /Sweep\_119916/ray\_angle\_resolution, shape: (), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-a184-1e65eb-561142, chunk\_size: (1,)  
creating dataset /Sweep\_119916/ray\_index, shape: (120,), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-9f6d-78a3a0-26b2d4, chunk\_size: (120,)  
creating dataset /Sweep\_119916/relative\_beta, shape: (120, 57), type: float64  
dataset created, uuid: d-6770f047-b3306fa2-e621-44c986-b67799, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/res\_file, shape: (5808,), type: |S1  
dataset created, uuid: d-6770f047-b3306fa2-aed1-6580eb-ad3044, chunk\_size: (5808,)  
creating dataset /Sweep\_119916/res\_file\_dim, shape: (5808,), type: \>f4  
dataset created, uuid: d-6770f047-b3306fa2-c3f1-7ebd4b-f23897, chunk\_size: (5808,)  
creating dataset /Sweep\_119916/rotation\_direction, shape: (), type: object  
dataset created, uuid: d-6770f047-b3306fa2-d4c4-32b981-85e098, chunk\_size: (1,)  
malloc\_consolidate(): invalid chunk size  
Aborted (core dumped)  
osboxes@osboxes:~$

---

<div class="post-metadata">

**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\
**Post date:** [July 30, 2020, 8:55pm UTC](https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357/2 "2020-07-30T20:55:14Z")

</div>

Hi Harri,

Did you install the h5pyd package with pip? There’s a recent fix for files with object references (that’s just about every NetCDF4 file I believe) that hasn’t made it to PyPI yet.

You can get the latest h5pyd by: $ git clone [https://github.com/HDFGroup/h5pyd](https://github.com/HDFGroup/h5pyd) and $ python setup.py install.

Let us know if that works for you!

---

<div class="post-metadata">

**Author:** ![harri.hytonen](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/e19b73/32.png) [@harri.hytonen](https://forum.hdfgroup.org/u/harri.hytonen)\
**Post date:** [August 3, 2020, 5:55pm UTC](https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357/3 "2020-08-03T17:55:37Z")

</div>

Hi John,

Yes, I installed with pip and h5pyd version was 0.7.1. I re-installed from GitHub and the version is now 0.7.2. The problem still exists.

BR,  
Harri

osboxes@osboxes:~$ /usr/local/bin/hsinfo  
server name: Highly Scalable Data Service (HSDS)  
server state: READY  
endpoint: [http://34.245.68.198](http://34.245.68.198)  
username: admin  
password: \*\*\*\*\*\*\*\*\*\*\*  
server version: 0.6\_beta  
node count: 1  
up: 1 days, 19 hours 4 min 6 sec  
h5pyd version: 0.7.2  
osboxes@osboxes:~$ /usr/local/bin/hsload -v -u test\_user2 -p \*\*\*\* WLS.nc /home/test\_user2/  
creating group /Sweep\_119916  
creating dataset /Sweep\_119916/absolute\_beta, shape: (120, 57), type: float64  
dataset created, uuid: d-eb1c72e1-a102ce27-9a95-780c79-9e09d8, chunk\_size: (120, 57)  
creating dataset /Sweep\_119916/atmospherical\_structures\_type, shape: (120, 57), type: int32  
dataset created, uuid: d-eb1c72e1-a102ce27-9068-f450ba-b0c75f, chunk\_size: (120, 57)

…deleted lines…

creating dataset /Sweep\_119916/sweep\_mode, shape: (), type: object  
dataset created, uuid: d-eb1c72e1-a102ce27-c76e-fd8f14-ca69db, chunk\_size: (1,)  
free(): double free detected in tcache 2  
Aborted (core dumped)  
osboxes@osboxes:~$

---

<div class="post-metadata">

**Author:** ![jreadey](https://forum.hdfgroup.org/user_avatar/forum.hdfgroup.org/jreadey/32/870_2.png) [@jreadey](https://forum.hdfgroup.org/u/jreadey)\
**Post date:** [August 4, 2020, 7:31pm UTC](https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357/4 "2020-08-04T19:31:57Z")

</div>

> [@harri.hytonen](#):
>
> file

Hey Harri,

I’ve pushed some fixes for h5pyd on github that should resolve the problem. Can you do a git pull/python setup.py install, and try loading the file? Let me know how it goes.

HDF5 is not happy with getting the fill value for variable length datasets, so the code change guards against that.

---

<div class="post-metadata">

**Author:** ![harri.hytonen](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/h/e19b73/32.png) [@harri.hytonen](https://forum.hdfgroup.org/u/harri.hytonen)\
**Post date:** [August 5, 2020, 9:03am UTC](https://forum.hdfgroup.org/t/hsload-netcdf4-file/7357/5 "2020-08-05T09:03:25Z")

</div>

Hi John,

Thanks for the fix. I can confirm that h5pyd 0.7.3 now uploads my NetCDF4 ok.

BR,  
Harri