hsload NetCDF4 file


#1

Is hsload capable of uploading NetCDF4 files? I have a file that fails to upload and hsload gives a core dump. Any ideas what goes wrong?

BR,
Harri

sboxes@osboxes:~$ python --version
Python 3.7.3
osboxes@osboxes:~$ hsload -v -u -p WLS.nc /home/test_user2/
creating attribute title in /
creating attribute Conventions in /
creating attribute institution in /
creating attribute references in /
creating attribute source in /
creating attribute history in /
creating attribute comment in /
creating attribute instrument_name in /
create_links: /
got link: Sweep_119916
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: Sweep_119916
WARNING: could not find map item to src id: 3640696641629669709
got link: altitude
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: altitude
WARNING: could not find map item to src id: 3644576009099261484
got link: georeference_correction
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: georeference_correction
WARNING: could not find map item to src id: 3641622515394424284
got link: instrument_type
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: instrument_type
WARNING: could not find map item to src id: 3631887032700034734
got link: latitude
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: latitude
WARNING: could not find map item to src id: 3632029474817689284
got link: longitude
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: longitude
WARNING: could not find map item to src id: 3634328897574112734
got link: sweep
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: sweep
WARNING: could not find map item to src id: 3643754786686253109
got link: sweep_fixed_angle
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: sweep_fixed_angle
WARNING: could not find map item to src id: 3645238800993654084
got link: sweep_group_name
creating multilink <HDF5 file “WLS.nc” (mode r+)> with title: sweep_group_name
WARNING: could not find map item to src id: 3644000426664657384
creating group /Sweep_119916
create_links: /Sweep_119916
got link: scan_file_dim
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: scan_file_dim
WARNING: could not find map item to src id: 3650311484571351834
got link: scan_file
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: scan_file
WARNING: could not find map item to src id: 3650690845721431809
got link: settings_file_dim
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: settings_file_dim
WARNING: could not find map item to src id: 3651231544372120509
got link: settings_file
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: settings_file
WARNING: could not find map item to src id: 3651545498427359109
got link: res_file_dim
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: res_file_dim
WARNING: could not find map item to src id: 3611222751203149659
got link: res_file
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: res_file
WARNING: could not find map item to src id: 3613480022720675334
got link: ray_angle_resolution
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: ray_angle_resolution
WARNING: could not find map item to src id: 3628471328858318484
got link: range_gate_length
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: range_gate_length
WARNING: could not find map item to src id: 3627715513540151484
got link: ray_accumulation_time
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: ray_accumulation_time
WARNING: could not find map item to src id: 3624147193143805359
got link: time_reference
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: time_reference
WARNING: could not find map item to src id: 3680526655406074134
got link: sweep_mode
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: sweep_mode
WARNING: could not find map item to src id: 3679451072068682634
got link: sweep_index
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: sweep_index
WARNING: could not find map item to src id: 3681840611266887534
got link: rotation_direction
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: rotation_direction
WARNING: could not find map item to src id: 3682206890997999234
got link: time
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: time
WARNING: could not find map item to src id: 3681497587391719434
got link: range
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: range
WARNING: could not find map item to src id: 3686420561396780259
got link: timestamp
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: timestamp
WARNING: could not find map item to src id: 3683407474561087584
got link: ray_index
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: ray_index
WARNING: could not find map item to src id: 3692462723469126834
got link: azimuth
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: azimuth
WARNING: could not find map item to src id: 3694891506924236559
got link: elevation
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: elevation
WARNING: could not find map item to src id: 3657548416242800859
got link: cnr
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: cnr
WARNING: could not find map item to src id: 3653052768590703684
got link: gate_index
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: gate_index
WARNING: could not find map item to src id: 3596555573557720434
got link: radial_wind_speed
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: radial_wind_speed
WARNING: could not find map item to src id: 3598792496201295459
got link: radial_wind_speed_ci
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: radial_wind_speed_ci
WARNING: could not find map item to src id: 3815032712219870634
got link: radial_wind_speed_status
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: radial_wind_speed_status
WARNING: could not find map item to src id: 3711249094598566209
got link: doppler_spectrum_width
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: doppler_spectrum_width
WARNING: could not find map item to src id: 3722750568853672884
got link: doppler_spectrum_mean_error
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: doppler_spectrum_mean_error
WARNING: could not find map item to src id: 3226027288263324009
got link: atmospherical_structures_type
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: atmospherical_structures_type
WARNING: could not find map item to src id: 3718717131338454759
got link: relative_beta
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: relative_beta
WARNING: could not find map item to src id: 3233913930410197359
got link: absolute_beta
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: absolute_beta
WARNING: could not find map item to src id: 3165387643890387009
got link: instrumental_function_x_max
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: instrumental_function_x_max
WARNING: could not find map item to src id: 3211175517261342459
got link: instrumental_function_y_average
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: instrumental_function_y_average
WARNING: could not find map item to src id: 3324685896631057584
got link: instrumental_function_amplitude
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: instrumental_function_amplitude
WARNING: could not find map item to src id: 3324278919152044584
got link: instrumental_function_half_height_width
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: instrumental_function_half_height_width
WARNING: could not find map item to src id: 3320563796165054484
got link: instrumental_function_status
creating multilink <HDF5 group “/Sweep_119916” (0 members)> with title: instrumental_function_status
WARNING: could not find map item to src id: 3319476584899691184
creating dataset /Sweep_119916/absolute_beta, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-6cba-691c01-52694f, chunk_size: (120, 57)
creating dataset /Sweep_119916/atmospherical_structures_type, shape: (120, 57), type: int32
dataset created, uuid: d-6770f047-b3306fa2-bd1a-933d9d-772875, chunk_size: (120, 57)
creating dataset /Sweep_119916/azimuth, shape: (120,), type: float64
dataset created, uuid: d-6770f047-b3306fa2-4b63-dfbf47-18070e, chunk_size: (120,)
creating dataset /Sweep_119916/cnr, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-bfa9-1b14ea-f50b64, chunk_size: (120, 57)
creating dataset /Sweep_119916/doppler_spectrum_mean_error, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-f903-d849bc-b6756b, chunk_size: (120, 57)
creating dataset /Sweep_119916/doppler_spectrum_width, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-8a64-dd36d8-dc8f0b, chunk_size: (120, 57)
creating dataset /Sweep_119916/elevation, shape: (120,), type: float64
dataset created, uuid: d-6770f047-b3306fa2-02b1-a350ad-fdd855, chunk_size: (120,)
creating dataset /Sweep_119916/gate_index, shape: (57,), type: int32
dataset created, uuid: d-6770f047-b3306fa2-b460-cc5662-0dcb05, chunk_size: (57,)
creating dataset /Sweep_119916/instrumental_function_amplitude, shape: (), type: float64
dataset created, uuid: d-6770f047-b3306fa2-b5d1-871acc-55650f, chunk_size: (1,)
creating dataset /Sweep_119916/instrumental_function_half_height_width, shape: (), type: float64
dataset created, uuid: d-6770f047-b3306fa2-6dc7-cd7256-532ec6, chunk_size: (1,)
creating dataset /Sweep_119916/instrumental_function_status, shape: (), type: uint8
dataset created, uuid: d-6770f047-b3306fa2-82c2-27aa3a-fa3611, chunk_size: (1,)
creating dataset /Sweep_119916/instrumental_function_x_max, shape: (), type: float64
dataset created, uuid: d-6770f047-b3306fa2-69eb-f7ae23-1c4349, chunk_size: (1,)
creating dataset /Sweep_119916/instrumental_function_y_average, shape: (), type: float64
dataset created, uuid: d-6770f047-b3306fa2-4edc-b7a398-2b6593, chunk_size: (1,)
creating dataset /Sweep_119916/radial_wind_speed, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-3ea8-59d562-09b1c0, chunk_size: (120, 57)
creating dataset /Sweep_119916/radial_wind_speed_ci, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-d5b4-c10cad-160798, chunk_size: (120, 57)
creating dataset /Sweep_119916/radial_wind_speed_status, shape: (120, 57), type: uint8
dataset created, uuid: d-6770f047-b3306fa2-bb1c-8ac93c-067bf5, chunk_size: (120, 57)
creating dataset /Sweep_119916/range, shape: (57,), type: int32
dataset created, uuid: d-6770f047-b3306fa2-e2c3-c3e796-863dca, chunk_size: (57,)
creating dataset /Sweep_119916/range_gate_length, shape: (), type: float64
dataset created, uuid: d-6770f047-b3306fa2-59ce-bb2c23-56d184, chunk_size: (1,)
creating dataset /Sweep_119916/ray_accumulation_time, shape: (), type: int32
dataset created, uuid: d-6770f047-b3306fa2-c86f-f91354-4a615c, chunk_size: (1,)
creating dataset /Sweep_119916/ray_angle_resolution, shape: (), type: float64
dataset created, uuid: d-6770f047-b3306fa2-a184-1e65eb-561142, chunk_size: (1,)
creating dataset /Sweep_119916/ray_index, shape: (120,), type: float64
dataset created, uuid: d-6770f047-b3306fa2-9f6d-78a3a0-26b2d4, chunk_size: (120,)
creating dataset /Sweep_119916/relative_beta, shape: (120, 57), type: float64
dataset created, uuid: d-6770f047-b3306fa2-e621-44c986-b67799, chunk_size: (120, 57)
creating dataset /Sweep_119916/res_file, shape: (5808,), type: |S1
dataset created, uuid: d-6770f047-b3306fa2-aed1-6580eb-ad3044, chunk_size: (5808,)
creating dataset /Sweep_119916/res_file_dim, shape: (5808,), type: >f4
dataset created, uuid: d-6770f047-b3306fa2-c3f1-7ebd4b-f23897, chunk_size: (5808,)
creating dataset /Sweep_119916/rotation_direction, shape: (), type: object
dataset created, uuid: d-6770f047-b3306fa2-d4c4-32b981-85e098, chunk_size: (1,)
malloc_consolidate(): invalid chunk size
Aborted (core dumped)
osboxes@osboxes:~$


#2

Hi Harri,

Did you install the h5pyd package with pip? There’s a recent fix for files with object references (that’s just about every NetCDF4 file I believe) that hasn’t made it to PyPI yet.

You can get the latest h5pyd by: $ git clone https://github.com/HDFGroup/h5pyd and $ python setup.py install.

Let us know if that works for you!


#3

Hi John,

Yes, I installed with pip and h5pyd version was 0.7.1. I re-installed from GitHub and the version is now 0.7.2. The problem still exists.

BR,
Harri

osboxes@osboxes:~$ /usr/local/bin/hsinfo
server name: Highly Scalable Data Service (HSDS)
server state: READY
endpoint: http://34.245.68.198
username: admin
password: ***********
server version: 0.6_beta
node count: 1
up: 1 days, 19 hours 4 min 6 sec
h5pyd version: 0.7.2
osboxes@osboxes:~$ /usr/local/bin/hsload -v -u test_user2 -p **** WLS.nc /home/test_user2/
creating group /Sweep_119916
creating dataset /Sweep_119916/absolute_beta, shape: (120, 57), type: float64
dataset created, uuid: d-eb1c72e1-a102ce27-9a95-780c79-9e09d8, chunk_size: (120, 57)
creating dataset /Sweep_119916/atmospherical_structures_type, shape: (120, 57), type: int32
dataset created, uuid: d-eb1c72e1-a102ce27-9068-f450ba-b0c75f, chunk_size: (120, 57)

…deleted lines…

creating dataset /Sweep_119916/sweep_mode, shape: (), type: object
dataset created, uuid: d-eb1c72e1-a102ce27-c76e-fd8f14-ca69db, chunk_size: (1,)
free(): double free detected in tcache 2
Aborted (core dumped)
osboxes@osboxes:~$


#4

Hey Harri,

I’ve pushed some fixes for h5pyd on github that should resolve the problem. Can you do a git pull/python setup.py install, and try loading the file? Let me know how it goes.

HDF5 is not happy with getting the fill value for variable length datasets, so the code change guards against that.


#5

Hi John,

Thanks for the fix. I can confirm that h5pyd 0.7.3 now uploads my NetCDF4 ok.

BR,
Harri