HDF5 DIAG Error MPI

nabiz · December 4, 2024, 6:21pm

I run a code on jupyter notebook where netcdf files are read on a server with xarray on a super computer.

The opening and processing data goes seamless. But I get constantly this Error Warning, that might be of lacking MPI settings, which overblown my small Jupyter Notebook of 50 kb into 500 MB, which then makes it hard to git push.

Does anyone know what the error warning is and how can I suppress it on the notebook?

**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 473 in H5O__attr_open_by_name(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #008: H5Adense.c line 383 in H5A__dense_open(): can't locate attribute in name index**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 473 in H5O__attr_open_by_name(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #008: H5Adense.c line 383 in H5A__dense_open(): can't locate attribute in name index**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 473 in H5O__attr_open_by_name(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #008: H5Adense.c line 383 in H5A__dense_open(): can't locate attribute in name index**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronously open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #001: H5A.c line 641 in H5A__open_by_name_api_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #002: H5A.c line 464 in H5A__open_common(): unable to open attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Can't open object**
**  #003: H5VLcallback.c line 1138 in H5VL_attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #004: H5VLcallback.c line 1105 in H5VL__attr_open(): attribute open failed**
**    major: Virtual Object Layer**
**    minor: Can't open object**
**  #005: H5VLnative_attr.c line 169 in H5VL__native_attr_open(): can't open attribute**
**    major: Attribute**
**    minor: Can't open object**
**  #006: H5Aint.c line 638 in H5A__open_by_name(): unable to load attribute info from object header**
**    major: Attribute**
**    minor: Unable to initialize object**
**  #007: H5Oattribute.c line 491 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits'**
**    major: Attribute**
**    minor: Object not found**
**HDF5-DIAG: Error detected in HDF5 (1.14.3) MPI-process 0:**
**  #000: H5A.c line 679 in H5Aopen_by_name(): unable to synchronousl***emphasized text*

dave.allured · December 4, 2024, 6:55pm

This appears to be a known problem with Netcdf or Xarray. The fix may be as simple as ensuring that your installed packages are up to date. Please review the following tickets for more information. Possibly file a help request with one of those upstream packages, if you do not find your solution.

github.com/Unidata/netcdf4-python

Many HDF5-DIAG warnings when used with xarray and dask

opened 03:17PM - 31 Mar 23 UTC

nbud

Steps to reproduce: * Environment: Python 3.10, Ubuntu 22.04, the packages were… installed in a new clean virtual environment using `pip install netcdf4 dask xarray` * Create a new netcdf4 dataset, read it with xarray and dask; see example script below. * Expected behavior: the script runs without warning * Actual behavior: the script runs successfully but with many HDF5-DIAG warnings pip list: ``` Package Version ------------------ -------- cftime 1.6.2 click 8.1.3 cloudpickle 2.2.1 dask 2023.3.2 fsspec 2023.3.0 importlib-metadata 6.1.0 locket 1.0.0 netCDF4 1.6.3 numpy 1.24.2 packaging 23.0 pandas 1.5.3 partd 1.3.0 pip 23.0.1 python-dateutil 2.8.2 pytz 2023.3 PyYAML 6.0 setuptools 67.4.0 six 1.16.0 toolz 0.12.0 wheel 0.38.4 xarray 2023.3.0 zipp 3.15.0 ``` Example script (taken from https://github.com/pydata/xarray/issues/7549#issue-1596115847): ```python import argparse import pathlib import tempfile from typing import List import netCDF4 import xarray HERE = pathlib.Path(__file__).parent def add_arguments(parser: argparse.ArgumentParser): parser.add_argument('count', type=int, default=200, nargs='?') parser.add_argument('--file-cache-maxsize', type=int, required=False) def main(): parser = argparse.ArgumentParser() add_arguments(parser) opts = parser.parse_args() if opts.file_cache_maxsize is not None: xarray.set_options(file_cache_maxsize=opts.file_cache_maxsize) temp_dir = tempfile.mkdtemp(dir=HERE, prefix='work-dir-') work_dir = pathlib.Path(temp_dir) print("Working in", work_dir.name) print("Making", opts.count, "datasets") dataset_paths = make_many_datasets(work_dir, count=opts.count) print("Combining", len(dataset_paths), "datasets") dataset = xarray.open_mfdataset(dataset_paths, lock=False) dataset.to_netcdf(work_dir / 'combined.nc') def make_many_datasets( work_dir: pathlib.Path, count: int = 200 ) -> List[pathlib.Path]: dataset_paths = [] for i in range(count): variable = f'var_{i}' path = work_dir / f'{variable}.nc' dataset_paths.append(path) make_dataset(path, variable) return dataset_paths def make_dataset( path: pathlib.Path, variable: str, ) -> None: ds = netCDF4.Dataset(path, "w", format="NETCDF4") ds.createDimension("x", 1) var = ds.createVariable(variable, "i8", ("x",)) var[:] = 1 ds.close() if __name__ == '__main__': main() ``` Content of stdout: ``` Working in work-dir-kt4qcsng Making 200 datasets Combining 200 datasets ``` Content of stderr (partial): ``` HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1: #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute major: Attribute minor: Can't open object #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed major: Virtual Object Layer minor: Can't open object #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed major: Virtual Object Layer minor: Can't open object #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute major: Attribute minor: Can't open object #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header major: Attribute minor: Unable to initialize object #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits' major: Attribute minor: Object not found HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1: #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute major: Attribute minor: Can't open object #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed major: Virtual Object Layer minor: Can't open object #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed major: Virtual Object Layer minor: Can't open object #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute major: Attribute minor: Can't open object #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header major: Attribute minor: Unable to initialize object #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeGranularBitRoundNumberOfSignificantDigits' major: Attribute minor: Object not found HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 1: #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute major: Attribute minor: Can't open object #001: H5VLcallback.c line 1091 in H5VL_attr_open(): attribute open failed major: Virtual Object Layer minor: Can't open object #002: H5VLcallback.c line 1058 in H5VL__attr_open(): attribute open failed major: Virtual Object Layer minor: Can't open object #003: H5VLnative_attr.c line 130 in H5VL__native_attr_open(): can't open attribute major: Attribute minor: Can't open object #004: H5Aint.c line 545 in H5A__open_by_name(): unable to load attribute info from object header major: Attribute minor: Unable to initialize object #005: H5Oattribute.c line 494 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits' major: Attribute minor: Object not found HDF5-DIAG: Error detected in HDF5 (1.12.2) thread 2: #000: H5A.c line 528 in H5Aopen_by_name(): can't open attribute major: Attribute minor: Can't open object ... ``` Downgrading to `netcdf4=1.5.8` fixes the issue. I don't know if the root cause is netcdf4, xarray or dask, I just wanted to raise awareness here. Probably related: https://github.com/Unidata/netcdf4-python/issues/1241

github.com/Unidata/netcdf-c

Bug in hdf5open.c::get_quantize_info()

opened 12:05PM - 05 Apr 23 UTC

closed 08:52PM - 17 Apr 23 UTC

uweschulzweida

I get the following error messages since netCDF-c version 4.9.0: ``` ... … #000: H5A.c line 545 in H5Aopen_by_name(): can't open attribute #001: H5VLcallback.c line 1128 in H5VL_attr_open(): attribute open failed #002: H5VLcallback.c line 1095 in H5VL__attr_open(): attribute open failed #003: H5VLnative_attr.c line 132 in H5VL__native_attr_open(): can't open attribute #004: H5Aint.c line 559 in H5A__open_by_name(): unable to load attribute info from object header #005: H5Oattribute.c line 508 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitGroomNumberOfSignificantDigits' … #005: H5Oattribute.c line 508 in H5O__attr_open_by_name(): can't locate attribute: ‚_QuantizeGranularBitRoundNumberOfSignificantDigits' ... #005: H5Oattribute.c line 508 in H5O__attr_open_by_name(): can't locate attribute: '_QuantizeBitRoundNumberOfSignificantBits' ``` I think in routine hdf5open.c::get_quantize_info() the corresponding calls to H5Aexists() are missing. I have added the following lines: attr_exists = H5Aexists(datasetid, NC_QUANTIZE_BITGROOM_ATT_NAME); attr_exists = H5Aexists(datasetid, NC_QUANTIZE_GRANULARBR_ATT_NAME); attr_exists = H5Aexists(datasetid, NC_QUANTIZE_BITROUND_ATT_NAME); With these calls I don't get any error messages anymore. The problem occurs when opening NetCDF4 files in a multi-threaded environment when calling the routine nc_inq_var(). In the following C++ example, 2 files are opened in parallel and both threads are locked with a unique_lock: ``` #include "netcdf.h" #include <cstdio> #include <cstdlib> #include <thread> #include <mutex> static std::mutex openMutex; static void nce(int istat) { if (istat != NC_NOERR) { fprintf(stderr, "%s\n", nc_strerror(istat)); exit(-1); } } static int open_file(const char *filename) { std::unique_lock<std::mutex> lockedMutex(openMutex); int ncid; nce(nc_open(filename, NC_NOWRITE, &ncid)); int ndims, nvars, ngatts, unlimdimid; nc_inq(ncid, &ndims, &nvars, &ngatts, &unlimdimid); for (int varid = 0; varid < nvars; ++varid) { nc_type xtype; int nvdims, nvatts, dimids[9]; nc_inq_var(ncid, varid, NULL, &xtype, &nvdims, dimids, &nvatts); } return ncid; } static void close_file(int ncid) { nce(nc_close(ncid)); } static void read_file(const char *filename) { int ncid = open_file(filename); close_file(ncid); } int main(void) { std::thread t1(read_file, "ifile1.nc"); std::thread t2(read_file, "ifile2.nc"); t1.join(); t2.join(); return 0; } ```

github.com/pydata/xarray

Xarray does not support full range of netcdf-python compression options

opened 02:21PM - 19 Dec 22 UTC

closed 03:24PM - 21 Dec 23 UTC

rabernat

bug topic-backends

### What is your issue? ### Summary The [netcdf4-python API docs](https://…unidata.github.io/netcdf4-python/#Dataset.createVariable) say the following > If the optional keyword argument compression is set, the data will be compressed in the netCDF file using the specified compression algorithm. Currently `zlib`,`szip`,`zstd`,`bzip2`,`blosc_lz`,`blosc_lz4`,`blosc_lz4hc`, `blosc_zlib` and `blosc_zstd` are supported. Default is None (no compression). All of the compressors except `zlib` and `szip` use the HDF5 plugin architecture. > > If the optional keyword `zlib` is True, the data will be compressed in the netCDF file using zlib compression (default False). The use of this option is deprecated in favor of `compression='zlib'`. Although `compression` is considered a valid encoding option by Xarray https://github.com/pydata/xarray/blob/bbe63ab657e9cb16a7cbbf6338a8606676ddd7b0/xarray/backends/netCDF4_.py#L232-L242 ...it appears that we silently ignores the `compression` option when creating new netCDF4 variables: https://github.com/pydata/xarray/blob/bbe63ab657e9cb16a7cbbf6338a8606676ddd7b0/xarray/backends/netCDF4_.py#L488-L501 ### Code example ```python shape = (10, 20) chunksizes = (1, 10) encoding = { 'compression': 'zlib', 'shuffle': True, 'complevel': 8, 'fletcher32': False, 'contiguous': False, 'chunksizes': chunksizes } da = xr.DataArray( data=np.random.rand(*shape), dims=['y', 'x'], name="foo", attrs={"bar": "baz"} ) da.encoding = encoding ds = da.to_dataset() fname = "test.nc" ds.to_netcdf(fname, engine="netcdf4", mode="w") with xr.open_dataset(fname, engine="netcdf4") as ds1: display(ds1.foo.encoding) ``` ``` {'zlib': False, 'szip': False, 'zstd': False, 'bzip2': False, 'blosc': False, 'shuffle': False, 'complevel': 0, 'fletcher32': False, 'contiguous': False, 'chunksizes': (1, 10), 'source': 'test.nc', 'original_shape': (10, 20), 'dtype': dtype('float64'), '_FillValue': nan} ``` In addition to showing that `compression` is ignored, this also reveals several other encoding options that are not available when writing data from xarray (`szip`, `zstd`, `bzip2`, `blosc`). ### Proposal We should align with the recommendation from the netcdf4 docs and support `compression=` style encoding in NetCDF. We should deprecate `zlib=True` syntax.

Attention! https://support.hdfgroup.org is the NEW home for documentation from The HDF Group. (Details)

HDF5 DIAG Error MPI