[netcdfgroup] Problem with nc_inq_grps and ncdump reading HDF5file


If you are writing a new format, my guess is that you are writing an API for it, and since I have been writing these High-Level APIs for years (starting with the very own HDF5 High-Level API), and I am writing one now, here's my advice for it :slight_smile:

1) Do not use groups as a means to access your data (more on this later).
2) Encapsulate all HDF5 IDs.

For my case, I am using a model where datasets are accessed for read/write using their full path name, e.g "/path/to/dataset".

So, "/path/to/" is the group name and "dataset" is the dataset relative name.

API usage for read/write is

write( "/path/to/dataset", void * data buffer)

so, the model to read/write is just the full dataset name as a string and a buffer with data.

The group part is implicit in the string, as you can see no HDF5 group IDs or group paths to read/write (that is what I mean with
"not use groups as a means to access your data ")

How can you accomplish this?

Here's an example of functions part of a C++ class , regarding this part

void create_group(const std::string& group_name);
void create_dataset(const std::string& group_path, const std::string & dataset_name, other parameters here like dataset size or chunk size

void write(const std::string& path, const void* buf);

Let's start with the create_group function

Here just create the group using the HDF5 C API (I use the C API even in a C++ program, the HDF5 C++ API adds nothing of use here)

gid = H5Gcreate2(this->m_fid, group_name.c_str(), other parameters

As can be seen the group is created and closed immediately.

This means, you will have no groups open at all during your read/write calls (typically the core part of a program execution).

Since groups are closed immediately, you can even exceed the number of maximum open groups Russ mentioned.

For the dataset creation

create_dataset(const std::string& group_path, const std::string& dataset_name,

concatenate the group path with the dataset name and store it somehow in your C++ class

std::string absolute_dataset_name = group_path + "/" + dataset_name;

In my case , I am using a map with dataset ID/ full path of dataset, because I want to keep the datasets open (in memory), but any other way to store the path will do

//map with dataset name (full path) / HDF5 dataset ID

std::map<std::string, hid_t> m_map_datasets;

here is the HDF5 create call

did = H5Dcreate2(this->m_fid, absolute_dataset_name.c_str()

The trick here is to use the file ID (that must be stored in the class) and the full path. This takes group IDs out of the equation.

For the write/read call use simply

write(const std::string& path, const void* buf)

get the dataset ID from full path

//get dataset ID from map <path/ID>

hid_t did = this->m_map_datasets[path];

use it in the write call

H5Dwrite(did, other parameters

Hope this helps and ask any further questions if needed.

btw, I downloaded your file with only groups, it has a size of 37MB, lots of group metadata here.



Pedro Vicente

  ----- Original Message -----
  From: Russ Rew
  To: Andrew Dowsey
  Cc: netcdfgroup@unidata.ucar.edu
  Sent: Monday, November 17, 2014 2:11 PM
  Subject: Re: [netcdfgroup] Problem with nc_inq_grps and ncdump reading HDF5file

  Hi Andrew,

  You've run across a limitation in the number of simultaneously open groups permitted in netCDF-4. The groups documentation says

    ... Encoding both the open file id and group id in a single integer currently limits the number of groups per netCDF-4 file to no more than 32767. Similarly, the number of simultaneously open netCDF-4 files in one program context is limited to 32767.

  I think both those limits should actually be 65535 (== 2**16 - 1), but in any case, your HDF5 file has 119254 groups, which is too many for netCDF-4 to handle.

  The only workaround I can think of would be to close some groups if you don't need to have them all open simultaneously.


  On Mon, Nov 17, 2014 at 8:14 AM, Andrew Dowsey <andrew.dowsey@manchester.ac.uk> wrote:


    I’m trying to create HDF5 files that can be read by NetCDF4 and I’ve run into a problem in that nc_inq_grps seems to report some bad ids. ncdump bails with this error too. h5dump works fine. The problem is deterministic but I haven’t been able to figure out what causes it because slightly different HDF5 files work fine. I have created a test file that has this problem, which contains nothing but groups. It can be downloaded from http://personalpages.manchester.ac.uk/staff/andrew.dowsey/test.h5

    I am creating a new format for a type of instrument data we use, and for flexibility I would like it to be writeable/readable both by HDF5 and netCDF4 libraries.

    Any insight would be greatly appreciated!

    Kind regards,

    Andrew Dowsey PhD
    Lecturer and CADET Bioinformatics Research Lead
    Institute of Human Development, The University of Manchester

    t: +44 161 701 0244
    f: +44 161 701 0242

    Centre for Advanced Discovery and Experimental Therapeutics (CADET)
    Central Manchester University Hospitals NHS Foundation Trust
    Oxford Road
    Manchester M13 9WL

    netcdfgroup mailing list
    For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/


  netcdfgroup mailing list
  For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/