HDF5 read error when reading huge file

HDF5 Library C++ Users' Group
21

Have another update regarding H5C_MAX_ENTRY_SIZE definition. I experimented with changing its definition to:

#define H5C_MAX_ENTRY_SIZE ((size_t)(64 * 1024 * 1024))

Rebuilt the HDF5 library with this definition and re-ran the writer application. This time, a valid HDF5 file was generated which could be read in. But the generation of ~347G file took about 22hrs which seems quite excessive.

Is this change to H5C_MAX_ENTRY_SIZE something that you would recommend incorporating at least temporarily? A customer of ours is evaluating our flow and would like to have them move forward with the HDF5 write/read.

Like I mentioned in the previous post, I have the stripped-down version of writer code that I can share. Please let me know where to send it.

Thanks
-Kat

22

Hi Gerd,

Could you please share your thoughts on the above 2 posts?

Thanks
-Kat

23

Can you try a newer library version? HDF5 1.12.0 is relatively old and will EOL soon. We fixed several CVE issues that involved the writing of uninitialized data. I’d recommend 1.14.1 (release imminent) or 1.10.10. If you have to stick to 1.12, use 1.12.2.

G.

24

Nothing unusual here. How many fields does your compound have? How long are the (ASCII-only) field names?

G.

25

Yes. (Excessive…) How are you acquiring/writing the data? Element by element?

No. There’s something fishy here; we better figure out what’s happening.

I’ve shared a link.

G.

26

The compound type typically has about 20-30 fields. Field names are fairly small – about 10-20 characters.

Thanks
-Kat

27

All data for a dataset is acquired prior to writing it.

I have uploaded the sample writer program in the link you provided.

Thanks
-Kat.