Hello,
the code for handling external file reads and writes currently does not handle the case when the read() or write() operation returns a number smaller than the requested amount.
Linux currently transfers at most 0x7ffff000 bytes per read() or write(), meaning that if more than 2GB are read from an external file only 2GB are read and the rest is filled up with zeros (because HDF5 thinks that it has reached EOF).
http://man7.org/linux/man-pages/man2/read.2.html#NOTES
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=758170#10
(Note that the problem does not occur if the files datatype and the memory datatype do not match, because in this case HDF5 will use smaller reads into a buffer for converting the values.)
I've attached a patch which will restart the read() or write() operation if a number smaller than the requested amount is returned (similar to e.g. the code in H5FDcore.c). I've also added logic which will restart the syscall after EINTR.
Best regards,
Steffen Kieß
efl-short-read.patch (1.95 KB)