Hi all,
We are trying out the new S3 bucket access functionality in HDF5 1.10.6 using the ROS3 virtual filesystem driver.
On Windows (only; on Linux everything is fine), we observe that (at least the way we’ve built it and the underlying libcurl and libopenssl) the HDF5 library’s ROS3 driver is only able to access resources over HTTP, but not over HTTPS. This is of course a bit problematic for accessing any non-public S3 bucket!
For example, if I try running h5ls against an example HTTPS URL provided by HDF5 folks:
D:\hdf5-1.10.6\bin64\h5ls -v --enable-error-stack --vfd=ros3 https://s3.us-east-2.amazonaws.com/hdf5ros3/GMODO-SVM01.h5
then I receive a series of errors (reproduced below).
When the exact same command is run except having “https:” replaced by “http:” in the URL, all works fine, so it is the SSL transport that is at issue.
Of the errors, the main issue appears to be:
“Peer certificate cannot be authenticated with given CA certificates”
Looking further into it, it is my understanding that on Windows, libcurl needs to be told where to look for a CA certificate bundle, by passing the bundle filename into curl_easy_setopt() with the CURLOPT_CAINFO flag. See for instance [1], [2].
[1] https://curl.haxx.se/libcurl/c/CURLOPT_CAINFO.html
[2] https://github.com/curl/curl/issues/1538
However, the HDF5 API appears to have no way to pass this information to libcurl:
A) There is no usage of this flag anywhere in H5FDs3comms.c; and as far as I can see,
B) there appears to be no way for client code to obtain the libcurl CURL* handle that gets used in the HDF5 library under the hood by an HDF5 “file” hid_t that was opened using an FAPL with the ROS3 VFD in effect. So client code has no way to set that libcurl option by itself, either.
Note that other libraries that use libcurl as a backend do provide a way to pass this information to libcurl. For instance, libGDAL may be asked to do so either by the end-user, by setting the $CURL_CA_BUNDLE environment variable; or by client code, by calling
CPLSetConfigOption(“CURL_CA_BUNDLE”, filename);
Am I missing something, or is this an oversight in the new ROS3 driver?
Thanks in advance for your reply,
Kevin
For reference, below is the stack trace I obtain by running h5ls against an HTTPS URL, after rebuilding HDF5 1.10.6 having re-#define’d S3COMMS_DEBUG and S3COMMS_CURL_VERBOSITY to 1 in H5FDs3comms.c:
> D:\hdf5-1.10.6\bin64\h5ls -v --enable-error-stack --vfd=ros3 https://s3.us-east-2.amazonaws.com/hdf5ros3/GMODO-SVM01.h5
called H5FD_s3comms_s3r_open.
called H5FD_s3comms_parse_url.
called H5FD_s3comms_s3r_getsize.
called H5FD_s3comms_s3r_read.
CURL ERROR CODE: 60
HTTP CODE: 0
Peer certificate cannot be authenticated with given CA certificates
called H5FD_s3comms_free_purl.
HDF5-DIAG: Error detected in HDF5 (1.10.6) thread 0:
#000: D:\users\kevin\build\hdf5-1.10.6\src\H5F.c line 509 in H5Fopen(): unable to open file
major: File accessibilty
minor: Unable to open file
#001: D:\users\kevin\build\hdf5-1.10.6\src\H5Fint.c line 1498 in H5F_open(): unable to open file: time = Fri Feb 14 07:11:59 2020
, name = 'https://s3.us-east-2.amazonaws.com/hdf5ros3/GMODO-SVM01.h5', tent_flags = 0
major: File accessibilty
minor: Unable to open file
#002: D:\users\kevin\build\hdf5-1.10.6\src\H5FD.c line 734 in H5FD_open(): open failed
major: Virtual File Layer
minor: Unable to initialize object
#003: D:\users\kevin\build\hdf5-1.10.6\src\H5FDros3.c line 842 in H5FD_ros3_open(): could not open
major: Virtual File Layer
minor: Unable to open file
#004: D:\users\kevin\build\hdf5-1.10.6\src\H5FDs3comms.c line 1329 in H5FD_s3comms_s3r_open(): problem in H5FD_s3comms_s3r_getsize.
major: Invalid arguments to routine
minor: Bad value
#005: D:\users\kevin\build\hdf5-1.10.6\src\H5FDs3comms.c line 1022 in H5FD_s3comms_s3r_getsize(): problem in reading during getsize.
major: Invalid arguments to routine
minor: Bad value
#006: D:\users\kevin\build\hdf5-1.10.6\src\H5FDs3comms.c line 1836 in H5FD_s3comms_s3r_read(): problem while performing request.
major: Virtual File Layer
minor: Unable to open file