I have a 2-TB of HDF5 files each 50GB to 500GB in size that are hosted on S3 bucket.
I came to know that Spark Connector doesn’t work with S3. I don’t have enough hardware resources to copy this data locally.
So, I want to know what my storage options are to work with Spark Connector?
Also, would like to know if Spark Connector can read netCDF files.
Appreciate your help!