Hi,
I have a 2-TB of HDF5 files each 50GB to 500GB in size that are hosted on S3 bucket.
I came to know that Spark Connector doesn’t work with S3. I don’t have enough hardware resources to copy this data locally.
So, I want to know what my storage options are to work with Spark Connector?
Also, would like to know if Spark Connector can read netCDF files.
Appreciate your help!
Thank you!
Sulochana.