Dear HDF experts,
I build an application which operates on NetCDF data using Big Data
My design aims at avoiding unnecessarily writing data to disk. Instead, I
want to operate as much as possible in memory. The challenge is data
(de)serialization for distributed communications between computing nodes.
Since NetCDF4 and HDF5 already provide a portable data format, a simple and
efficient design would simply access and then exchange the raw binary data
over the network.
Currently, I fail to access this buffer without creating files. I am
investigating the use of the Apache Common VFS Ram file system to trick
NetCDF into working in memory.
But, a suggestion on the NetCDF Java mailing list (see ticket MQO-415619)
was to build an alternative to the core driver. I feel this is the more
desirable course of actions as it is about improving the existing solutions
instead of working around their limitations.
Do you think this approach is feasible ? Any starting pointers would be