I am currently trying to understand a problem I'm having where no data is being written to a data set. I think the problem is due to how I have my data spaces and selections set up, but I'm not sure. Here's the situation:
The application is being fed data in near real-time. There really isn't any a priori knowledge of the contents of the data. As the data arrives, its location within the data set is determined and added to the hyperslab selection. The end result could be a single hyperslab block, or it could be many. As an example, the file selection might be something like:
0: <1024797, 0, 0>, <1024799, 0, 1>
1: <1024797, 0, 3>, <1024799, 0, 5>
2: <1024797, 0, 7>, <1024799, 0, 7>
...and so on. The file is chunked in blocks of 10 in the 1st dimesion, and the entirety of the 2nd and 3rd dimensions. The application is designed to write "whole chunks", so every 10 are written together (most of the time that 1st dimension will be monotonically increasing, but not always).
The data is buffered before writing. The position in the buffer is chosen as if the entire block is selected (in this case, it'd be something like <1024790,0,0>,<1024799,0,119>). Originally, we were reading the block off disk before writing, but this seemed to result in a lot of overhead.
It appears to me that the buffer for the selection example above should consist of a total of 18 values in sequence as if each hyperslab block were the entirety of the selection... that is, element 0 in the buffer array would correspond to 1024797,0,0 and element 17 would be 1024799,0,7. Is that correct? That's not how the program is currently storing data in its buffer, and I think that's why I'm not seeing any real data on disk.
Is there a sensible way to write data in this fashion (select, semi-random elements within the data set)? If my statement above is true, it will be pretty difficult to manage the buffer as subsequent selections could radically change the required organization of the buffer, and therefore require significant shuffling of data in memory. I'm not keen on that.