Simple hyperslab selection - doing something wrong, but what?

Hi all,

I am baffled by this problem, no idea what I am doing wrong.

I need to read from 3D datasets which looks like this:

I am interested in accessing one column at a time, so I tried a lot of things along this line:

short data[251]; // short really is 16-bits signed
hsize_t count[3]; count[0] = 1; count[1] = 1; count[2] = 251;
hsize_t offs[3]; offs[0] = offs[1] = offs[2] = 0;
hsize_t stride[3]; stride[0] = stride[1] = stride[2] = 1;

int nrtoread = 251;
filedataset.selectHyperslab( H5S_SELECT_SET, count, offs, stride );
H5::DataSpace inputdataspace = filedataset.getSpace();
H5::DataSpace outputdataspace( 1, &nrtoread );
dataset_->read( data, H5::PredType::STD_I16LE, outputdataspace, inputdataspace );

I also tried reading into a 3D array [1][1][251] and various other ways, but I consistently get this data in the array:

3786 0 0 3555 0 0 3820 …

Thus, I get the right values into the right spots in the array but every time 2 values are not filled.

Any suggestions?

Hello,

I noticed that your usage pattern fits into of H5CPP an easy to use MIT licensed c++ template library.

While h5cpp is under heavy development, it is fully functional, profile, well documented, furnihed with examples and supports major linalg libraries.

Reading a cube into armadillo cube: arma:: cube is one step process, see armadillo example and instead arma::Mat use arma::Cube

The webpage is here, while the github is here.

Best wishes,

Steven

Hi Steven,

Thank you for the suggestion; you wrote a real useful library there. Unfortunately I have dumbed down my problem just for the sake of getting a simple answer. I also need to support not just slices in all directions, but also doing that with steps (‘strides’ in HDF-speak). What makes me despair is that for some reason the simplest of all (reading the fastest moving dimension simply into a single linear array) fails and I am clueless as to why.

With your experience you may be able to see what exactly is the wrong thinking turn I am making in my head. I feel that if I understand why my example does not work, that I can easily extrapolate to reading with steps, without inserting another library - however nice it may be.

Bert,

What are the actual dimensions, what is the available memory of the system you’re running it on, what engineering constraints do you have? Can you bang it into a simpler problem, solve it there?

if the cube fits into the computer’s memory it is really a good idea to read it one shot into an arma::cube and take it from there. If it doesn’t here is a partial read that updates a memory region, the initializer list arguments control the start and count. Stride will be added in the next revision, and honestly in the past years I could just do without strides. [just do it subsetting the array, the sieving mechanism does the same]

here is a solution with h5cpp, If the rows/cols/slices don’t match just swap them around.

{ // code block

double* pd = static_cast<double*>( calloc(ROW_SIZE,sizeof(double)) );

hid_t fd = h5::open(“your_datafile.h5”, H5F_ACC_RDONLY);

hid_t ds = h5::open(fd,“your_dataset”);

// reading data by columns

for( int i=data_rows ) {

h5::read(ds, pd, {0,i,0}, {COL_SIZE, 1, 1} ); // start: {rows,cols,slice} and count:{row,cols,slice}

// do the sieving/strides here, in most cases can be as fast as the stride option

}

H5Dclose(ds);

H5Fclose(fd)

}

ps.

here is a link that helps with posting code. Also, if you coded with C API then you address more people, greater chance of being helped. I am not familiar with the posted C++ wrap.

best

steve

The dimensions are in the screen shot I added. I don’t really think they are relevant further.

But no, the cubes I handle usually do NOT fit entirely in memory. The point is I need to process part of the cube (which can be huge, up to Terabytes). I’m trying to get the full last dimension, and the key is that the input data space is 3D, I request 1/1/251 and that into a 1-D output with size 251. The question is: what is wrong with that? There must be something wrong with it but as far as I can see in all examples I can find this should simply work.

You are saying:

h5::read(ds, pd, {0,i,0}, {COL_SIZE, 1, 1} ); // start: {rows,cols,slice} and count:{row,cols,slice}

{0,i,0} agree, but {COL_SIZE, 1, 1} is exactly mirrored, it should be {1,1,COL_SIZE}. If I try your suggestion I get an Abort, which makes sense.

So, as you can see in the screen shot and in the code I’m not even trying to fetch a specific column, just the first one. But the read() is skipping 2 out of 3 values. Why??