H5Sselect_elements works very slowly

Hdf-forum archives
1

Hello,

I'm learning HDF API and I try to use the H5Sselect_elements function in
order to write ramdom 4D point data.
My dataset is 100x100x25x1024 (the last dimension is extendible to
H5S_UNLIMITED) with a chunk size of 25.

···

------------------------------------------------------------------
// Selected point data space
hsize_t memArrayDim = DS3D_WRITE_BUFFER_SIZE;
hid_t memArray_s_id = H5Screate_simple( 1, &memArrayDim, NULL );
hid_t ds3d_select_s_id = H5Screate_simple( ds3dDSRank, ds3dDims, NULL );
H5Sselect_elements( ds3d_select_s_id, H5S_SELECT_SET,
DS3D_WRITE_BUFFER_SIZE, ( hsize_t* )coords );

// Write the new index into the right position
H5Dwrite( ds3d_current_d_id, H5T_NATIVE_INT, memArray_s_id,
ds3d_select_s_id, H5P_DEFAULT, tubeIndexValues );
------------------------------------------------------------------

I tried here to commit "DS3D_WRITE_BUFFER_SIZE" values at a time.
"coords" contains an array of 4D point coordinates,
"tubeIndexValues" contains the actual values

The main constraint is that I can't store the whole dataset in RAM to make a
single write call and I have to write data ramdomly (no other way I'm
affraid), point by point.

The H5Dwrite function works fine, but it takes about 22 seconds to store 2
data (DS3D_WRITE_BUFFER_SIZE = 2). The time consumption seems to be linear
with DS3D_WRITE_BUFFER_SIZE...

Is that normal ? Is there a mean to write my random data quickly ?

Thanks,

Yves

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/H5Sselect-elements-works-very-slowly-tp3375637p3375637.html
Sent from the hdf-forum mailing list archive at Nabble.com.

2

Any idea concerning this thread ?

Yves

···

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/H5Sselect-elements-works-very-slowly-tp3375637p3399790.html
Sent from the hdf-forum mailing list archive at Nabble.com.

3

Hi Yves,

···

On Oct 6, 2011, at 9:14 AM, rogez wrote:

Any idea concerning this thread ?

  I believe I've got a patch to address this and I'm trying to get it reviewed and checked in for the upcoming 1.8.8 release (in November).

  Quincey

4

Thank you Quincey,

Great! I will keep an eye on new releases.
Otherwise, I thought about it without having an assessment and I could maybe
use some hyperslab selection and write my elements by packets of about 25
elements... To be validated on my side.

Yves

···

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/H5Sselect-elements-works-very-slowly-tp3375637p3408813.html
Sent from the hdf-forum mailing list archive at Nabble.com.

5

Hi Yves,
  I haven't forgotten about you - I've just been delayed in getting this change checked into the library. Hopefully, I'll get this done before the end of December and will get a snapshot to you.

  Quincey

···

On Oct 10, 2011, at 2:11 AM, rogez wrote:

Thank you Quincey,

Great! I will keep an eye on new releases.
Otherwise, I thought about it without having an assessment and I could maybe
use some hyperslab selection and write my elements by packets of about 25
elements... To be validated on my side.

Yves

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/H5Sselect-elements-works-very-slowly-tp3375637p3408813.html
Sent from the hdf-forum mailing list archive at Nabble.com.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

6

Thank you very much, Quincey.
Anyway, I was working on other aspects of our simulation software these last months. I will get into HDF again in early January. Thus, your timing is fine :wink: !

Yves Rogez

*IPAG*
/Institut de Planétologie et d'Astrophysique de Grenoble /
Bat D de Physique - BP. 53 - 38041 Grenoble - FRANCE

tel : +33 (0)4 76 63 52 80
lab : +33 (0)4 76 63 52 89

···

Le jeudi 1 décembre 2011 00:38:11, Quincey Koziol a écrit :

Hi Yves,
  I haven't forgotten about you - I've just been delayed in getting this change checked into the library. Hopefully, I'll get this done before the end of December and will get a snapshot to you.

  Quincey

On Oct 10, 2011, at 2:11 AM, rogez wrote:

Thank you Quincey,

Great! I will keep an eye on new releases.
Otherwise, I thought about it without having an assessment and I could maybe
use some hyperslab selection and write my elements by packets of about 25
elements... To be validated on my side.

Yves

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/H5Sselect-elements-works-very-slowly-tp3375637p3408813.html
Sent from the hdf-forum mailing list archive at Nabble.com.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org