Hi,
Being new to hdf5, I am not sure whether it is possible to store around 1 million small datasets within one hdf5 file. However, in order to find out, I have tried writing 1 million datasets containing an array of 100 integers each to an hdf5 file using the H5Dcompact property setting. The problem is that writing the datasets seems to grind to a halt at around 720000. Is there a way of speeding up the saving of such a large amount of datasets? I was also wondering if there is a way to bypass the constant dataset dataset create/write operations and somehow create the datasets in memory before finally writing them to the hdf5 file in one write operation. I would be very grateful for any ideas!
Thanks!
Patrick
···
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
Patrick,
Did you use 1.6.8 version? If so, please try 1.8.2 (we are coming with 1.8.3 in a week a so).
Elena
···
On May 3, 2009, at 9:12 AM, Patrick wrote:
Hi,
Being new to hdf5, I am not sure whether it is possible to store around 1 million small datasets within one hdf5 file. However, in order to find out, I have tried writing 1 million datasets containing an array of 100 integers each to an hdf5 file using the H5Dcompact property setting. The problem is that writing the datasets seems to grind to a halt at around 720000. Is there a way of speeding up the saving of such a large amount of datasets? I was also wondering if there is a way to bypass the constant dataset dataset create/write operations and somehow create the datasets in memory before finally writing them to the hdf5 file in one write operation. I would be very grateful for any ideas!
Thanks!
Patrick
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
I failed to mention that one has to use the latest file format by using the H5Pset_libver_bounds function. Use H5F_LIBVER_LATEST for both "low" and "high" parameters to set up the access property list before creating the file.
Elena
···
On May 3, 2009, at 2:32 PM, Elena Pourmal wrote:
Patrick,
Did you use 1.6.8 version? If so, please try 1.8.2 (we are coming with 1.8.3 in a week a so).
Elena
On May 3, 2009, at 9:12 AM, Patrick wrote:
Hi,
Being new to hdf5, I am not sure whether it is possible to store around 1 million small datasets within one hdf5 file. However, in order to find out, I have tried writing 1 million datasets containing an array of 100 integers each to an hdf5 file using the H5Dcompact property setting. The problem is that writing the datasets seems to grind to a halt at around 720000. Is there a way of speeding up the saving of such a large amount of datasets? I was also wondering if there is a way to bypass the constant dataset dataset create/write operations and somehow create the datasets in memory before finally writing them to the hdf5 file in one write operation. I would be very grateful for any ideas!
Thanks!
Patrick
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
