Best way to write using HDF5 and Table performance

Hello,

I would like to know how HDF5 uses "cache" or "delayed writting". For
instance, if I use a Table and add data using H5TBappend_records, should I
add one record at a time or should I buffer some more data to add more than
one record at once ? Which one is best ?
Does HDF5 buffer data before writting it to disk ?

Also, is Table efficient ? I have a lot of data to add every seconds, each
in a separate Table (about 5000 sample every 50ms, with one sample per
Table).
Is it best to use Table or should I use a lower API (for instance to keep
the Dataset open and avoid open/close each time) ?

Thank you for your input,
Regards,
Guillaume.

···

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Best-way-to-write-using-HDF5-and-Table-performance-tp3292733p3292733.html
Sent from the hdf-forum mailing list archive at Nabble.com.

Hi Guillaume,

Hello,

I would like to know how HDF5 uses "cache" or "delayed writing".

  Yes, there's an internal cache for the "chunks" that are created for datasets.

For
instance, if I use a Table and add data using H5TBappend_records, should I
add one record at a time or should I buffer some more data to add more than
one record at once ? Which one is best ?
Does HDF5 buffer data before writting it to disk ?

Also, is Table efficient ? I have a lot of data to add every seconds, each
in a separate Table (about 5000 sample every 50ms, with one sample per
Table).
Is it best to use Table or should I use a lower API (for instance to keep
the Dataset open and avoid open/close each time) ?

  There's a lot of variability here, given your application and the system you are running on. I would recommend writing a few simple benchmarks that mimic [as closely as possible] how your application will work and see what the numbers you get back look like.

  Quincey

···

On Aug 29, 2011, at 8:48 AM, guillaume wrote: