Read performance of Packet Table HL API.

Eryk · May 5, 2009, 2:34pm

Dear mailing list readers,

Why on earth the read performance for Packet Table is so much worse
than for write and how to improve it???

Writing 10k vectors of length 150k
107 s 1.78333 min
Number of packets is : 10000
Reading 10k vectors of length 150k
209 s 3.48333 min

I would have expected that reading is much faster than writing,
however in case of PT it takes twice as much time as writing!

File access properties used for read and write are exactly the same:

hid_t caccprop = H5Pcreate(H5P_FILE_ACCESS);
err = H5Pset_cache(caccprop, 1 , 1024 , 256*160000*4 , 1.0 ) ;
hid_t fileID = H5Fopen("PTcppexampleFL.h5", H5F_ACC_RDONLY, caccprop);

Packet table definition is as follows.

hsize_t dims[] = {150000};
hid_t array_type = H5Tarray_create2(H5T_NATIVE_INT , 1, dims );
/* Create a new fixed-length packet table without compression. */
hid_t ptable = H5PTcreate_fl(fileID, "/examplePacketTable",
array_type, 256 , -1);

Eryk

···

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Ger_van_Diepen · May 6, 2009, 6:18am

Timing writes is only meaningful if an fsync is done before closing the file. Otherwise the data are still in the file system cache and you merely measure the time to copy data to the file system cache. I don't know if HDF5 is doing an fsync.
So in your case where you write 1.5 GB, I can imagine that a considerable part of it is still in the file system cache when closing the file (depending on your memory size).
Reading will always need to read the entire file.

Cheers,
Ger

W Eryk Wolski <wewolski@gmail.com> 05/05/09 4:34 PM >>>