I am trying to measure the I/O speed programmatically from our application. I've tried the naïve approach of periodically getting the file size and calculating the growth like this (simplified):
This produces good clean results on Windows but running the same program on Linux produces very "jerky" results.
The file size jumps by about 300MB every minute or so, but the reported free space is always constant (and very small, about 2 KB), resulting in the reported I/O speed of 0 most of the time, jumping to over 100MB/s once a minute or so.
I could understand that there may be some caching going on and the cache is dumped from time to time. However running iotop on the same machine proves that our application writes to the file all the time at a steady rate, suggesting that the file size increase jump should be reflected as free space.
So why does getFreeSpace() keep returning a constant value?
Is there a better way of measuring the I/O speed?
I am trying to measure the I/O speed programmatically from our application. I've tried the na�ve approach of periodically getting the file size and calculating the growth like this (simplified):
Are you interested in the speed of your storage system, or are you
interested in how fast your application can move data?
What you have done will measure the former, kind of.
What you really want to do is
- compute how much data application passes to or receives from HDF5 routine
- time how long routine takes to finish
Maybe you get really fancy and include open and close time in addition
to data transfer time.
Now you've got a metric that actually relates to application behavior.
==rob
···
On Fri, Aug 16, 2013 at 04:52:13PM +0100, Peter Pichler wrote:
This produces good clean results on Windows but running the same program on Linux produces very "jerky" results.
The file size jumps by about 300MB every minute or so, but the reported free space is always constant (and very small, about 2 KB), resulting in the reported I/O speed of 0 most of the time, jumping to over 100MB/s once a minute or so.
I could understand that there may be some caching going on and the cache is dumped from time to time. However running iotop on the same machine proves that our application writes to the file all the time at a steady rate, suggesting that the file size increase jump should be reflected as free space.
So why does getFreeSpace() keep returning a constant value?
Is there a better way of measuring the I/O speed?
I have similar questions as yours and would be interested in running benchmarks to define what would be the best chunk size to use to optimize performances of my application.
I also would like to compare performances with a bare binary file write operation in order to measure the overhead (if any) of the HDF5 API.
I guess this is a common question and I'm wondering if the HDF Group has any plan to add hdf5 examples focused on benchmarking and optimizing applications ?
I'm still in the learning curve and evaluation of HDF5 API and do not have answer to your question but will follow this thread with great interest !
I am trying to measure the I/O speed programmatically from our application. I've tried the naïve approach of periodically getting the file size and calculating the growth like this (simplified):
This produces good clean results on Windows but running the same program on Linux produces very "jerky" results.
The file size jumps by about 300MB every minute or so, but the reported free space is always constant (and very small, about 2 KB), resulting in the reported I/O speed of 0 most of the time, jumping to over 100MB/s once a minute or so.
I could understand that there may be some caching going on and the cache is dumped from time to time. However running iotop on the same machine proves that our application writes to the file all the time at a steady rate, suggesting that the file size increase jump should be reflected as free space.
So why does getFreeSpace() keep returning a constant value?
Is there a better way of measuring the I/O speed?
>
[ (pas de nom de fichier) (0.2 Ko) ]
Une messagerie gratuite, garantie à vie et des services en plus, ça vous tente ?
Je crée ma boîte mail www.laposte.net
Are you interested in the speed of your storage system, or are you interested in how fast your application can move data?
The former. We have metrics for the latter (well, sort of) but want to benchmark the storage system, particularly when the storage is on a network drive.
What you have done will measure the former, kind of.
That's what I assumed, but it does not work reliably as I mentioned in my first post (file growing in large chunks, empty space remaining constant...).
What you really want to do is
- compute how much data application passes to or receives from HDF5 routine
- time how long routine takes to finish
I don't really need that kind of metrics. All I need is the overall I/O speed in MB/s, measured at say 10 seconds intervals (that is, how much raw data we have shifted to the disk in the past 10 seconds).
I can try to measure that at the interface that you mentioned. The functions we use are add_group_string_attribute, add_group_real64_attribute, add_group_int64_attribute and friends. To measure the I/O speed at this level I would need to know how calling each of these functions translates to the number of raw bytes written to the file. It such a translation available?
> Are you interested in the speed of your storage system, or are you interested in how fast your application can move data?
The former. We have metrics for the latter (well, sort of) but want to benchmark the storage system, particularly when the storage is on a network drive.
oh. then just use IOR to a POSIX file and keep HDF5 out of it. HDF5
is going to cache things, or maybe write a byte at the very end before
filling in the dataset.
==rob
···
On Fri, Aug 23, 2013 at 11:46:39AM +0100, Peter Pichler wrote:
> What you have done will measure the former, kind of.
That's what I assumed, but it does not work reliably as I mentioned in my first post (file growing in large chunks, empty space remaining constant...).
> What you really want to do is
> - compute how much data application passes to or receives from HDF5 routine
> - time how long routine takes to finish
I don't really need that kind of metrics. All I need is the overall I/O speed in MB/s, measured at say 10 seconds intervals (that is, how much raw data we have shifted to the disk in the past 10 seconds).
I can try to measure that at the interface that you mentioned. The functions we use are add_group_string_attribute, add_group_real64_attribute, add_group_int64_attribute and friends. To measure the I/O speed at this level I would need to know how calling each of these functions translates to the number of raw bytes written to the file. It such a translation available?
just use IOR to a POSIX file and keep HDF5 out of it.
Unfortunately the powers that be want the I/O speed displayed prominently on the main GUI page of our product while collecting the data - which may take hours. I do not have the luxury of writing to another file during that time just to measure the speed, I need to tap on the existing infrastructure. Hence my experiment with getFileSize and getFreeSpace.
