HDF5 benchmarks

Hdf-forum archives
1

Is there an established HDF5 benchmark suite eg for measuring file IO including for parallel file systems?
Many thanks, Michael

···

--
Dr. Michael Bane
Snr Research Apps & Collab Consultant
IT Services
The University of Manchester
M13 9PL
http://www.manchester.ac.uk/rac
@mkbane_mcr

2

h5perf, included in hdf5, is a good start. Note that the access
pattern may not match what an actual application is trying to do.
h5perf has a ton of features, though, so you can probably find a
configuration that comes close.

==rob

···

On Wed, Jan 18, 2012 at 12:51:47PM +0000, michael bane wrote:

Is there an established HDF5 benchmark suite eg for measuring file IO including for parallel file systems?

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

3

Rob
Thanks for that. We've no parallel filesys so HDF5 build serially only
(ie on top of NFS), so have tried h5perf_serial. Just wanted to check
whether the attachments made sense, one is on the login node and other
on a compute node. The data was compiled from running:

#!/bin/bash

# script to run series of HDF5 benchmarks
# requires HDF5 module to be pre-loaded

EXE=h5perf_serial
RW=-w
API="-A HDF5"
##ITERS="-i1000"

(
for size in 20 40 60 80 100 200 400 600 800 1000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i1000
done

for size in 1000 1200 1400 1600 1800 2000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i500
done

for size in 1000 2000 3000 4000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i100
done

) 2>&1 | tee h5perf_serial_$HOSTNAME.txt

···

On 23 Jan 2012, at 16:54, Rob Latham wrote:

On Wed, Jan 18, 2012 at 12:51:47PM +0000, michael bane wrote:
> Is there an established HDF5 benchmark suite eg for measuring file
> IO including for parallel file systems?

h5perf, included in hdf5, is a good start. Note that the access
pattern may not match what an actual application is trying to do.
h5perf has a ton of features, though, so you can probably find a
configuration that comes close.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

No thoughts on that big peak? (Is it even big?!)
Thanks, M

NB:
X axis is num elements per dimension, 2 dimensions
Y axis is reported MB/s average reported

···

On 31 Jan 2012, at 23:22, michael wrote:

Rob
Thanks for that. We've no parallel filesys so HDF5 build serially only
(ie on top of NFS), so have tried h5perf_serial. Just wanted to check
whether the attachments made sense, one is on the login node and other
on a compute node. The data was compiled from running:

#!/bin/bash

# script to run series of HDF5 benchmarks
# requires HDF5 module to be pre-loaded

EXE=h5perf_serial
RW=-w
API="-A HDF5"
##ITERS="-i1000"

(
for size in 20 40 60 80 100 200 400 600 800 1000; do
time ${EXE} ${RW} ${API} -e ${size},${size} -i1000
done

for size in 1000 1200 1400 1600 1800 2000; do
time ${EXE} ${RW} ${API} -e ${size},${size} -i500
done

for size in 1000 2000 3000 4000; do
time ${EXE} ${RW} ${API} -e ${size},${size} -i100
done

) 2>&1 | tee h5perf_serial_$HOSTNAME.txt

On 23 Jan 2012, at 16:54, Rob Latham wrote:

On Wed, Jan 18, 2012 at 12:51:47PM +0000, michael bane wrote:

Is there an established HDF5 benchmark suite eg for measuring file
IO including for parallel file systems?

h5perf, included in hdf5, is a good start. Note that the access
pattern may not match what an actual application is trying to do.
h5perf has a ton of features, though, so you can probably find a
configuration that comes close.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

<h5perf_R3-Atmos-08.png><h5perf_login.png>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

5

I've written a performance test for writing data in a chunked data set
and reading back the data in various traversal orders and with various
sizes (e.g. per line, plane).

Cheers,
Ger

"Michael Bane" 02/01/12 1:33 AM >>>

X axis is num elements per dimension, 2 dimensions

Y axis is reported MB/s average reported

···

___________
Michael Bane
IT Services, UoM

----- Reply message -----
From: "michael"
Date: Tue, Jan 31, 2012 23:22
Subject: [Hdf-forum] HDF5 benchmarks
To: "HDF Users Discussion List"

Rob
Thanks for that. We've no parallel filesys so HDF5 build serially only
(ie on top of NFS), so have tried h5perf_serial. Just wanted to check
whether the attachments made sense, one is on the login node and other
on a compute node. The data was compiled from running:

#!/bin/bash

# script to run series of HDF5 benchmarks
# requires HDF5 module to be pre-loaded

EXE=h5perf_serial
RW=-w
API="-A HDF5"
##ITERS="-i1000"

(
for size in 20 40 60 80 100 200 400 600 800 1000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i1000
done

for size in 1000 1200 1400 1600 1800 2000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i500
done

for size in 1000 2000 3000 4000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i100
done

) 2>&1 | tee h5perf_serial_$HOSTNAME.txt

On 23 Jan 2012, at 16:54, Rob Latham wrote:

On Wed, Jan 18, 2012 at 12:51:47PM +0000, michael bane wrote:
> Is there an established HDF5 benchmark suite eg for measuring file
> IO including for parallel file systems?

h5perf, included in hdf5, is a good start. Note that the access
pattern may not match what an actual application is trying to do.
h5perf has a ton of features, though, so you can probably find a
configuration that comes close.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

6

X axis is num elements per dimension, 2 dimensions

Y axis is reported MB/s average reported

···

___________
Michael Bane
IT Services, UoM

----- Reply message -----
From: "michael" <michael.bane@manchester.ac.uk>
Date: Tue, Jan 31, 2012 23:22
Subject: [Hdf-forum] HDF5 benchmarks
To: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>

Rob
Thanks for that. We've no parallel filesys so HDF5 build serially only
(ie on top of NFS), so have tried h5perf_serial. Just wanted to check
whether the attachments made sense, one is on the login node and other
on a compute node. The data was compiled from running:

#!/bin/bash

# script to run series of HDF5 benchmarks
# requires HDF5 module to be pre-loaded

EXE=h5perf_serial
RW=-w
API="-A HDF5"
##ITERS="-i1000"

(
for size in 20 40 60 80 100 200 400 600 800 1000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i1000
done

for size in 1000 1200 1400 1600 1800 2000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i500
done

for size in 1000 2000 3000 4000; do
  time ${EXE} ${RW} ${API} -e ${size},${size} -i100
done

) 2>&1 | tee h5perf_serial_$HOSTNAME.txt

On 23 Jan 2012, at 16:54, Rob Latham wrote:

On Wed, Jan 18, 2012 at 12:51:47PM +0000, michael bane wrote:
> Is there an established HDF5 benchmark suite eg for measuring file
> IO including for parallel file systems?

h5perf, included in hdf5, is a good start. Note that the access
pattern may not match what an actual application is trying to do.
h5perf has a ton of features, though, so you can probably find a
configuration that comes close.

==rob

--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org