dumping particle trajectories

Dear hierarchical people,

I have currently converted a piece of code from using a simple ascii format for output into using HDF5. What the code does is at every iteration dumping some information about particle energy/trajectory/position to the ascii file (this is a particle tracking code).

Initially I then did the same with the HDF5 library, having a unlimited row dimension in a 2D array and using h5extend_f to extend by one element each time and writing a hyperslab of one row to the file. As some (perhaps most) of you might have guessed or know already, this was a rather bad idea. The file (without compression) was about the same size as the ascii file (but obviously with higher precision), and reading the file in subsequent analysis was at least an order of magnitude slower.

I then realized that I probably needed to write less frequently and rather keeping a semi-large hyperslab in memory. I chose a hyperslab of 1000 rows, but otherwise using the same procedure. This seems to be both fast and with compression creating quite a bit smaller file. I tried even larger slabs, but did not see any speed improvement in my initial testing

My question really was just if there are some recommended ways to do this? I would imagine I am not the first that want to use HDF5 in this way, dumping some data at every iteration of a given simulation, without having to keep it all in memory until the end?

Thanks for all explanations/suggestions/experiences related to this problem you can provide me so I can make the best design choices in my program! :slight_smile:

Cheers,
Yngve

Hi,

  what's the reason for using a 2D extendable dataset instead of a sequence of 1D arrays
in a group, using one group per time step? How many particles and time steps do you
have typically? I assume in your case the number of particles is constant over time?

Cheers,
  Werner

···

On Wed, 16 Mar 2011 03:52:10 -0500, Yngve Inntjore Levinsen <yngve.inntjore.levinsen@cern.ch> wrote:

Dear hierarchical people,

I have currently converted a piece of code from using a simple ascii format for output into using HDF5. What the code does is at every iteration dumping some information about particle energy/trajectory/position to the ascii file (this is a particle tracking code).

Initially I then did the same with the HDF5 library, having a unlimited row dimension in a 2D array and using h5extend_f to extend by one element each time and writing a hyperslab of one row to the file. As some (perhaps most) of you might have guessed or know already, this was a rather bad idea. The file (without compression) was about the same size as the ascii file (but obviously with higher precision), and reading the file in subsequent analysis was at least an order of magnitude slower.

I then realized that I probably needed to write less frequently and rather keeping a semi-large hyperslab in memory. I chose a hyperslab of 1000 rows, but otherwise using the same procedure. This seems to be both fast and with compression creating quite a bit smaller file. I tried even larger slabs, but did not see any speed improvement in my initial testing

My question really was just if there are some recommended ways to do this? I would imagine I am not the first that want to use HDF5 in this way, dumping some data at every iteration of a given simulation, without having to keep it all in memory until the end?

Thanks for all explanations/suggestions/experiences related to this problem you can provide me so I can make the best design choices in my program! :slight_smile:

Cheers,
Yngve

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

A Wednesday 16 March 2011 09:52:10 Yngve Inntjore Levinsen escrigué:

Dear hierarchical people,

I have currently converted a piece of code from using a simple ascii
format for output into using HDF5. What the code does is at every
iteration dumping some information about particle
energy/trajectory/position to the ascii file (this is a particle
tracking code).

Initially I then did the same with the HDF5 library, having a
unlimited row dimension in a 2D array and using h5extend_f to
extend by one element each time and writing a hyperslab of one row
to the file. As some (perhaps most) of you might have guessed or
know already, this was a rather bad idea. The file (without
compression) was about the same size as the ascii file (but
obviously with higher precision), and reading the file in subsequent
analysis was at least an order of magnitude slower.

I then realized that I probably needed to write less frequently and
rather keeping a semi-large hyperslab in memory. I chose a hyperslab
of 1000 rows, but otherwise using the same procedure. This seems to
be both fast and with compression creating quite a bit smaller file.
I tried even larger slabs, but did not see any speed improvement in
my initial testing

My question really was just if there are some recommended ways to do
this? I would imagine I am not the first that want to use HDF5 in
this way, dumping some data at every iteration of a given
simulation, without having to keep it all in memory until the end?

For getting a good performance is very important your chunksize.
Typical figures for serial I/O are between 32 KB and 1 MB, depending the
final size of the dataset. Which one you are using?

···

--
Francesc Alted

Yes of course Francesc, I was thinking float = half of 64bit instead of 4x 8bit :slight_smile:
I was thinking that it might be beneficial to keep the size in powers of 2, so that is why I chose 1024 and not 1000. I keep it as a variable so I can easily change it.

Werner, I was thinking that I should eventually move to a sequence of 1D arrays, but it requires slightly more rewriting. The number of lines I have to write depends on whether or not the particle is still alive. I am starting out with an equal amount of particles, but have no means to know if I need to write the position of a given particle 0 times or one million times. Typically I have something like 1 million timesteps, but I do not write down trajectories all the time (when is dependent on the Monte Carlo so no way to know in advance)

Ideally I would've written all analysis into the code itself so I didn't have to write the trajectories all the time (I have not made this choice!), but that requires too much work for me to handle at the moment. Using HDF5 will reduce the storage space needed by about a factor 6 from my estimates, improve precision, and significantly reduce CPU hours needed as well. This is already a great improvement!

Cheers,
Yngve

···

On Wednesday 16 March 2011 02:09:36 PM Werner Benger wrote:

Hi,

  what's the reason for using a 2D extendable dataset instead of a sequence
of 1D arrays
in a group, using one group per time step? How many particles and time
steps do you
have typically? I assume in your case the number of particles is constant
over time?

Cheers,
  Werner

On Wed, 16 Mar 2011 03:52:10 -0500, Yngve Inntjore Levinsen > <yngve.inntjore.levinsen@cern.ch> wrote:

> Dear hierarchical people,
>
> I have currently converted a piece of code from using a simple ascii
> format for output into using HDF5. What the code does is at every
> iteration dumping some information about particle
> energy/trajectory/position to the ascii file (this is a particle
> tracking code).
>
> Initially I then did the same with the HDF5 library, having a unlimited
> row dimension in a 2D array and using h5extend_f to extend by one
> element each time and writing a hyperslab of one row to the file. As
> some (perhaps most) of you might have guessed or know already, this was
> a rather bad idea. The file (without compression) was about the same
> size as the ascii file (but obviously with higher precision), and
> reading the file in subsequent analysis was at least an order of
> magnitude slower.
>
> I then realized that I probably needed to write less frequently and
> rather keeping a semi-large hyperslab in memory. I chose a hyperslab of
> 1000 rows, but otherwise using the same procedure. This seems to be both
> fast and with compression creating quite a bit smaller file. I tried
> even larger slabs, but did not see any speed improvement in my initial
> testing
>
> My question really was just if there are some recommended ways to do
> this? I would imagine I am not the first that want to use HDF5 in this
> way, dumping some data at every iteration of a given simulation, without
> having to keep it all in memory until the end?
>
> Thanks for all explanations/suggestions/experiences related to this
> problem you can provide me so I can make the best design choices in my
> program! :slight_smile:
>
> Cheers,
> Yngve

Yngve,

  especially if the number of particles might change over time, using 1D
arrays might be more appropriate, possibly combined with index lookup arrays
that allows to identify particles at T0 to T1 and nice versa. I'm using
such a 1D layout for particles and particle trajectories as part of my
F5 library, here is a coding example on how to write particle positions
with some fields given on them (it's all HDF5-based):

  http://svn.origo.ethz.ch/wsvn/f5/doc/Particles_8c-example.html

It's inefficient only very few particles because the overhead on
the metadata structure is then more prominent, but for millions
of particles that would be well. I haven't tried this structure yet
with a million timesteps, which would lead to a million groups then.
I would assume HDF5 is able to handle such a situation well, but
it could make sense to bundle groups of similar timesteps hierarchically,
too.

    Werner

···

On Wed, 16 Mar 2011 08:29:27 -0500, Yngve Inntjore Levinsen <yngve.inntjore.levinsen@cern.ch> wrote:

Yes of course Francesc, I was thinking float = half of 64bit instead of 4x 8bit :slight_smile:
I was thinking that it might be beneficial to keep the size in powers of 2, so that is why I chose 1024 and not 1000. I keep it as a variable so I can easily change it.

Werner, I was thinking that I should eventually move to a sequence of 1D arrays, but it requires slightly more rewriting. The number of lines I have to write depends on whether or not the particle is still alive. I am starting out with an equal amount of particles, but have no means to know if I need to write the position of a given particle 0 times or one million times. Typically I have something like 1 million timesteps, but I do not write down trajectories all the time (when is dependent on the Monte Carlo so no way to know in advance)

Ideally I would've written all analysis into the code itself so I didn't have to write the trajectories all the time (I have not made this choice!), but that requires too much work for me to handle at the moment. Using HDF5 will reduce the storage space needed by about a factor 6 from my estimates, improve precision, and significantly reduce CPU hours needed as well. This is already a great improvement!

Cheers,
Yngve

On Wednesday 16 March 2011 02:09:36 PM Werner Benger wrote:

Hi,

  what's the reason for using a 2D extendable dataset instead of a sequence
of 1D arrays
in a group, using one group per time step? How many particles and time
steps do you
have typically? I assume in your case the number of particles is constant
over time?

Cheers,
  Werner

On Wed, 16 Mar 2011 03:52:10 -0500, Yngve Inntjore Levinsen >> <yngve.inntjore.levinsen@cern.ch> wrote:

> Dear hierarchical people,
>
> I have currently converted a piece of code from using a simple ascii
> format for output into using HDF5. What the code does is at every
> iteration dumping some information about particle
> energy/trajectory/position to the ascii file (this is a particle
> tracking code).
>
> Initially I then did the same with the HDF5 library, having a unlimited
> row dimension in a 2D array and using h5extend_f to extend by one
> element each time and writing a hyperslab of one row to the file. As
> some (perhaps most) of you might have guessed or know already, this was
> a rather bad idea. The file (without compression) was about the same
> size as the ascii file (but obviously with higher precision), and
> reading the file in subsequent analysis was at least an order of
> magnitude slower.
>
> I then realized that I probably needed to write less frequently and
> rather keeping a semi-large hyperslab in memory. I chose a hyperslab of
> 1000 rows, but otherwise using the same procedure. This seems to be both
> fast and with compression creating quite a bit smaller file. I tried
> even larger slabs, but did not see any speed improvement in my initial
> testing
>
> My question really was just if there are some recommended ways to do
> this? I would imagine I am not the first that want to use HDF5 in this
> way, dumping some data at every iteration of a given simulation, without
> having to keep it all in memory until the end?
>
> Thanks for all explanations/suggestions/experiences related to this
> problem you can provide me so I can make the best design choices in my
> program! :slight_smile:
>
> Cheers,
> Yngve

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Thank you for your reply,

I currently use a chunk size of 9x1024 floats, which by my calculations would be 4.5 kB. You say I should still go up by a factor 10-100 in chunk size then?

Cheers,
Yngve

···

On Wednesday 16 March 2011 11:05:14 AM Francesc Alted wrote:

A Wednesday 16 March 2011 09:52:10 Yngve Inntjore Levinsen escrigué:
> Dear hierarchical people,
>
> I have currently converted a piece of code from using a simple ascii
> format for output into using HDF5. What the code does is at every
> iteration dumping some information about particle
> energy/trajectory/position to the ascii file (this is a particle
> tracking code).
>
> Initially I then did the same with the HDF5 library, having a
> unlimited row dimension in a 2D array and using h5extend_f to
> extend by one element each time and writing a hyperslab of one row
> to the file. As some (perhaps most) of you might have guessed or
> know already, this was a rather bad idea. The file (without
> compression) was about the same size as the ascii file (but
> obviously with higher precision), and reading the file in subsequent
> analysis was at least an order of magnitude slower.
>
> I then realized that I probably needed to write less frequently and
> rather keeping a semi-large hyperslab in memory. I chose a hyperslab
> of 1000 rows, but otherwise using the same procedure. This seems to
> be both fast and with compression creating quite a bit smaller file.
> I tried even larger slabs, but did not see any speed improvement in
> my initial testing
>
> My question really was just if there are some recommended ways to do
> this? I would imagine I am not the first that want to use HDF5 in
> this way, dumping some data at every iteration of a given
> simulation, without having to keep it all in memory until the end?

For getting a good performance is very important your chunksize.
Typical figures for serial I/O are between 32 KB and 1 MB, depending the
final size of the dataset. Which one you are using?

9x1024 floats is something like 36 KB (single precision) or 72 KB (double
precision). You can try several values up to 1 MB and select the one is
giving you best performance. If the chunksize is a power of 2, it would be
great, but it is not really necessary.

HTH,

···

2011/3/16 Yngve Inntjore Levinsen <yngve.inntjore.levinsen@cern.ch>

On Wednesday 16 March 2011 11:05:14 AM Francesc Alted wrote:
> For getting a good performance is very important your chunksize.
> Typical figures for serial I/O are between 32 KB and 1 MB, depending the
> final size of the dataset. Which one you are using?
>
>

Thank you for your reply,

I currently use a chunk size of 9x1024 floats, which by my calculations
would be 4.5 kB. You say I should still go up by a factor 10-100 in chunk
size then?

--
Francesc Alted