Speeding up HDF5 write operation

Hdf-forum archives
1

Hi All,

I am writing a HDF5 file in parallel. But to my surprise, the performance of the
parallel write isn't better compared to the serial binary write operation. To
write 265875 real numbers, my HDF write takes about 0.1 seconds whereas the
serial binary operation takes around 0.07 seconds. This is surprising as
parallel should be atleast as fast as serial if not any faster.

Can anybody give me any suggestions as to what can be done to noticably speedup
this write operation ?

Will the performance of HDF5 write be better than binary for very large arrays ?
If not how can I bring any substantial speedup ?

Regards,
Nikhil

···

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

2

Hi Nikhil,

Hi All,

I am writing a HDF5 file in parallel. But to my surprise, the performance of the
parallel write isn't better compared to the serial binary write operation. To
write 265875 real numbers, my HDF write takes about 0.1 seconds whereas the
serial binary operation takes around 0.07 seconds. This is surprising as
parallel should be atleast as fast as serial if not any faster.

Can anybody give me any suggestions as to what can be done to noticably speedup
this write operation ?

  Hmm, are you using collective or independent parallel I/O? Also, that's a pretty small dataset, so you are not likely to see much difference either way.

Will the performance of HDF5 write be better than binary for very large arrays ?

  Our goal is to make HDF5 writes be equivalent to binary for large raw data I/O operations, but to make the files produced self-describing, portable, etc. also.

If not how can I bring any substantial speedup ?

  This is a very hard question to answer without more details... :slight_smile:

    Quincey

···

On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

3

Maybe I am missing something here, but I am not sure why one would ever expect a 'parallel write' of N blocks of data to be 'faster' than a serial write of the same total amount of data ... assuming the jobs are running on the same machine, writing to the same file.

Ultimately you would be constrained by the file subsystem/disk subsystem's ability to write this data to a disk which I would assume is , in general , a serial bottleneck operation unless you are dealing with sophisticated disk arrays where data could be written in parallel on multiple disks... HDF cannot perform magic.

My simplistic understanding of P-HDF is that it solves the problem of multiple MPI jobs writing to the same HDF file.

···

On Jul 9, 2008, at 4:39 PM, Nikhil Laghave wrote:

Will the performance of HDF5 write be better than binary for very large arrays ?
If not how can I bring any substantial speedup ?

Andrew

4

Hi Andrew,

Maybe I am missing something here, but I am not sure why one would ever expect a 'parallel write' of N blocks of data to be 'faster' than a serial write of the same total amount of data ... assuming the jobs are running on the same machine, writing to the same file.

Ultimately you would be constrained by the file subsystem/disk subsystem's ability to write this data to a disk which I would assume is , in general , a serial bottleneck operation unless you are dealing with sophisticated disk arrays where data could be written in parallel on multiple disks... HDF cannot perform magic.

  You are correct, we rely on the MPI layer to perform the magic. :slight_smile: A good MPI implementation with a good parallel file system can certainly make the parallel write go faster than a serial write, in the right circumstances, though.

My simplistic understanding of P-HDF is that it solves the problem of multiple MPI jobs writing to the same HDF file.

  The HDF5 library does rely on MPI for the coordination aspect of multiple MPI processes writing to the file, but it also leverages the MPI collective I/O operations when requested by the application, which can sometimes speed up I/O for large raw data writes.

  Quincey

···

On Jul 10, 2008, at 3:40 PM, Andrew Cunningham wrote:

On Jul 9, 2008, at 4:39 PM, Nikhil Laghave wrote:

Will the performance of HDF5 write be better than binary for very large arrays ?
If not how can I bring any substantial speedup ?

Andrew

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

5

Hi Andrew,

The disk on which the processors are writing is a network disk(not root
processors local disk)

Regards,
Nikhil

Hi Nikhil,
  I am somewhat unclear about what you are comparing...
  "serial write operation"= On a single CPU, in a single,serial write
operation using, say, fwrite() or FORTRAN equivalent, you write 1.2
million elements to a file on root processors local disk ( or a
network disk??)
  "parallel HDF operation" = Using e.g. 15 CPUS, and the PHDF API, you
split the write of 1.2 million elements operation into 15 parallel
tasks writing to a HDF file on root processor's local hard disk?

Andrew

> Hello,
>
> I ran my parallel code for much bigger array sizes(approx. 1.2
> million) and with
> large no. of processors(120). However, the write operation is still
> slower than
> binary writes. This is quite surprising since only the root writes
> the array for
> binary writes whereas in my code, about 15 processors write it in
> parallel, and
> is still slower than the serial write operation in binary.
>
> I

Andrew

Regards,
Nikhil

···

On Jul 23, 2008, at 10:09 AM, Nikhil Laghave wrote:

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

6

Hello,

I ran my parallel code for much bigger array sizes(approx. 1.2 million) and with
large no. of processors(120). However, the write operation is still slower than
binary writes. This is quite surprising since only the root writes the array for
binary writes whereas in my code, about 15 processors write it in parallel, and
is still slower than the serial write operation in binary.

It is mentioned below that the writes may be serialized due to the irregular
pattern I am writing. May be this is what is happening. Is there some tool by
which I can by which I find out where exactly the problem lies.

Can I find out which HDF5 function calls take the most amount of time ?

Thanks.

Regards,
Nikhil

Hi Nikhil,

> Hi,
>
> Sorry about that. Its attached this time.

  OK, I took a look at your section of code and although it's doing
parallel writes, they may be getting serialized somewhere under HDF5
by the MPI implementation due to the [apparently] non-regular pattern
you are writing. It's also very likely that you are writing too small
of an amount of data to see much benefit from parallel I/O.

  Quincey

> Regards,
> NIkhil
>
>> Hi Nikhil,
>>
>>
>>> Hi,
>>>
>>> Thanks for your reply.
>>>
>>> I am attaching part of my code that does the parallel write.
>>> Points to notice are:
>>>
>>> 1. for 'nprocs' processors, there are 'nend' diagonal processors
>>> that are
>>> actually doing the write, where:
>>>
>>> nprocs = nend * (nend+1) / 2
>>>
>>> 2. the subroutine for parallel write, 'phdfwrite' is present in the
>>> file hdfmodule.f
>>>
>>> 3. This subroutine is called only by the diagonal processors(nend)
>>>
>>> Please find attached the source files.
>>
>> There was no attachment on your message.
>>
>> Quincey
>>
>>> I also notice that for 265875 real nos.,
>>> there is no speed difference even between INDEPENDENT and COLLECTIVE
>>> IO. Is this
>>> because of the small size of the array. Also do you find anything
>>> that I may be
>>> doing which reduces the speed ?
>>>
>>> Best Regards,
>>> Nikhil
>>>
>>>> Hi Nikhil,
>>>>
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am writing a HDF5 file in parallel. But to my surprise, the
>>>>> performance of the
>>>>> parallel write isn't better compared to the serial binary write
>>>>> operation. To
>>>>> write 265875 real numbers, my HDF write takes about 0.1 seconds
>>>>> whereas the
>>>>> serial binary operation takes around 0.07 seconds. This is
>>>>> surprising as
>>>>> parallel should be atleast as fast as serial if not any faster.
>>>>>
>>>>> Can anybody give me any suggestions as to what can be done to
>>>>> noticably speedup
>>>>> this write operation ?
>>>>
>>>> Hmm, are you using collective or independent parallel I/O? Also,
>>>> that's a pretty small dataset, so you are not likely to see much
>>>> difference either way.
>>>>
>>>>> Will the performance of HDF5 write be better than binary for very
>>>>> large arrays ?
>>>>
>>>> Our goal is to make HDF5 writes be equivalent to binary for large
>>>> raw
>>>> data I/O operations, but to make the files produced self-
>>>> describing,
>>>> portable, etc. also.
>>>>
>>>>> If not how can I bring any substantial speedup ?
>>>>
>>>> This is a very hard question to answer without more details... :slight_smile:
>>>>
>>>> Quincey
>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Nikhil
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> This mailing list is for HDF software users discussion.
>>>>> To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org

>>>>> .
>>>>> To unsubscribe, send a message to hdf-forum-
>>>>> unsubscribe@hdfgroup.org.
>>>>>
>>>>>
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>> This mailing list is for HDF software users discussion.
>>>> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
>>>> .
>>>> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org
>>>> .
>>>>
>>>
>>>
>>> Regards,
>>> Nikhil
>>>
>>>
>>>
>>> ----------------------------------------------------------------------
>>> This mailing list is for HDF software users discussion.
>>> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
>>> .
>>> To unsubscribe, send a message to hdf-forum-
>>> unsubscribe@hdfgroup.org.
>>>
>>>
>>
>
>
> Regards,
> Nikhil
> <
> parallel
> .f
> >
> <
> hdfmodule
> .f
> >
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil

···

On Jul 10, 2008, at 2:34 PM, Nikhil Laghave wrote:
>> On Jul 10, 2008, at 2:07 PM, Nikhil Laghave wrote:
>>>> On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

7

Hi Nikhil,
  I am somewhat unclear about what you are comparing...
  "serial write operation"= On a single CPU, in a single,serial write operation using, say, fwrite() or FORTRAN equivalent, you write 1.2 million elements to a file on root processors local disk ( or a network disk??)
  "parallel HDF operation" = Using e.g. 15 CPUS, and the PHDF API, you split the write of 1.2 million elements operation into 15 parallel tasks writing to a HDF file on root processor's local hard disk?

Andrew

···

On Jul 23, 2008, at 10:09 AM, Nikhil Laghave wrote:

Hello,

I ran my parallel code for much bigger array sizes(approx. 1.2 million) and with
large no. of processors(120). However, the write operation is still slower than
binary writes. This is quite surprising since only the root writes the array for
binary writes whereas in my code, about 15 processors write it in parallel, and
is still slower than the serial write operation in binary.

I

Andrew

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

8

Hello Nikhil,

If you haven't already, you might see if there is consulting advice available for the system where you are running your code.

There can be different profiling tools available on different systems that could help you see where the bottlenecks are. Bracketing the suspect area with timing calls and using printf's to save the results for later analysis is the brute-force standby quick-and-dirty insights.

The characteristics of the underlying parallel filesystem can make a big difference in the performance you'll get. Some systems have filesystems that are accessible from all nodes, but that in fact are not true parallel filesystems. You might want to confirm you're using an appropriate filesystem for your writes.

Are the serial write and parallel writes going to the same filesystem? If not, performance characteristics could be different.
When you do the serial write are the other processors continuing to do 'real work', while with the parallel write the others are blocking until the I/O completes? If so, then the runtime may indeed be longer for the parallel case due to the blocked processes. (I haven't looked at the code, but know some applications behave this way).

-Ruth

···

On Jul 23, 2008, at 12:09 PM, Nikhil Laghave wrote:

Hello,

I ran my parallel code for much bigger array sizes(approx. 1.2 million) and with
large no. of processors(120). However, the write operation is still slower than
binary writes. This is quite surprising since only the root writes the array for
binary writes whereas in my code, about 15 processors write it in parallel, and
is still slower than the serial write operation in binary.

It is mentioned below that the writes may be serialized due to the irregular
pattern I am writing. May be this is what is happening. Is there some tool by
which I can by which I find out where exactly the problem lies.

Can I find out which HDF5 function calls take the most amount of time ?

Thanks.

Regards,
Nikhil

Hi Nikhil,

On Jul 10, 2008, at 2:34 PM, Nikhil Laghave wrote:

Hi,

Sorry about that. Its attached this time.

  OK, I took a look at your section of code and although it's doing
parallel writes, they may be getting serialized somewhere under HDF5
by the MPI implementation due to the [apparently] non-regular pattern
you are writing. It's also very likely that you are writing too small
of an amount of data to see much benefit from parallel I/O.

  Quincey

Regards,
NIkhil

Hi Nikhil,

On Jul 10, 2008, at 2:07 PM, Nikhil Laghave wrote:

Hi,

Thanks for your reply.

I am attaching part of my code that does the parallel write.
Points to notice are:

1. for 'nprocs' processors, there are 'nend' diagonal processors
that are
actually doing the write, where:

nprocs = nend * (nend+1) / 2

2. the subroutine for parallel write, 'phdfwrite' is present in the
file hdfmodule.f

3. This subroutine is called only by the diagonal processors(nend)

Please find attached the source files.

  There was no attachment on your message.

    Quincey

I also notice that for 265875 real nos.,
there is no speed difference even between INDEPENDENT and COLLECTIVE
IO. Is this
because of the small size of the array. Also do you find anything
that I may be
doing which reduces the speed ?

Best Regards,
Nikhil

Hi Nikhil,

On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

Hi All,

I am writing a HDF5 file in parallel. But to my surprise, the
performance of the
parallel write isn't better compared to the serial binary write
operation. To
write 265875 real numbers, my HDF write takes about 0.1 seconds
whereas the
serial binary operation takes around 0.07 seconds. This is
surprising as
parallel should be atleast as fast as serial if not any faster.

Can anybody give me any suggestions as to what can be done to
noticably speedup
this write operation ?

  Hmm, are you using collective or independent parallel I/O? Also,
that's a pretty small dataset, so you are not likely to see much
difference either way.

Will the performance of HDF5 write be better than binary for very
large arrays ?

  Our goal is to make HDF5 writes be equivalent to binary for large
raw
data I/O operations, but to make the files produced self-
describing,
portable, etc. also.

If not how can I bring any substantial speedup ?

  This is a very hard question to answer without more details... :slight_smile:

    Quincey

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org

.
To unsubscribe, send a message to hdf-forum-
unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org
.

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-
unsubscribe@hdfgroup.org.

Regards,
Nikhil
<
parallel
.f

<
hdfmodule
.f

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

9

Hi,

Hi Nikhil,

> Hi,
>
> Sorry about that. Its attached this time.

  OK, I took a look at your section of code and although it's doing
parallel writes, they may be getting serialized somewhere under HDF5
by the MPI implementation due to the [apparently] non-regular pattern
you are writing.

I assume by non-regular you mean the sizes of the arrays written by the
processors. Yes, thats true and because the sizes of the arrays are not the same
for all processors. This is the reason I could not set the chunk size as the
size of array written my each processor(it being non-regular). Hence I added the
sizes of arrays of all processors and set it as chunk size since this value is
constant.

It's also very likely that you are writing too small
of an amount of data to see much benefit from parallel I/O.

Yes this is very likely. I am trying to get an account on NERSC so that I can
run the larger runs of code. Due to memory restrictions, this is maximum I could
run on my work cluster.

  Quincey

Best Regards,
Nikhil

> Regards,
> NIkhil
>
>> Hi Nikhil,
>>
>>
>>> Hi,
>>>
>>> Thanks for your reply.
>>>
>>> I am attaching part of my code that does the parallel write.
>>> Points to notice are:
>>>
>>> 1. for 'nprocs' processors, there are 'nend' diagonal processors
>>> that are
>>> actually doing the write, where:
>>>
>>> nprocs = nend * (nend+1) / 2
>>>
>>> 2. the subroutine for parallel write, 'phdfwrite' is present in the
>>> file hdfmodule.f
>>>
>>> 3. This subroutine is called only by the diagonal processors(nend)
>>>
>>> Please find attached the source files.
>>
>> There was no attachment on your message.
>>
>> Quincey
>>
>>> I also notice that for 265875 real nos.,
>>> there is no speed difference even between INDEPENDENT and COLLECTIVE
>>> IO. Is this
>>> because of the small size of the array. Also do you find anything
>>> that I may be
>>> doing which reduces the speed ?
>>>
>>> Best Regards,
>>> Nikhil
>>>
>>>> Hi Nikhil,
>>>>
>>>>
>>>>> Hi All,
>>>>>
>>>>> I am writing a HDF5 file in parallel. But to my surprise, the
>>>>> performance of the
>>>>> parallel write isn't better compared to the serial binary write
>>>>> operation. To
>>>>> write 265875 real numbers, my HDF write takes about 0.1 seconds
>>>>> whereas the
>>>>> serial binary operation takes around 0.07 seconds. This is
>>>>> surprising as
>>>>> parallel should be atleast as fast as serial if not any faster.
>>>>>
>>>>> Can anybody give me any suggestions as to what can be done to
>>>>> noticably speedup
>>>>> this write operation ?
>>>>
>>>> Hmm, are you using collective or independent parallel I/O? Also,
>>>> that's a pretty small dataset, so you are not likely to see much
>>>> difference either way.
>>>>
>>>>> Will the performance of HDF5 write be better than binary for very
>>>>> large arrays ?
>>>>
>>>> Our goal is to make HDF5 writes be equivalent to binary for large
>>>> raw
>>>> data I/O operations, but to make the files produced self-
>>>> describing,
>>>> portable, etc. also.
>>>>
>>>>> If not how can I bring any substantial speedup ?
>>>>
>>>> This is a very hard question to answer without more details... :slight_smile:
>>>>
>>>> Quincey
>>>>
>>>>>
>>>>>
>>>>> Regards,
>>>>> Nikhil
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ----------------------------------------------------------------------
>>>>> This mailing list is for HDF software users discussion.
>>>>> To subscribe to this list, send a message to

hdf-forum-subscribe@hdfgroup.org

>>>>> .
>>>>> To unsubscribe, send a message to hdf-forum-
>>>>> unsubscribe@hdfgroup.org.
>>>>>
>>>>>
>>>>
>>>>
>>>> ----------------------------------------------------------------------
>>>> This mailing list is for HDF software users discussion.
>>>> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
>>>> .
>>>> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org
>>>> .
>>>>
>>>
>>>
>>> Regards,
>>> Nikhil
>>>
>>>
>>>
>>> ----------------------------------------------------------------------
>>> This mailing list is for HDF software users discussion.
>>> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
>>> .
>>> To unsubscribe, send a message to hdf-forum-
>>> unsubscribe@hdfgroup.org.
>>>
>>>
>>
>
>
> Regards,
> Nikhil
> <
> parallel
> .f
> >
> <
> hdfmodule
> .f
> >
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil

···

On Jul 10, 2008, at 2:34 PM, Nikhil Laghave wrote:
>> On Jul 10, 2008, at 2:07 PM, Nikhil Laghave wrote:
>>>> On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

10

Hi,

Sorry about that. Its attached this time.

Regards,
NIkhil

Hi Nikhil,

> Hi,
>
> Thanks for your reply.
>
> I am attaching part of my code that does the parallel write.
> Points to notice are:
>
> 1. for 'nprocs' processors, there are 'nend' diagonal processors
> that are
> actually doing the write, where:
>
> nprocs = nend * (nend+1) / 2
>
> 2. the subroutine for parallel write, 'phdfwrite' is present in the
> file hdfmodule.f
>
> 3. This subroutine is called only by the diagonal processors(nend)
>
> Please find attached the source files.

  There was no attachment on your message.

    Quincey

> I also notice that for 265875 real nos.,
> there is no speed difference even between INDEPENDENT and COLLECTIVE
> IO. Is this
> because of the small size of the array. Also do you find anything
> that I may be
> doing which reduces the speed ?
>
> Best Regards,
> Nikhil
>
>> Hi Nikhil,
>>
>>
>>> Hi All,
>>>
>>> I am writing a HDF5 file in parallel. But to my surprise, the
>>> performance of the
>>> parallel write isn't better compared to the serial binary write
>>> operation. To
>>> write 265875 real numbers, my HDF write takes about 0.1 seconds
>>> whereas the
>>> serial binary operation takes around 0.07 seconds. This is
>>> surprising as
>>> parallel should be atleast as fast as serial if not any faster.
>>>
>>> Can anybody give me any suggestions as to what can be done to
>>> noticably speedup
>>> this write operation ?
>>
>> Hmm, are you using collective or independent parallel I/O? Also,
>> that's a pretty small dataset, so you are not likely to see much
>> difference either way.
>>
>>> Will the performance of HDF5 write be better than binary for very
>>> large arrays ?
>>
>> Our goal is to make HDF5 writes be equivalent to binary for large
>> raw
>> data I/O operations, but to make the files produced self-describing,
>> portable, etc. also.
>>
>>> If not how can I bring any substantial speedup ?
>>
>> This is a very hard question to answer without more details... :slight_smile:
>>
>> Quincey
>>
>>>
>>>
>>> Regards,
>>> Nikhil
>>>
>>>
>>>
>>>
>>> ----------------------------------------------------------------------
>>> This mailing list is for HDF software users discussion.
>>> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
>>> .
>>> To unsubscribe, send a message to hdf-forum-
>>> unsubscribe@hdfgroup.org.
>>>
>>>
>>
>>
>> ----------------------------------------------------------------------
>> This mailing list is for HDF software users discussion.
>> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
>> .
>> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
>>
>
>
> Regards,
> Nikhil
>
>
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
>
>

Regards,
Nikhil

parallel.f (2.14 KB)

hdfmodule.f (8.03 KB)

···

On Jul 10, 2008, at 2:07 PM, Nikhil Laghave wrote:
>> On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

11

Hi Nikhil,

Hi,

Sorry about that. Its attached this time.

  OK, I took a look at your section of code and although it's doing parallel writes, they may be getting serialized somewhere under HDF5 by the MPI implementation due to the [apparently] non-regular pattern you are writing. It's also very likely that you are writing too small of an amount of data to see much benefit from parallel I/O.

  Quincey

···

On Jul 10, 2008, at 2:34 PM, Nikhil Laghave wrote:

Regards,
NIkhil

Hi Nikhil,

On Jul 10, 2008, at 2:07 PM, Nikhil Laghave wrote:

Hi,

Thanks for your reply.

I am attaching part of my code that does the parallel write.
Points to notice are:

1. for 'nprocs' processors, there are 'nend' diagonal processors
that are
actually doing the write, where:

nprocs = nend * (nend+1) / 2

2. the subroutine for parallel write, 'phdfwrite' is present in the
file hdfmodule.f

3. This subroutine is called only by the diagonal processors(nend)

Please find attached the source files.

  There was no attachment on your message.

    Quincey

I also notice that for 265875 real nos.,
there is no speed difference even between INDEPENDENT and COLLECTIVE
IO. Is this
because of the small size of the array. Also do you find anything
that I may be
doing which reduces the speed ?

Best Regards,
Nikhil

Hi Nikhil,

On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

Hi All,

I am writing a HDF5 file in parallel. But to my surprise, the
performance of the
parallel write isn't better compared to the serial binary write
operation. To
write 265875 real numbers, my HDF write takes about 0.1 seconds
whereas the
serial binary operation takes around 0.07 seconds. This is
surprising as
parallel should be atleast as fast as serial if not any faster.

Can anybody give me any suggestions as to what can be done to
noticably speedup
this write operation ?

  Hmm, are you using collective or independent parallel I/O? Also,
that's a pretty small dataset, so you are not likely to see much
difference either way.

Will the performance of HDF5 write be better than binary for very
large arrays ?

  Our goal is to make HDF5 writes be equivalent to binary for large
raw
data I/O operations, but to make the files produced self-describing,
portable, etc. also.

If not how can I bring any substantial speedup ?

  This is a very hard question to answer without more details... :slight_smile:

    Quincey

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-
unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil
<parallel.f><hdfmodule.f>----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

12

Hi,

Thanks for your reply.

I am attaching part of my code that does the parallel write.
Points to notice are:

1. for 'nprocs' processors, there are 'nend' diagonal processors that are
actually doing the write, where:

nprocs = nend * (nend+1) / 2

2. the subroutine for parallel write, 'phdfwrite' is present in the file hdfmodule.f

3. This subroutine is called only by the diagonal processors(nend)

Please find attached the source files. I also notice that for 265875 real nos.,
there is no speed difference even between INDEPENDENT and COLLECTIVE IO. Is this
because of the small size of the array. Also do you find anything that I may be
doing which reduces the speed ?

Best Regards,
Nikhil

Hi Nikhil,

> Hi All,
>
> I am writing a HDF5 file in parallel. But to my surprise, the
> performance of the
> parallel write isn't better compared to the serial binary write
> operation. To
> write 265875 real numbers, my HDF write takes about 0.1 seconds
> whereas the
> serial binary operation takes around 0.07 seconds. This is
> surprising as
> parallel should be atleast as fast as serial if not any faster.
>
> Can anybody give me any suggestions as to what can be done to
> noticably speedup
> this write operation ?

  Hmm, are you using collective or independent parallel I/O? Also,
that's a pretty small dataset, so you are not likely to see much
difference either way.

> Will the performance of HDF5 write be better than binary for very
> large arrays ?

  Our goal is to make HDF5 writes be equivalent to binary for large raw
data I/O operations, but to make the files produced self-describing,
portable, etc. also.

> If not how can I bring any substantial speedup ?

  This is a very hard question to answer without more details... :slight_smile:

    Quincey

>
>
> Regards,
> Nikhil
>
>
>
>
> ----------------------------------------------------------------------
> This mailing list is for HDF software users discussion.
> To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
> .
> To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
>
>

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil

···

On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

13

Hi Nikhil,

Hi,

Thanks for your reply.

I am attaching part of my code that does the parallel write.
Points to notice are:

1. for 'nprocs' processors, there are 'nend' diagonal processors that are
actually doing the write, where:

nprocs = nend * (nend+1) / 2

2. the subroutine for parallel write, 'phdfwrite' is present in the file hdfmodule.f

3. This subroutine is called only by the diagonal processors(nend)

Please find attached the source files.

  There was no attachment on your message.

    Quincey

···

On Jul 10, 2008, at 2:07 PM, Nikhil Laghave wrote:

I also notice that for 265875 real nos.,
there is no speed difference even between INDEPENDENT and COLLECTIVE IO. Is this
because of the small size of the array. Also do you find anything that I may be
doing which reduces the speed ?

Best Regards,
Nikhil

Hi Nikhil,

On Jul 9, 2008, at 6:39 PM, Nikhil Laghave wrote:

Hi All,

I am writing a HDF5 file in parallel. But to my surprise, the
performance of the
parallel write isn't better compared to the serial binary write
operation. To
write 265875 real numbers, my HDF write takes about 0.1 seconds
whereas the
serial binary operation takes around 0.07 seconds. This is
surprising as
parallel should be atleast as fast as serial if not any faster.

Can anybody give me any suggestions as to what can be done to
noticably speedup
this write operation ?

  Hmm, are you using collective or independent parallel I/O? Also,
that's a pretty small dataset, so you are not likely to see much
difference either way.

Will the performance of HDF5 write be better than binary for very
large arrays ?

  Our goal is to make HDF5 writes be equivalent to binary for large raw
data I/O operations, but to make the files produced self-describing,
portable, etc. also.

If not how can I bring any substantial speedup ?

  This is a very hard question to answer without more details... :slight_smile:

    Quincey

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org
.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Regards,
Nikhil

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.