I have a HDF5 file ~500MB that is opened for parallel access and independent read (each process reads his own part of the file).
I thought the memory requirement would be 500MB/Number_of_processes...but I have memory failures and I was wondering
whether it could be much more than that ?
Barbara
Hi Barbara,
On Feb 14, 2011, at 5:01 PM, Collignon, Barbara C. wrote:
I have a HDF5 file ~500MB that is opened for parallel access and independent read (each process reads his own part of the file).
I thought the memory requirement would be 500MB/Number_of_processes...but I have memory failures and I was wondering whether it could be much more than that ?
It should be in that range of memory, plus a little for buffering and caching. Is the dataset chunked or contiguous?
Quincey
Well I did nt chunked anything so I guess it is contiguous
________________________________________
From: hdf-forum-bounces@hdfgroup.org [hdf-forum-bounces@hdfgroup.org] On Behalf Of Quincey Koziol [koziol@hdfgroup.org]
Sent: Tuesday, February 15, 2011 8:50 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] memory requirement
Hi Barbara,
On Feb 14, 2011, at 5:01 PM, Collignon, Barbara C. wrote:
I have a HDF5 file ~500MB that is opened for parallel access and independent read (each process reads his own part of the file).
I thought the memory requirement would be 500MB/Number_of_processes...but I have memory failures and I was wondering whether it could be much more than that ?
It should be in that range of memory, plus a little for buffering and caching. Is the dataset chunked or contiguous?
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Barbara,
Well I did nt chunked anything so I guess it is contiguous
How many datasets are in the file?
Quincey
On Feb 15, 2011, at 9:33 AM, Collignon, Barbara C. wrote:
________________________________________
From: hdf-forum-bounces@hdfgroup.org [hdf-forum-bounces@hdfgroup.org] On Behalf Of Quincey Koziol [koziol@hdfgroup.org]
Sent: Tuesday, February 15, 2011 8:50 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] memory requirementHi Barbara,
On Feb 14, 2011, at 5:01 PM, Collignon, Barbara C. wrote:
I have a HDF5 file ~500MB that is opened for parallel access and independent read (each process reads his own part of the file).
I thought the memory requirement would be 500MB/Number_of_processes...but I have memory failures and I was wondering whether it could be much more than that ?
It should be in that range of memory, plus a little for buffering and caching. Is the dataset chunked or contiguous?
Quincey
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
