Hi Hdf users,
I am trying to write a C program to read a large (39GB) of raw binary
data and store it into a dataset in a hdf file. Attached is the code.
The problem i am facing is this: whenever I run this code, i see a *lot*
of memory usage; I am guessing that this is because the dataset that i
write into is not being flushed into the disk.....
What can i do to alleviate this? I am using hyperslabs to select one
"height" of a 3D cube structured data and write it to dataset. Is there
anyway i can ask the program to write the data to the disk, thus freeing
up the main memory? I have seen this article: http://www.hdfgroup.org/HDF5/faq/linux-mem.html
(am running a debian squeeze with 4GB memory)
but i am willing to take the time hit for disk write...
is your data set chunked? It might help to have the data in smaller chunks so each of them can be written independently, and it might be even more efficient if the chunk size is the same as your hyperslab size.
Hi Hdf users,
I am trying to write a C program to read a large (39GB) of raw binary
data and store it into a dataset in a hdf file. Attached is the code.
The problem i am facing is this: whenever I run this code, i see a *lot*
of memory usage; I am guessing that this is because the dataset that i
write into is not being flushed into the disk.....
What can i do to alleviate this? I am using hyperslabs to select one
"height" of a 3D cube structured data and write it to dataset. Is there
anyway i can ask the program to write the data to the disk, thus freeing
up the main memory? I have seen this article: http://www.hdfgroup.org/HDF5/faq/linux-mem.html
(am running a debian squeeze with 4GB memory)
but i am willing to take the time hit for disk write...
Thanks,
--Pratik
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
