Hi,
I am having trouble to read from a 721GB file using 4096 nodes.
When I test with a few nodes, it works, but when I test with more nodes, it
takes significantly more time.
What the test program does it only read in the data and deleting it.
Here's the timing information:
Nodes | Time For Running Entire Program
16 4:28
32 6:55
64 8:56
128 11:22
256 13:25
512 15:34
768 28:34
800 29:04
I am running the program in a Cray XK6 system, and the file system is Lustre
*There is a big gap after 512 nodes, and with 4096 nodes, it couldn't finish
in 6 hours.
Is this normal? Shouldn't it be a lot faster?*
Here is my reading function, it's similar to the sample hdf5 parallel
program:
#include <hdf5.h>
#include <stdio.h>
#include <stdlib.h>
#include <assert.h>
void readData(const char* filename, int region_index[3], int
region_count[3], float* flow_field[6])
{
char attributes[6][50];
sprintf(attributes[0], "/uvel");
sprintf(attributes[1], "/vvel");
sprintf(attributes[2], "/wvel");
sprintf(attributes[3], "/pressure");
sprintf(attributes[4], "/temp");
sprintf(attributes[5], "/OH");
herr_t status;
hid_t file_id;
hid_t dset_id;
hid_t dset_plist;
// open file spaces
hid_t acc_tpl = H5Pcreate(H5P_FILE_ACCESS);
status = H5Pset_fapl_mpio(acc_tpl, MPI_COMM_WORLD, MPI_INFO_NULL);
file_id = H5Fopen(filename, H5F_ACC_RDONLY, acc_tpl);
status = H5Pclose(acc_tpl);
for (int i = 0; i < 6; ++i)
{
// open dataset
dset_id = H5Dopen(file_id, attributes[i], H5P_DEFAULT);
// get dataset space
hid_t spac_id = H5Dget_space(dset_id);
hsize_t htotal_size3[3];
status = H5Sget_simple_extent_dims(spac_id, htotal_size3, NULL);
hsize_t region_size3[3] = {htotal_size3[0] / region_count[0],
htotal_size3[1] / region_count[1],
htotal_size3[2] / region_count[2]};
// hyperslab
hsize_t start[3] = {region_index[0] * region_size3[0],
region_index[1] * region_size3[1],
region_index[2] * region_size3[2]};
hsize_t count[3] = {region_size3[0], region_size3[1], region_size3[2]};
status = H5Sselect_hyperslab(spac_id, H5S_SELECT_SET, start, NULL,
count, NULL);
hid_t memspace = H5Screate_simple(3, count, NULL);
// read
hid_t xfer_plist = H5Pcreate(H5P_DATASET_XFER);
status = H5Pset_dxpl_mpio(xfer_plist, H5FD_MPIO_COLLECTIVE);
flow_field[i] = (float *) malloc(count[0] * count[1] * count[2] *
sizeof(float));
status = H5Dread(dset_id, H5T_NATIVE_FLOAT, memspace, spac_id,
xfer_plist, flow_field[i]);
// clean up
H5Dclose(dset_id);
H5Sclose(spac_id);
H5Pclose(xfer_plist);
}
H5Fclose(file_id);
}
*Do you see any problem with this function? I am new to hdf5 parallel.*
Thanks in advance!
···
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Slow-Reading-721GB-File-in-Parallel-tp4021429.html
Sent from the hdf-forum mailing list archive at Nabble.com.