Hi,
I'm trying to install HDF5 1.8.9 against the various permutations of compiler and MPI on our RHEL6 system and am having problems getting it to pass "make check" against Intel MPI 4.0.3.008.
The full output is below but, in short, testphdf5 is failing with:
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x0000000002798920 ***
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x000000000136a330 ***
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x00000000020e7890*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x00000000024d4370 ***
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x0000000002452f30 ***
It fails regardless of compiler used. These messages are from Intel 12.1.5.339 but I get similar output from GCC 4.7.1 and the patched GCC 4.4.6 shipped with RHEL6. I've successfully built and passed "make check" against OpenMPI 1.6.1 and MVAPICH2 1.8 with all three compilers.
Does anyone have experience building against Intel MPI, please? It feels like an Intel MPI problem, but it'd be good to know if it's an issue with HDF5.
Thanks,
Mark
···
--
-----------------------------------------------------------------
Mark Dixon Email : m.c.dixon@leeds.ac.uk
HPC/Grid Systems Support Tel (int): 35429
Information Systems Services Tel (ext): +44(0)113 343 5429
University of Leeds, LS2 9JT, UK
-----------------------------------------------------------------
Problem build notes:
export CC=mpiicc
export CXX=mpiicpc
export FC=mpiifort
export F77=mpiifort
export F90=mpiifort
export RUNPARALLEL="mpiexec.hydra -n \$\${NPROCS:=6}"
./configure --enable-parallel --disable-shared --enable-fortran --with-zlib --with-pic
make
make check
Problem test:
$ cd testpar
$ mpiexec.hydra -n 6 ./testphdf5
PHDF5 TESTS START
MPI-process 3.MPI-process 1.MPI-process 2. hostname=login1.polaris.leeds.ac.uk
hostname=login1.polaris.leeds.ac.uk
For help use: ./testphdf5 -help
Linked with hdf5 version 1.8 release 9
For help use: ./testphdf5 -help
Linked with hdf5 version 1.8 release 9
For help use: ./testphdf5 -help
Linked with hdf5 version 1.8 release 9
MPI-process 0. hostname=login1.polaris.leeds.ac.uk
For help use: ./testphdf5 -help
Linked with hdf5 version 1.8 release 9
MPI-process 4. hostname=login1.polaris.leeds.ac.uk
For help use: ./testphdf5 -help
Linked with hdf5 version 1.8 release 9
MPI-process 5. hostname=login1.polaris.leeds.ac.uk
For help use: ./testphdf5 -help
Linked with hdf5 version 1.8 release 9
*** Hint ***
You can use environment variable HDF5_PARAPREFIX to run parallel test files in a
different directory or to add file type prefix. E.g.,
HDF5_PARAPREFIX=pfs:/PFS/user/me
export HDF5_PARAPREFIX
*** End of Hint ***
Test filenames are:
ParaTest.h5
Testing -- fapl_mpio duplicate (mpiodup) Test filenames are:
ParaTest.h5
Testing -- fapl_mpio duplicate (mpiodup) Test filenames are:
ParaTest.h5
Testing -- fapl_mpio duplicate (mpiodup) Test filenames are:
ParaTest.h5
Testing -- fapl_mpio duplicate (mpiodup) Test filenames are:
ParaTest.h5
Testing -- fapl_mpio duplicate (mpiodup) Test filenames are:
ParaTest.h5
Testing -- fapl_mpio duplicate (mpiodup) Testing -- fapl_mpiposix duplicate (posixdup) Testing -- fapl_mpiposix duplicate (posixdup) Testing -- fapl_mpiposix duplicate (posixdup) Testing -- fapl_mpiposix duplicate (posixdup) Testing -- fapl_mpiposix duplicate (posixdup) Testing -- fapl_mpiposix duplicate (posixdup) Testing -- dataset using split communicators (split) Testing -- dataset using split communicators (split) Testing -- dataset using split communicators (split) Testing -- dataset using split communicators (split) Testing -- dataset using split communicators (split) Testing -- dataset using split communicators (split) Testing -- dataset independent write (idsetw) Testing -- dataset independent write (idsetw) Testing -- dataset independent write (idsetw) Testing -- dataset independent write (idsetw) Testing -- dataset independent write (idsetw) Testing -- dataset independent write (idsetw) Testing -- dataset independent read (idsetr) Testing -- dataset independent read (idsetr) Testing -- dataset independent read (idsetr) Testing -- dataset independent read (idsetr) Testing -- dataset independent read (idsetr) Testing -- dataset independent read (idsetr) Testing -- dataset collective write (cdsetw) Testing -- dataset collective write (cdsetw) Testing -- dataset collective write (cdsetw) Testing -- dataset collective write (cdsetw) Testing -- dataset collective write (cdsetw) Testing -- dataset collective write (cdsetw) Testing -- dataset collective read (cdsetr) Testing -- dataset collective read (cdsetr) Testing -- dataset collective read (cdsetr) Testing -- dataset collective read (cdsetr) Testing -- dataset collective read (cdsetr) Testing -- dataset collective read (cdsetr) Testing -- extendible dataset independent write (eidsetw) Testing -- extendible dataset independent write (eidsetw) Testing -- extendible dataset independent write (eidsetw) Testing -- extendible dataset independent write (eidsetw) Testing -- extendible dataset independent write (eidsetw) Testing -- extendible dataset independent write (eidsetw) Testing -- extendible dataset independent read (eidsetr) Testing -- extendible dataset independent read (eidsetr) Testing -- extendible dataset independent read (eidsetr) Testing -- extendible dataset independent read (eidsetr) Testing -- extendible dataset independent read (eidsetr) Testing -- extendible dataset independent read (eidsetr) Testing -- extendible dataset collective write (ecdsetw) Testing -- extendible dataset collective write (ecdsetw) Testing -- extendible dataset collective write (ecdsetw) Testing -- extendible dataset collective write (ecdsetw) Testing -- extendible dataset collective write (ecdsetw) Testing -- extendible dataset collective write (ecdsetw) Testing -- extendible dataset collective read (ecdsetr) Testing -- extendible dataset collective read (ecdsetr) Testing -- extendible dataset collective read (ecdsetr) Testing -- extendible dataset collective read (ecdsetr) Testing -- extendible dataset collective read (ecdsetr) Testing -- extendible dataset collective read (ecdsetr) Testing -- extendible dataset independent write #2 (eidsetw2) Testing -- extendible dataset independent write #2 (eidsetw2) Testing -- extendible dataset independent write #2 (eidsetw2) Testing -- extendible dataset independent write #2 (eidsetw2) Testing -- extendible dataset independent write #2 (eidsetw2) Testing -- extendible dataset independent write #2 (eidsetw2) Testing -- chunked dataset with none-selection (selnone) Testing -- chunked dataset with none-selection (selnone) Testing -- chunked dataset with none-selection (selnone) Testing -- chunked dataset with none-selection (selnone) Testing -- chunked dataset with none-selection (selnone) Testing -- chunked dataset with none-selection (selnone) Testing -- parallel extend Chunked allocation on serial file (calloc) Testing -- parallel extend Chunked allocation on serial file (calloc) Testing -- parallel extend Chunked allocation on serial file (calloc) Testing -- parallel extend Chunked allocation on serial file (calloc) Testing -- parallel extend Chunked allocation on serial file (calloc) Testing -- parallel extend Chunked allocation on serial file (calloc) Testing -- parallel read of dataset written serially with filters (fltread) Testing -- parallel read of dataset written serially with filters (fltread) Testing -- parallel read of dataset written serially with filters (fltread) Testing -- parallel read of dataset written serially with filters (fltread) Testing -- parallel read of dataset written serially with filters (fltread) Testing -- parallel read of dataset written serially with filters (fltread) Testing -- compressed dataset collective read (cmpdsetr) Testing -- compressed dataset collective read (cmpdsetr) Testing -- compressed dataset collective read (cmpdsetr) Testing -- compressed dataset collective read (cmpdsetr) Testing -- compressed dataset collective read (cmpdsetr) Testing -- compressed dataset collective read (cmpdsetr) Testing -- multiple datasets write (ndsetw) Testing -- multiple datasets write (ndsetw) Testing -- multiple datasets write (ndsetw) Testing -- multiple datasets write (ndsetw) Testing -- multiple datasets write (ndsetw) Testing -- multiple datasets write (ndsetw) Testing -- multiple groups write (ngrpw) Testing -- multiple groups write (ngrpw) Testing -- multiple groups write (ngrpw) Testing -- multiple groups write (ngrpw) Testing -- multiple groups write (ngrpw) Testing -- multiple groups write (ngrpw) Testing -- multiple groups read (ngrpr) Testing -- multiple groups read (ngrpr) Testing -- multiple groups read (ngrpr) Testing -- multiple groups read (ngrpr) Testing -- multiple groups read (ngrpr) Testing -- multiple groups read (ngrpr) Testing -- compact dataset test (compact) Testing -- compact dataset test (compact) Testing -- compact dataset test (compact) Testing -- compact dataset test (compact) Testing -- compact dataset test (compact) Testing -- compact dataset test (compact) Testing -- collective group and dataset write (cngrpw) Testing -- collective group and dataset write (cngrpw) Testing -- collective group and dataset write (cngrpw) Testing -- collective group and dataset write (cngrpw) Testing -- collective group and dataset write (cngrpw) Testing -- collective group and dataset write (cngrpw) Testing -- independent group and dataset read (ingrpr) Testing -- independent group and dataset read (ingrpr) Testing -- independent group and dataset read (ingrpr) Testing -- independent group and dataset read (ingrpr) Testing -- independent group and dataset read (ingrpr) Testing -- independent group and dataset read (ingrpr) Testing -- big dataset test (bigdset) Testing -- big dataset test (bigdset) Testing -- big dataset test (bigdset) Testing -- big dataset test (bigdset) Testing -- big dataset test (bigdset) Testing -- big dataset test (bigdset) Testing -- dataset fill value (fill) Testing -- dataset fill value (fill) Testing -- dataset fill value (fill) Testing -- dataset fill value (fill) Testing -- dataset fill value (fill) Testing -- dataset fill value (fill) Testing -- simple collective chunk io (cchunk1) Testing -- simple collective chunk io (cchunk1) Testing -- simple collective chunk io (cchunk1) Testing -- simple collective chunk io (cchunk1) Testing -- simple collective chunk io (cchunk1) Testing -- simple collective chunk io (cchunk1) Testing -- noncontiguous collective chunk io (cchunk2) Testing -- noncontiguous collective chunk io (cchunk2) Testing -- noncontiguous collective chunk io (cchunk2) Testing -- noncontiguous collective chunk io (cchunk2) Testing -- noncontiguous collective chunk io (cchunk2) Testing -- noncontiguous collective chunk io (cchunk2) Testing -- multi-chunk collective chunk io (cchunk3) Testing -- multi-chunk collective chunk io (cchunk3) Testing -- multi-chunk collective chunk io (cchunk3) Testing -- multi-chunk collective chunk io (cchunk3) Testing -- multi-chunk collective chunk io (cchunk3) Testing -- multi-chunk collective chunk io (cchunk3) Testing -- collective chunk io with partial non-selection (cchunk4) Testing -- collective chunk io with partial non-selection (cchunk4) Testing -- collective chunk io with partial non-selection (cchunk4) Testing -- collective chunk io with partial non-selection (cchunk4) Testing -- collective chunk io with partial non-selection (cchunk4) Testing -- collective chunk io with partial non-selection (cchunk4) Testing -- linked chunk collective IO without optimization (cchunk5) Testing -- linked chunk collective IO without optimization (cchunk5) Testing -- linked chunk collective IO without optimization (cchunk5) Testing -- linked chunk collective IO without optimization (cchunk5) Testing -- linked chunk collective IO without optimization (cchunk5) Testing -- linked chunk collective IO without optimization (cchunk5) Testing -- multi-chunk collective IO without optimization (cchunk6) Testing -- multi-chunk collective IO without optimization (cchunk6) Testing -- multi-chunk collective IO without optimization (cchunk6) Testing -- multi-chunk collective IO without optimization (cchunk6) Testing -- multi-chunk collective IO without optimization (cchunk6) Testing -- multi-chunk collective IO without optimization (cchunk6) Testing -- linked chunk collective IO with optimization (cchunk7) Testing -- linked chunk collective IO with optimization (cchunk7) Testing -- linked chunk collective IO with optimization (cchunk7) Testing -- linked chunk collective IO with optimization (cchunk7) Testing -- linked chunk collective IO with optimization (cchunk7) Testing -- linked chunk collective IO with optimization (cchunk7) Testing -- linked chunk collective IO transferring to multi-chunk (cchunk8) Testing -- linked chunk collective IO transferring to multi-chunk (cchunk8) Testing -- linked chunk collective IO transferring to multi-chunk (cchunk8) Testing -- linked chunk collective IO transferring to multi-chunk (cchunk8) Testing -- linked chunk collective IO transferring to multi-chunk (cchunk8) Testing -- linked chunk collective IO transferring to multi-chunk (cchunk8) Testing -- multiple chunk collective IO with optimization (cchunk9) Testing -- multiple chunk collective IO with optimization (cchunk9) Testing -- multiple chunk collective IO with optimization (cchunk9) Testing -- multiple chunk collective IO with optimization (cchunk9) Testing -- multiple chunk collective IO with optimization (cchunk9) Testing -- multiple chunk collective IO with optimization (cchunk9) Testing -- multiple chunk collective IO transferring to independent IO (cchunk10) Testing -- multiple chunk collective IO transferring to independent IO (cchunk10) Testing -- multiple chunk collective IO transferring to independent IO (cchunk10) Testing -- multiple chunk collective IO transferring to independent IO (cchunk10) Testing -- multiple chunk collective IO transferring to independent IO (cchunk10) Testing -- multiple chunk collective IO transferring to independent IO (cchunk10) Testing -- collective irregular contiguous write (ccontw) Testing -- collective irregular contiguous write (ccontw) Testing -- collective irregular contiguous write (ccontw) Testing -- collective irregular contiguous write (ccontw) Testing -- collective irregular contiguous write (ccontw) Testing -- collective irregular contiguous write (ccontw) Testing -- collective irregular contiguous read (ccontr) Testing -- collective irregular contiguous read (ccontr) Testing -- collective irregular contiguous read (ccontr) Testing -- collective irregular contiguous read (ccontr) Testing -- collective irregular contiguous read (ccontr) Testing -- collective irregular contiguous read (ccontr) Testing -- collective irregular simple chunk write (cschunkw) Testing -- collective irregular simple chunk write (cschunkw) Testing -- collective irregular simple chunk write (cschunkw) Testing -- collective irregular simple chunk write (cschunkw) Testing -- collective irregular simple chunk write (cschunkw) Testing -- collective irregular simple chunk write (cschunkw) Testing -- collective irregular simple chunk read (cschunkr) Testing -- collective irregular simple chunk read (cschunkr) Testing -- collective irregular simple chunk read (cschunkr) Testing -- collective irregular simple chunk read (cschunkr) Testing -- collective irregular simple chunk read (cschunkr) Testing -- collective irregular simple chunk read (cschunkr) Testing -- collective irregular complex chunk write (ccchunkw) Testing -- collective irregular complex chunk write (ccchunkw) Testing -- collective irregular complex chunk write (ccchunkw) Testing -- collective irregular complex chunk write (ccchunkw) Testing -- collective irregular complex chunk write (ccchunkw) Testing -- collective irregular complex chunk write (ccchunkw) Testing -- collective irregular complex chunk read (ccchunkr) Testing -- collective irregular complex chunk read (ccchunkr) Testing -- collective irregular complex chunk read (ccchunkr) Testing -- collective irregular complex chunk read (ccchunkr) Testing -- collective irregular complex chunk read (ccchunkr) Testing -- collective irregular complex chunk read (ccchunkr) Testing -- null dataset test (null) Testing -- null dataset test (null) Testing -- null dataset test (null) Testing -- null dataset test (null) Testing -- null dataset test (null) Testing -- null dataset test (null) Testing -- I/O mode confusion test -- hangs quickly on failure (I/Omodeconf) Testing -- I/O mode confusion test -- hangs quickly on failure (I/Omodeconf) Testing -- I/O mode confusion test -- hangs quickly on failure (I/Omodeconf) Testing -- I/O mode confusion test -- hangs quickly on failure (I/Omodeconf) Testing -- I/O mode confusion test -- hangs quickly on failure (I/Omodeconf) Testing -- I/O mode confusion test -- hangs quickly on failure (I/Omodeconf) Testing -- round robin object header flush confusion test (rrobjflushconf) Testing -- round robin object header flush confusion test (rrobjflushconf) Testing -- round robin object header flush confusion test (rrobjflushconf) Testing -- round robin object header flush confusion test (rrobjflushconf) Testing -- round robin object header flush confusion test (rrobjflushconf) Testing -- round robin object header flush confusion test (rrobjflushconf) Testing -- test lower dim size comp in span tree to mpi derived type (tldsc) Testing -- test lower dim size comp in span tree to mpi derived type (tldsc) Testing -- test lower dim size comp in span tree to mpi derived type (tldsc) Testing -- test lower dim size comp in span tree to mpi derived type (tldsc) Testing -- test lower dim size comp in span tree to mpi derived type (tldsc) Testing -- test lower dim size comp in span tree to mpi derived type (tldsc) *** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x0000000002798920 ***
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x000000000136a330 ***
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x00000000020e7890*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x00000000024d4370 ***
*** glibc detected *** ./testphdf5: double free or corruption (!prev): 0x0000000002452f30 ***
======= Backtrace: =========
/lib64/libc.so.6[0x38d7475916]
/lib64/libc.so.6[0x38d7478443]
/apps/developers/libraries/intelmpi/4.0.3.008/1/bit-64-intel-12.1.5.339/lib/libmpi.so.4(MPID_Segment_free+0xa)[0x7f9e57a6696a]
/apps/developers/libraries/intelmpi/4.0.3.008/1/bit-64-intel-12.1.5.339/lib/libmpi.so.4(MPID_Dataloop_create_struct+0x6b7)[0x7f5b23221af7]
/apps/developers/libraries/intelmpi/4.0.3.008/1/bit-64-intel-12.1.5.339/lib/libmpi.so.4(MPID_Dataloop_create+0x63f)[0x7f9e579829cf]
/apps/developers/libraries/intelmpi/4.0.3.008/1/bit-64-intel-12.1.5.339/lib/libmpi.so.4(MPID_Type_commit+0x91)[0x7fe49f5f3851]
/apps/developers/libraries/intelmpi/4.0.3.008/1/bit-64-intel-12.1.5.339/lib/libmpi.so.4(MPID_Type_commit+0x91)[0x7f9e57a0d851]
/apps/developers/libraries/intelmpi/4.0.3.008/1/bit-64-intel-12.1.5.339/lib/libmpi.so.4(MPI_Type_commit+0x1b2)[0x7fe885b1f622./testphdf5[0x49eec6]
./testphdf5[0x49eec6./testphdf5[0x49b528]
./testphdf5[0x4a23b6./testphdf5[0x4413d9]
./testphdf5./testphdf5[0x43fae5]
./testphdf5[0x./testphdf5[0x44cea7]
./testphdf5[0x410b13]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x38d741ecdd======= Memory map: ========
======= Memory map: ========
/lib64/libc.so.6(__libc_start_main+0xfd)[0x38d741ecdd]
./testphdf5[0x410279]
======= Memory map: ========
00400000-007d4000 r-xp 00000000 fd:04 2757697 /scratch/issmcd/testing/hdf5-1.8.9/testpar/testphdf5
009d3000-009e0000 rw-p 003d3000 fd:04 2757697 /scratch/issmcd/testing/hdf5-1.8.9/testpar/testphdf5
009e0000-009e5000 rw-p 00000000 00:00 0 012c9000-065e3000 rw-p 00000000 00:00 0 [heap]
3385800000-3385816000 r-xp 00000000 fd:00 917507 /lib64/libgcc_s-4.4.6-20120305.so.1
3385816000-3385a15000 ---p 00016000 fd:00 917507 /lib64/libgcc_s-4.4.6-20120305.so.1
3385a15000-3385a16000 rw-p 00015000 fd:00 917507 /lib64/libgcc_s-4.4.6-20120305.so.1
38d7000000-38d7020000 r-xp 00000000 fd:00 917514 /lib64/ld-2.12.so
38d721f000-38d7220000 r--p 0001f000 fd:00 917514 /lib64/ld-2.12.so
38d7220000-38d7221000 rw-p 00020000 fd:00 917514 /lib64/ld-2.12.so
38d7221000-38d7222000 rw-p 00000000 00:00 0 38d74000400000-007d4000 r-xp 00000000 fd:04 2757697 /scratch/issmcd/testing/hdf5-1.8.9/testpar/testphdf5
009d3000-009e0000 rw-p 003d3000 fd:04 2757697 /scratch/issmcd/testing/hdf5-1.8.9/testpar/testphdf5
009e0000-009e5000 rw-p 00000000 00:00 0 00d00000-035e4000 rw-p 00000000 00:00 0 [heap]
3385800000-3385816000 r-xp 00000000 fd:00 917507 /lib64/libgcc_s-4.4.6-20120305.so.1
3385816000-3385a15000 ---p 00016000 fd:00 917507 /lib64/libgcc_s-4.4.6-20120305.so.1
3385a15000-3385a16000 rw-p 00015000 fd:00 917507 /lib64/libgcc_s-4.4.6-20120305.so.1
38d7000000-38d7020000 r-xp 00000000 fd:00 917514 /lib64/ld-2.12.so
38d721f000-38d7220000 r--p 0001f000 fd:00 917514 /lib64/ld-2.12.so
38d7220000-38d7221000 rw-p 00020000 fd:00 917514 /lib64/ld-2.12.so
38d7221000-38d7222000 rw-p 00000000 00:00 0 38d7400000-38d7589000 r-xp 00000000 fd:00 917521 /lib64/libc-2.12.so
38d7589000-38d7789000 ---p 00189000 fd:00 917521 /lib64/libc-2.12.so
38d7789000-38d778d000 r--p 00189000 fd:00 917521 /lib64/libc-2.12.so
38d778d000-38d778e000 rw-p 0018d000 fd:00 917521 /lib64/libc-2.12.so
38d778e000-38d7793000 rw-p 00000000 00:00 0 38d7800000-38d7883000 r-xp 00000000 fd:00 917531 /lib64/libm-2.12.so
38d7883000-38d7a82000 ---p 00083000 fd:00 917531 /lib64/libm-2.12.so
38d7a82000-38d7a83000 r--p 00082000 fd:00 917531 /lib64/libm-2.12.so
38d7a83000-38d7a84000 rw-p 00083000 fd:00 917531 /lib64/libm-2.12.so
38d7c00000-38d7c02000 r-xp 00000000 fd:00 917536 /lib64/libdl-2.12.so
38d7c02000-38d7e02000 ---p 00002000 fd:00 917536 /lib64/libdl-2.12.so
38d7e02000-38d7e03000 r--p 00002000 fd:00 917536 /lib64/libdl-2.12.so
38d7e03000-38d7e04000 rw-p 00003000 fd:00 917536 /lib64/libdl-2.12.so
38d8000000-38d8017000 r-xp 00000000 fd:00 917527 /lib64/libpthread-2.12.so
38d8017000-38d8217000 ---p 00017000 fd:00 917527 /lib64/libpthread-2.12.so
38d8217000-38d8218000 r--p 00017000 fd:00 917527 /lib64/libpthread-2.12.so
38d8218000-38d8219000 rw-p 00018000 fd:00 917527 /lib64/libpthread-2.12.so
38d8219000-38d821d000 rw-p 00000000 00:00 0 38d8400000-38d8415000 r-xp 00000000 fd:00 917525 /lib64/libz.so.1.2.3
38d8415000-38d8614000 ---p 00015000 fd:00 917525 /lib64/libz.so.1.2.3
38d8614000-38d8615000 r--p 00014000 fd:00 917525 /lib64/libz.so.1.2.3
38d8615000-38d8616000 rw-p 00015000 fd:00 917525 /lib64/libz.so.1.2.3
38d8800000-38d8807000 r-xp 00000000 fd:00 917537 /lib64/librt-2.12.so
38d8807000-38d8a06000 ---p 00007000 fd:00 917537 /lib64/librt-2.12.so
38d8a06000-38d8a07000 r--p 00006000 fd:00 917537 /lib64/librt-2.12.so
38d8a07000-38d8a08000 rw-p 00007000 fd:00 917537 /lib64/librt-2.12.so
7fe49a547000-7fe49c101000 rw-p 00000000 00:00 0 7fe49c101000-7fe49f2e3000 rw-s 00000000 00:10 3655858 /dev/shm/Intel_MPI_nXrSJI (deleted)
7fe49f2e3000-7fe49f2e7000 rw-p 00000000 00:00 0 7fe49f2e7000-7fe49f314000 r-xp 00000000 00:2d 6701796 /apps2/developers/libraries/intelmpi/4.0.3.008/1/default/intel64/lib/libmpigf.so.4.0
7fe49f314000-7fe49f414000 ---p 0002d000 00:2d 6701796 /apps2/developers/libraries/intelmpi/4.0.3.008/1/default/intel64/lib/libmpigf.so.4.0
7fe49f414000-7fe49f415000 rw-p 0002d000 00:2d 6701796 /apps2/developers/libraries/intelmpi/4.0.3.008/1/default/intel64/lib/libmpigf.so.4.0
7fe49f415000-7fe49f779000 r-xp 00000000 00:2d 6701772 /apps2/developers/libraries/intelmpi/4.0.3.008/1/default/intel64/lib/libmpi.so.4.0APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)
