Hi Albert, Dana,
Thanks for your replies. I give the output of uname below
Linux 3.0.74-0.6.10-default #1 SMP x86_64 x86_64 x86_64 GNU/Linux
I was earlier using v1.8.9 and then tried using v1.8.11. I have given
the libhdf5.settings file for both versions below. I downloaded the
library files of v1.8.11 from the HDF5 website, so haven't compiled
anything. v1.8.9 has been compiled by the system administrators.
···
-------------------
SUMMARY OF THE HDF5 CONFIGURATION
=================================
General Information:
-------------------
HDF5 Version: 1.8.9
Configured on: Thu Nov 29 11:11:42 CET 2012
Configured by: wwn@qa01
Configure mode: production
Host system: x86_64-unknown-linux-gnu
Uname information: Linux qa01 3.0.42-0.7-default #1 SMP
Tue Oct 9 11:58:45 UTC 2012 (a8dc443) x86_64 x86_64 x86_64 GNU/Linux
Byte sex: little-endian
Libraries:
Installation point:
/afs/rzg/common/soft/hdf5/1.8.9/amd64_sles11/intel/12.1/serial
Compiling Options:
------------------
Compilation Mode: production
C Compiler:
/afs/@cell/common/soft/intel/ics/2011.0.013/12.1/bin/intel64/bin/icc (
Intel(R) C Intel(R) 64 Compiler Version 12.1 Build 20120928)
CFLAGS: -fPIC
H5_CFLAGS: -std=c99 -O3
AM_CFLAGS:
CPPFLAGS:
H5_CPPFLAGS: -D_POSIX_C_SOURCE=199506L -DNDEBUG
-UH5_DEBUG_API
AM_CPPFLAGS: -D_LARGEFILE_SOURCE
-D_LARGEFILE64_SOURCE -D_BSD_SOURCE
Shared C Library: yes
Static C Library: yes
Statically Linked Executables: no
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS:
Extra libraries: -lz -lrt -lm
Archiver: ar
Ranlib: ranlib
Debugged Packages:
API Tracing: no
Languages:
----------
Fortran: yes
Fortran Compiler:
/afs/@cell/common/soft/intel/ics/2011.0.013/12.1/bin/intel64/bin/ifort (
Intel(R) Fortran Intel(R) 64 Compiler Version 12.1 Build 20120928)
Fortran 2003 Compiler: yes
Fortran Flags: -fPIC
H5 Fortran Flags: -O3
AM Fortran Flags:
Shared Fortran Library: yes
Static Fortran Library: yes
C++: yes
C++ Compiler:
/afs/@cell/common/soft/intel/ics/2011.0.013/12.1/bin/intel64/bin/icpc (
Intel(R) C++ Intel(R) 64 Compiler Version 12.1 Build 20120928)
C++ Flags:
H5 C++ Flags:
AM C++ Flags:
Shared C++ Library: yes
Static C++ Library: yes
Features:
---------
Parallel HDF5: no
High Level library: yes
Threadsafety: no
Default API Mapping: v18
With Deprecated Public Symbols: yes
I/O filters (external): deflate(zlib)
I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset
MPE: no
Direct VFD: no
dmalloc: no
Clear file buffers before write: yes
Using memory checker: no
Function Stack Tracing: no
GPFS: no
Strict File Format Checks: no
Optimization Instrumentation: no
Large File Support (LFS): yes
-----------------------------------------------
v1.8.11
***************************************
SUMMARY OF THE HDF5 CONFIGURATION
=================================
General Information:
-------------------
HDF5 Version: 1.8.11
Configured on: Wed May 8 16:20:56 CDT 2013
Configured by: hdftest@koala
Configure mode: production
Host system: x86_64-unknown-linux-gnu
Uname information: Linux koala 2.6.18-348.1.1.el5 #1 SMP
Tue Jan 22 16:19:19 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
Byte sex: little-endian
Libraries: static, shared
Installation point:
/mnt/scr1/pre-release/hdf5/v1811/thg-builds/koala
Compiling Options:
------------------
Compilation Mode: production
C Compiler: /usr/bin/gcc ( gcc (GCC) 4.1.2 20080704 )
CFLAGS:
H5_CFLAGS: -std=c99 -pedantic -Wall -Wextra
-Wundef -Wshadow -Wpointer-arith -Wbad-function-cast -Wcast-qual
-Wcast-align -Wwrite-strings -Wconversion -Wag
gregate-return -Wstrict-prototypes -Wmissing-prototypes
-Wmissing-declarations -Wredundant-decls -Wnested-externs -Winline
-Wno-long-long -Wfloat-equal -Wmissing-format-attribut
e -Wmissing-noreturn -Wpacked -Wdisabled-optimization -Wformat=2
-Wunreachable-code -Wendif-labels -Wdeclaration-after-statement
-Wold-style-definition -Winvalid-pch -Wvariadic-
macros -Wnonnull -Winit-self -Wmissing-include-dirs -Wswitch-default
-Wswitch-enum -Wunused-macros -Wunsafe-loop-optimizations -Wc++-compat
-Wvolatile-register-var -O3 -fomit-fr
ame-pointer -finline-functions
AM_CFLAGS:
CPPFLAGS:
H5_CPPFLAGS: -D_POSIX_C_SOURCE=199506L -DNDEBUG
-UH5_DEBUG_API
AM_CPPFLAGS:
-I/mnt/hdf/packages/szip/shared/encoder/Linux2.6-x86_64-gcc/include
-D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_BSD_SOURCE
Shared C Library: yes
Static C Library: yes
Statically Linked Executables: yes
LDFLAGS:
H5_LDFLAGS:
AM_LDFLAGS:
-L/mnt/hdf/packages/szip/shared/encoder/Linux2.6-x86_64-gcc/lib
Extra libraries: -lsz -lz -lrt -ldl -lm
Archiver: ar
Ranlib: ranlib
Debugged Packages:
API Tracing: no
Languages:
----------
Fortran: yes
Fortran Compiler: /usr/bin/gfortran ( GNU Fortran (GCC)
4.1.2 20080704 )
Fortran 2003 Compiler: no
Fortran Flags:
H5 Fortran Flags:
AM Fortran Flags:
Shared Fortran Library: yes
Static Fortran Library: yes
C++: yes
C++ Compiler: /usr/bin/g++ ( g++ (GCC) 4.1.2 20080704 )
C++ Flags:
H5 C++ Flags:
AM C++ Flags:
Shared C++ Library: yes
Static C++ Library: yes
Features:
---------
Parallel HDF5: no
High Level library: yes
Threadsafety: no
Default API Mapping: v18
With Deprecated Public Symbols: yes
I/O filters (external): deflate(zlib),szip(encoder)
I/O filters (internal): shuffle,fletcher32,nbit,scaleoffset
MPE: no
Direct VFD: no
dmalloc: no
Clear file buffers before write: yes
Using memory checker: no
Function Stack Tracing: no
GPFS: no
Strict File Format Checks: no
Optimization Instrumentation: no
Large File Support (LFS): yes
****************************************
I had another query. What value does H5P_DEFAULT contain and where is
the value for this variable set. I was not able to find it from the
v1.8.9 source, please let me know. I ask, since the code I am working
with, sets the file access to H5P_DEFAULT if MPI is not being used, else
its set to H5Pset_fapl_mpio.
Please let me know if you need more information.
Thanks
Nitya
-----------------------------
Message: 4
Date: Tue, 2 Jul 2013 14:33:07 -0500
From: Albert Cheng <acheng@hdfgroup.org>
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] HDF5 - reading files larger than 2GB
Message-ID: <4C52ED59-20DB-4F48-A642-17CF4202D3BB@hdfgroup.org>
Content-Type: text/plain; charset="iso-8859-1"
Hi Nitya,
Can you also give more information about the machine in which you ran your application
that got the failures? E.g.,
1. OS version ("uname -a" usually shows it.)
1.1. For Mac, try "sw_vers".
2. Did you build the HDF5 library from source?
2.1. if yes, please send a copy of the output of configure.
2.2. If the output of configure is not available, please send a copy of libhdf5.settings.
You can usually find "libhdf5.settings" in the same directory where you find the
HDF5 library, libhdf5.a or libhdf5.so.
This will help us to try to repeat your problems in the appropriate platform.
Thanks.
-Albert Change
THG Staff
On Jul 2, 2013, at 11:20 AM, Dana Robinson <derobins@hdfgroup.org> wrote:
Hi Nitya,
I'll take a look into this. It's a holiday weekend here, so I probably won't get to this until next week.
Dana
On Mon, Jul 1, 2013 at 1:37 PM, Nitya Hariharan <nty@mpa-garching.mpg.de> wrote:
Hi,I am having some trouble reading in a HDF5 file that is ~5GB. I was
initially having some problems even writing out a file this large and
looked at some postings in the HDF5 forum related to this.I tried using the serial version of the HDF5 library, v1.8.9 and was
able to write out a file of size ~5GB. However, when I try to read it
back in, I get the following error.----------------
#000: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dio.c line
182 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#001: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dio.c line
550 in H5D__read(): can't read data
major: Dataset
minor: Read failed
#002: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dcontig.c
line 543 in H5D__contig_read(): contiguous read failed
major: Dataset
minor: Read failed
#003: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dselect.c
line 278 in H5D__select_read(): read error
major: Dataspace
minor: Read failed
#004: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dselect.c
line 213 in H5D__select_io(): read error
major: Dataspace
minor: Read failed
#005: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dcontig.c
line 873 in H5D__contig_readvv(): can't perform vectorized sieve buffer read
major: Dataset
minor: Can't operate on object
#006: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5V.c line
1457 in H5V_opvv(): can't perform operation
major: Internal error (too specific to document in detail)
minor: Can't operate on object
#007: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Dcontig.c
line 674 in H5D__contig_readvv_sieve_cb(): block read failed
major: Dataset
minor: Read failed
#008: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Fio.c line
113 in H5F_block_read(): read through metadata accumulator failed
major: Low-level I/O
minor: Read failed
#009: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5Faccum.c
line 258 in H5F_accum_read(): driver read request failed
major: Low-level I/O
minor: Read failed
#010: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5FDint.c
line 142 in H5FD_read(): driver read request failed
major: Virtual File Layer
minor: Read failed
#011: /home/hdftest/snapshots-bin-hdf5_1_8_11/current/src/H5FDsec2.c
line 725 in H5FD_sec2_read(): file read failed: time = Mon Jul 1
20:21:57 2013
, filename = '/tmp/file.hdf5', file descriptor = 5, errno = 14, error
message = 'Bad address', buf = 0x2ae9619ea010, total read size =
4677466176, bytes this sub-read = 4677466176, bytes actually read =
18446744073709551615, offset = 744468544
-----------------I looked at the forum again and saw this posting, which mentioned that
there was a bug fix done for POSIX issues.http://mail.lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/2012-December/006348.html
I was using v1.8.9, but thought of using the latest 1.8.11 to rule out
any issues with the HDF5 version I was using. However, I still get the
above error.Please could someone provide some feedback on why this is happening. If
I am able to write out such a large file, I should be able to read it in
as well? Ofcourse, small sized files work perfectly fine in my application.Since the library I am using, that in turn calls the HDF5 routines,
needs the interface of v1.6 of the HDF5 library, I use the flag
H5_USE_16_API while compiling. Would this in anyway cause the problem.Thanks in advance.
--
RegardsNitya
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org