Relative Path for External Storage?

Hdf-forum archives
1

Hello Everyone,

I am trying to use the HDF5 API
H5Pset_external (hid_t plist, external filename, off_t offset, hsize_t size)
to create a dataset with external storage files.

However, it seems only absolute file path works. So, I have to rebuild the
dataset every time when I move data to a different location or try to access
data in external HDD from different PC. Is there any method could let me set
the filename relative to the host .h5 file path? I try to set the external
file path without starting with '\' or '/'. But it does not work.

Thanks,

LittleBigBrain

···

--
View this message in context: http://hdf-forum.184993.n3.nabble.com/Relative-Path-for-External-Storage-tp1004629p1004629.html
Sent from the hdf-forum mailing list archive at Nabble.com.

2

I've got a problem with excessive memory leakage using HDF5 in a Fortran
program. Here's a simple example of adding a large fixed size matrix to
a H5 file. It Iterates through a loop where the dataspace is enlarged,
another matrix is added to the file and the dataspace is closed.
Shouldn't it free all the memory at the end of each loop and return to a
certain base memory footprint? Instead it continues to grow throughout
the process.

Jack Berkery
GE Energy Applications
Schenectady, NY

!

···

************************************************************************
******
! *
! * Project: memtest.f90
! * Subject: Test appending a matrix to H5 file.
! *
!
************************************************************************
******

program main
  
    use HDF5

    implicit none

    integer :: i, j, h5err
    integer :: nrows = 1000
    integer :: ncols = 500
    real, dimension(500,1000) :: matrix

    integer(HID_T) :: file ! File id
    integer(HID_T) :: hdata ! Group id for storing hourly data
    integer(HID_T) :: dset ! Dataset id
    integer(HID_T) :: dspc ! Dataspace id
    integer(HID_T) :: dprp ! Dataproperties id
    integer(HID_T) :: dtyp ! Datatype id
    integer(HID_T) :: mset ! Memory Dataset id
    integer(HID_T) :: mspc ! Memory Dataspace id

    ! Variables for fetching dimension sizes from dataspaces within the
file.
    integer(HSIZE_T) :: dimd(2), maxdim(2)

    ! Initialize FORTRAN interface.
    call H5open_f ( h5err )
    ! Create a new file using default properties.
    call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err )
    ! Hourly data - Stores info about the settings at a given year.
    call H5Gcreate_f ( file, "HData", hdata, h5err )
    ! Create dataspace to store expandable matrix of values
    call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err,
(/H5S_UNLIMITED_F, nrows/) )
    ! Define the datatype for this set
    call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err )
    ! Create property objects
    call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err )
    ! Set properties - chunked (one at a time)
    call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err )
    ! Set properties - using compression level 7 [0-9] (9 most
compression)
    call H5Pset_deflate_f ( dprp, 7, h5err )
    ! Create the dataset with modified properties.
    call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp )
    ! Create memory dataspaces to enable chunked write
    call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err )

    ! Fill in data variables
    do i = 1, ncols
        do j = 1, nrows
            matrix (i,j) = 100.0 * real(i) + real(j)
        end do
    end do

    ! Write dataspace in a loop
    ! New data has to be appended to the dataset already in the file.
  
    do i = 1, 30
        print *, "Loop", i
        ! Get a new pointer to a copy of the dataspace for this set
        call H5Dget_space_f ( dset, dspc, h5err )
        ! Figure out how big is the dataset in the file.
        call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err )
        ! Extend size of the dataset in the file
        call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err )
        ! Close the dataspace now that it has been extended
        call H5Sclose_f ( dspc, h5err )
        ! Fetch the dataspace of the extended dataset
        call H5Dget_space_f ( dset, dspc, h5err )
        ! Select the appropriate slab in the extended dataset
        call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, &
                 (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix) )
        ! Write the data from memory to the selected slab
        call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix),
h5err, &
                 mspc, dspc, H5P_DEFAULT_F )
        ! Close the extended dataspace to free memory
        call H5Sclose_f ( dspc, h5err )
        call sleep (3) ! for a better view of process explorer
graph
    end do

! Cleanup, end access and release resources

    call sleep (5) ! for a better view of process
explorer graph
    call H5Dclose_f ( dset, h5err ) ! dataset
   !call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside loop)
    call H5Pclose_f ( dprp, h5err ) ! properties
    call H5Tclose_f ( dtyp, h5err ) ! data types

    call H5Fclose_f ( file, h5err ) ! file id
    call H5close_f ( h5err ) ! fortran interface
    call sleep (5)

end program main

3

John,

I couldn't compile your program on Mac or Linux due to the usage of (/..,.../) in the function calls.

I also don't see anything obviously wrong. Could you please comment out h5pset_deflate_f call, rerun the program and check
if you still see memory growth?

Thank you!

Elena

···

On Jul 29, 2010, at 4:01 PM, Berkery, John (GE Energy, Non-GE) wrote:

I've got a problem with excessive memory leakage using HDF5 in a Fortran
program. Here's a simple example of adding a large fixed size matrix to
a H5 file. It Iterates through a loop where the dataspace is enlarged,
another matrix is added to the file and the dataspace is closed.
Shouldn't it free all the memory at the end of each loop and return to a
certain base memory footprint? Instead it continues to grow throughout
the process.

Jack Berkery
GE Energy Applications
Schenectady, NY

!
************************************************************************
******
! *
! * Project: memtest.f90
! * Subject: Test appending a matrix to H5 file.
! *
!
************************************************************************
******

program main
  
   use HDF5

   implicit none

   integer :: i, j, h5err
   integer :: nrows = 1000
   integer :: ncols = 500
   real, dimension(500,1000) :: matrix

   integer(HID_T) :: file ! File id
   integer(HID_T) :: hdata ! Group id for storing hourly data
   integer(HID_T) :: dset ! Dataset id
   integer(HID_T) :: dspc ! Dataspace id
   integer(HID_T) :: dprp ! Dataproperties id
   integer(HID_T) :: dtyp ! Datatype id
   integer(HID_T) :: mset ! Memory Dataset id
   integer(HID_T) :: mspc ! Memory Dataspace id

   ! Variables for fetching dimension sizes from dataspaces within the
file.
   integer(HSIZE_T) :: dimd(2), maxdim(2)

   ! Initialize FORTRAN interface.
   call H5open_f ( h5err )
   ! Create a new file using default properties.
   call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err )
   ! Hourly data - Stores info about the settings at a given year.
   call H5Gcreate_f ( file, "HData", hdata, h5err )
   ! Create dataspace to store expandable matrix of values
   call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err,
(/H5S_UNLIMITED_F, nrows/) )
   ! Define the datatype for this set
   call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err )
   ! Create property objects
   call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err )
   ! Set properties - chunked (one at a time)
   call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err )
   ! Set properties - using compression level 7 [0-9] (9 most
compression)
   call H5Pset_deflate_f ( dprp, 7, h5err )
   ! Create the dataset with modified properties.
   call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp )
   ! Create memory dataspaces to enable chunked write
   call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err )

   ! Fill in data variables
   do i = 1, ncols
       do j = 1, nrows
           matrix (i,j) = 100.0 * real(i) + real(j)
       end do
   end do

   ! Write dataspace in a loop
   ! New data has to be appended to the dataset already in the file.
  
   do i = 1, 30
       print *, "Loop", i
       ! Get a new pointer to a copy of the dataspace for this set
       call H5Dget_space_f ( dset, dspc, h5err )
       ! Figure out how big is the dataset in the file.
       call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err )
       ! Extend size of the dataset in the file
       call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err )
       ! Close the dataspace now that it has been extended
       call H5Sclose_f ( dspc, h5err )
       ! Fetch the dataspace of the extended dataset
       call H5Dget_space_f ( dset, dspc, h5err )
       ! Select the appropriate slab in the extended dataset
       call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, &
                (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix) )
       ! Write the data from memory to the selected slab
       call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix),
h5err, &
                mspc, dspc, H5P_DEFAULT_F )
       ! Close the extended dataspace to free memory
       call H5Sclose_f ( dspc, h5err )
       call sleep (3) ! for a better view of process explorer
graph
   end do

! Cleanup, end access and release resources

   call sleep (5) ! for a better view of process
explorer graph
   call H5Dclose_f ( dset, h5err ) ! dataset
  !call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside loop)
   call H5Pclose_f ( dprp, h5err ) ! properties
   call H5Tclose_f ( dtyp, h5err ) ! data types

   call H5Fclose_f ( file, h5err ) ! file id
   call H5close_f ( h5err ) ! fortran interface
   call sleep (5)

end program main

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

We do compile the main app on Linux with such imbedded data statements
but I don't know which compiler is used. I'm using visual fortran on
Windows.

Yes, the memory allocation continues to grow without deflate but not as
much. Whereas it grew from 5.7 to 12.1 Mb before, it now grows from 5.7
to 9.9 Mb. Compression is a must in the real app though, otherwise the
H5 file would grow to an enormous size. There is very little difference
in memory allocation between compression level 1 and 7.

Attached are graphs of the program runs with and without. There is a
difference in how memory is accumulating and it is inconsistent between
iterations of the loop which is also disconcerting. One would expect a
regular stair step pattern more similar to the run with compression.

Jack Berkery
GE Energy Applications
Schenectady, NY

nocompression.jpg689×511 120 KB

compression.jpg644×511 118 KB

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Sunday, August 01, 2010 7:14 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 1.6.10 memory growth

John,

I couldn't compile your program on Mac or Linux due to the usage of
(/..,.../) in the function calls.

I also don't see anything obviously wrong. Could you please comment out
h5pset_deflate_f call, rerun the program and check if you still see
memory growth?

Thank you!

Elena
On Jul 29, 2010, at 4:01 PM, Berkery, John (GE Energy, Non-GE) wrote:

I've got a problem with excessive memory leakage using HDF5 in a
Fortran program. Here's a simple example of adding a large fixed size
matrix to a H5 file. It Iterates through a loop where the dataspace is

enlarged, another matrix is added to the file and the dataspace is

closed.

Shouldn't it free all the memory at the end of each loop and return to

a certain base memory footprint? Instead it continues to grow
throughout the process.

Jack Berkery
GE Energy Applications
Schenectady, NY

!
**********************************************************************
**
******
! *
! * Project: memtest.f90
! * Subject: Test appending a matrix to H5 file.
! *
!
**********************************************************************
**
******

program main
  
   use HDF5

   implicit none

   integer :: i, j, h5err
   integer :: nrows = 1000
   integer :: ncols = 500
   real, dimension(500,1000) :: matrix

   integer(HID_T) :: file ! File id
   integer(HID_T) :: hdata ! Group id for storing hourly data
   integer(HID_T) :: dset ! Dataset id
   integer(HID_T) :: dspc ! Dataspace id
   integer(HID_T) :: dprp ! Dataproperties id
   integer(HID_T) :: dtyp ! Datatype id
   integer(HID_T) :: mset ! Memory Dataset id
   integer(HID_T) :: mspc ! Memory Dataspace id

   ! Variables for fetching dimension sizes from dataspaces within the

file.
   integer(HSIZE_T) :: dimd(2), maxdim(2)

   ! Initialize FORTRAN interface.
   call H5open_f ( h5err )
   ! Create a new file using default properties.
   call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err )
   ! Hourly data - Stores info about the settings at a given year.
   call H5Gcreate_f ( file, "HData", hdata, h5err )
   ! Create dataspace to store expandable matrix of values
   call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err,
(/H5S_UNLIMITED_F, nrows/) )
   ! Define the datatype for this set
   call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err )
   ! Create property objects
   call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err )
   ! Set properties - chunked (one at a time)
   call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err )
   ! Set properties - using compression level 7 [0-9] (9 most
compression)
   call H5Pset_deflate_f ( dprp, 7, h5err )
   ! Create the dataset with modified properties.
   call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp )
   ! Create memory dataspaces to enable chunked write
   call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err )

   ! Fill in data variables
   do i = 1, ncols
       do j = 1, nrows
           matrix (i,j) = 100.0 * real(i) + real(j)
       end do
   end do

   ! Write dataspace in a loop
   ! New data has to be appended to the dataset already in the file.
  
   do i = 1, 30
       print *, "Loop", i
       ! Get a new pointer to a copy of the dataspace for this set
       call H5Dget_space_f ( dset, dspc, h5err )
       ! Figure out how big is the dataset in the file.
       call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err )
       ! Extend size of the dataset in the file
       call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err )
       ! Close the dataspace now that it has been extended
       call H5Sclose_f ( dspc, h5err )
       ! Fetch the dataspace of the extended dataset
       call H5Dget_space_f ( dset, dspc, h5err )
       ! Select the appropriate slab in the extended dataset
       call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, &
                (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix)

)

       ! Write the data from memory to the selected slab
       call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix),

h5err, &
                mspc, dspc, H5P_DEFAULT_F )
       ! Close the extended dataspace to free memory
       call H5Sclose_f ( dspc, h5err )
       call sleep (3) ! for a better view of process explorer
graph
   end do

! Cleanup, end access and release resources

   call sleep (5) ! for a better view of process
explorer graph
   call H5Dclose_f ( dset, h5err ) ! dataset
  !call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside

loop)

   call H5Pclose_f ( dprp, h5err ) ! properties
   call H5Tclose_f ( dtyp, h5err ) ! data types

   call H5Fclose_f ( file, h5err ) ! file id
   call H5close_f ( h5err ) ! fortran interface
   call sleep (5)

end program main

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

5

John

Thank you for your report. Could you please try with the latest 1.8.5-snap4 ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/hdf5/snapshots/v18/ and see if the problem is still there? (BTW, we stopped supporting HDF5 1.6.*)

I will enter a bug report and we will investigate.

Thank you!

Elena

···

On Aug 2, 2010, at 7:39 AM, Berkery, John (GE Energy, Non-GE) wrote:

We do compile the main app on Linux with such imbedded data statements
but I don't know which compiler is used. I'm using visual fortran on
Windows.

Yes, the memory allocation continues to grow without deflate but not as
much. Whereas it grew from 5.7 to 12.1 Mb before, it now grows from 5.7
to 9.9 Mb. Compression is a must in the real app though, otherwise the
H5 file would grow to an enormous size. There is very little difference
in memory allocation between compression level 1 and 7.

Attached are graphs of the program runs with and without. There is a
difference in how memory is accumulating and it is inconsistent between
iterations of the loop which is also disconcerting. One would expect a
regular stair step pattern more similar to the run with compression.

Jack Berkery
GE Energy Applications
Schenectady, NY

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Sunday, August 01, 2010 7:14 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 1.6.10 memory growth

John,

I couldn't compile your program on Mac or Linux due to the usage of
(/..,.../) in the function calls.

I also don't see anything obviously wrong. Could you please comment out
h5pset_deflate_f call, rerun the program and check if you still see
memory growth?

Thank you!

Elena
On Jul 29, 2010, at 4:01 PM, Berkery, John (GE Energy, Non-GE) wrote:

I've got a problem with excessive memory leakage using HDF5 in a
Fortran program. Here's a simple example of adding a large fixed size
matrix to a H5 file. It Iterates through a loop where the dataspace is

enlarged, another matrix is added to the file and the dataspace is

closed.

Shouldn't it free all the memory at the end of each loop and return to

a certain base memory footprint? Instead it continues to grow
throughout the process.

Jack Berkery
GE Energy Applications
Schenectady, NY

!
**********************************************************************
**
******
! *
! * Project: memtest.f90
! * Subject: Test appending a matrix to H5 file.
! *
!
**********************************************************************
**
******

program main
  
  use HDF5

  implicit none

  integer :: i, j, h5err
  integer :: nrows = 1000
  integer :: ncols = 500
  real, dimension(500,1000) :: matrix

  integer(HID_T) :: file ! File id
  integer(HID_T) :: hdata ! Group id for storing hourly data
  integer(HID_T) :: dset ! Dataset id
  integer(HID_T) :: dspc ! Dataspace id
  integer(HID_T) :: dprp ! Dataproperties id
  integer(HID_T) :: dtyp ! Datatype id
  integer(HID_T) :: mset ! Memory Dataset id
  integer(HID_T) :: mspc ! Memory Dataspace id

  ! Variables for fetching dimension sizes from dataspaces within the

file.
  integer(HSIZE_T) :: dimd(2), maxdim(2)

  ! Initialize FORTRAN interface.
  call H5open_f ( h5err )
  ! Create a new file using default properties.
  call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err )
  ! Hourly data - Stores info about the settings at a given year.
  call H5Gcreate_f ( file, "HData", hdata, h5err )
  ! Create dataspace to store expandable matrix of values
  call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err,
(/H5S_UNLIMITED_F, nrows/) )
  ! Define the datatype for this set
  call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err )
  ! Create property objects
  call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err )
  ! Set properties - chunked (one at a time)
  call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err )
  ! Set properties - using compression level 7 [0-9] (9 most
compression)
  call H5Pset_deflate_f ( dprp, 7, h5err )
  ! Create the dataset with modified properties.
  call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp )
  ! Create memory dataspaces to enable chunked write
  call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err )

  ! Fill in data variables
  do i = 1, ncols
      do j = 1, nrows
          matrix (i,j) = 100.0 * real(i) + real(j)
      end do
  end do

  ! Write dataspace in a loop
  ! New data has to be appended to the dataset already in the file.
  
  do i = 1, 30
      print *, "Loop", i
      ! Get a new pointer to a copy of the dataspace for this set
      call H5Dget_space_f ( dset, dspc, h5err )
      ! Figure out how big is the dataset in the file.
      call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err )
      ! Extend size of the dataset in the file
      call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err )
      ! Close the dataspace now that it has been extended
      call H5Sclose_f ( dspc, h5err )
      ! Fetch the dataspace of the extended dataset
      call H5Dget_space_f ( dset, dspc, h5err )
      ! Select the appropriate slab in the extended dataset
      call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, &
               (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix)

)

      ! Write the data from memory to the selected slab
      call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix),

h5err, &
               mspc, dspc, H5P_DEFAULT_F )
      ! Close the extended dataspace to free memory
      call H5Sclose_f ( dspc, h5err )
      call sleep (3) ! for a better view of process explorer
graph
  end do

! Cleanup, end access and release resources

  call sleep (5) ! for a better view of process
explorer graph
  call H5Dclose_f ( dset, h5err ) ! dataset
!call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside

loop)

  call H5Pclose_f ( dprp, h5err ) ! properties
  call H5Tclose_f ( dtyp, h5err ) ! data types

  call H5Fclose_f ( file, h5err ) ! file id
  call H5close_f ( h5err ) ! fortran interface
  call sleep (5)

end program main

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<nocompression.jpg><compression.jpg>_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

6

I know. That's my problem. I'm a contractor for GE and have no authority
to change any versions of libraries or compilers. I was originally
working with 1.6.5 then built 1.6.10 and saw no improvement. I attempted
to build 1.8.5 from source but the visual studio and fortran compilers
were not up to it. Are there binary distributions of fortran libs/dlls
for windows?

Jack Berkery

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Monday, August 02, 2010 11:17 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 1.6.10 memory growth

John

Thank you for your report. Could you please try with the latest
1.8.5-snap4 ftp://ftp.hdfgroup.uiuc.edu/pub/outgoing/hdf5/snapshots/v18/
and see if the problem is still there? (BTW, we stopped supporting HDF5
1.6.*)

I will enter a bug report and we will investigate.

Thank you!

Elena
On Aug 2, 2010, at 7:39 AM, Berkery, John (GE Energy, Non-GE) wrote:

We do compile the main app on Linux with such imbedded data statements

but I don't know which compiler is used. I'm using visual fortran on
Windows.

Yes, the memory allocation continues to grow without deflate but not
as much. Whereas it grew from 5.7 to 12.1 Mb before, it now grows from

5.7 to 9.9 Mb. Compression is a must in the real app though, otherwise

the
H5 file would grow to an enormous size. There is very little
difference in memory allocation between compression level 1 and 7.

Attached are graphs of the program runs with and without. There is a
difference in how memory is accumulating and it is inconsistent
between iterations of the loop which is also disconcerting. One would
expect a regular stair step pattern more similar to the run with

compression.

Jack Berkery
GE Energy Applications
Schenectady, NY

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org
[mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Elena Pourmal
Sent: Sunday, August 01, 2010 7:14 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 1.6.10 memory growth

John,

I couldn't compile your program on Mac or Linux due to the usage of
(/..,.../) in the function calls.

I also don't see anything obviously wrong. Could you please comment
out h5pset_deflate_f call, rerun the program and check if you still
see memory growth?

Thank you!

Elena
On Jul 29, 2010, at 4:01 PM, Berkery, John (GE Energy, Non-GE) wrote:

I've got a problem with excessive memory leakage using HDF5 in a
Fortran program. Here's a simple example of adding a large fixed size

matrix to a H5 file. It Iterates through a loop where the dataspace
is

enlarged, another matrix is added to the file and the dataspace is

closed.

Shouldn't it free all the memory at the end of each loop and return
to

a certain base memory footprint? Instead it continues to grow
throughout the process.

Jack Berkery
GE Energy Applications
Schenectady, NY

!
*********************************************************************
*
**
******
! *
! * Project: memtest.f90
! * Subject: Test appending a matrix to H5 file.
! *
!
*********************************************************************
*
**
******

program main
  
  use HDF5

  implicit none

  integer :: i, j, h5err
  integer :: nrows = 1000
  integer :: ncols = 500
  real, dimension(500,1000) :: matrix

  integer(HID_T) :: file ! File id
  integer(HID_T) :: hdata ! Group id for storing hourly data
  integer(HID_T) :: dset ! Dataset id
  integer(HID_T) :: dspc ! Dataspace id
  integer(HID_T) :: dprp ! Dataproperties id
  integer(HID_T) :: dtyp ! Datatype id
  integer(HID_T) :: mset ! Memory Dataset id
  integer(HID_T) :: mspc ! Memory Dataspace id

  ! Variables for fetching dimension sizes from dataspaces within the

file.
  integer(HSIZE_T) :: dimd(2), maxdim(2)

  ! Initialize FORTRAN interface.
  call H5open_f ( h5err )
  ! Create a new file using default properties.
  call H5Fcreate_f ( "test03.h5", H5F_ACC_TRUNC_F, file, h5err )
  ! Hourly data - Stores info about the settings at a given year.
  call H5Gcreate_f ( file, "HData", hdata, h5err )
  ! Create dataspace to store expandable matrix of values
  call H5Screate_simple_f ( 2, (/0, nrows/), dspc, h5err,
(/H5S_UNLIMITED_F, nrows/) )
  ! Define the datatype for this set
  call H5Tcopy_f ( H5T_NATIVE_REAL, dtyp, h5err )
  ! Create property objects
  call H5Pcreate_f ( H5P_DATASET_CREATE_F, dprp, h5err )
  ! Set properties - chunked (one at a time)
  call H5Pset_chunk_f ( dprp, 2, (/1, nrows/), h5err )
  ! Set properties - using compression level 7 [0-9] (9 most
compression)
  call H5Pset_deflate_f ( dprp, 7, h5err )
  ! Create the dataset with modified properties.
  call H5Dcreate_f ( hdata, "vals", dtyp, dspc, dset, h5err, dprp )
  ! Create memory dataspaces to enable chunked write
  call H5Screate_simple_f ( 2, (/ncols, nrows/), mspc, h5err )

  ! Fill in data variables
  do i = 1, ncols
      do j = 1, nrows
          matrix (i,j) = 100.0 * real(i) + real(j)
      end do
  end do

  ! Write dataspace in a loop
  ! New data has to be appended to the dataset already in the file.
  
  do i = 1, 30
      print *, "Loop", i
      ! Get a new pointer to a copy of the dataspace for this set
      call H5Dget_space_f ( dset, dspc, h5err )
      ! Figure out how big is the dataset in the file.
      call H5Sget_simple_extent_dims_f ( dspc, dimd, maxdim, h5err )
      ! Extend size of the dataset in the file
      call H5Dextend_f ( dset, dimd+(/ncols,0/), h5err )
      ! Close the dataspace now that it has been extended
      call H5Sclose_f ( dspc, h5err )
      ! Fetch the dataspace of the extended dataset
      call H5Dget_space_f ( dset, dspc, h5err )
      ! Select the appropriate slab in the extended dataset
      call H5Sselect_hyperslab_f ( dspc, H5S_SELECT_SET_F, &
               (/dimd(1),0/), (/1,1/), h5err, (/1,1/), shape(matrix)

)

      ! Write the data from memory to the selected slab
      call H5Dwrite_f ( dset, H5T_NATIVE_REAL, matrix, shape(matrix),

h5err, &
               mspc, dspc, H5P_DEFAULT_F )
      ! Close the extended dataspace to free memory
      call H5Sclose_f ( dspc, h5err )
      call sleep (3) ! for a better view of process explorer
graph
  end do

! Cleanup, end access and release resources

  call sleep (5) ! for a better view of process
explorer graph
  call H5Dclose_f ( dset, h5err ) ! dataset
!call H5Sclose_f ( dspc, h5err ) ! dataspace (released inside

loop)

  call H5Pclose_f ( dprp, h5err ) ! properties
  call H5Tclose_f ( dtyp, h5err ) ! data types

  call H5Fclose_f ( file, h5err ) ! file id
  call H5close_f ( h5err ) ! fortran interface
  call sleep (5)

end program main

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
<nocompression.jpg><compression.jpg>__________________________________
_____________ Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org