HDF5 and memory mapped files on Windows

Hdf-forum archives
1

I was reading about file drivers and saw this:

H5FD_CORE: This driver performs I/O directly to memory and can be used to create small temporary files that never exist on permanent storage.

Does this mean if I create a memory mapped file I can configure hdf5 to use the memory mapped file for the storage?

If so how do I do this?

2

Hi Andrew,

I read it as the other way around. If you use this property, it will create a mmaped() file.

You can set the backing_store to true and it will write/flush to disk.

A fortran example would be along the lines of:

call h5open_f(ierr)

call h5pcreate_f(H5P_FILE_ACCESS_F, p_id, ierr)

call h5pset_fapl_core_f(p_id, 1048576, .true., ierr)

call h5fcreate_f(filename, H5F_ACC_TRUNC_F, f_id, ierr, &
                         access_prp = p_id)

Hope this helps.

Timothy

···

________________________________
From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of Ryland, Andrew <Andrew.Ryland@keithley.com>
Sent: Thursday, March 10, 2016 8:42 AM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 and memory mapped files on Windows

I was reading about file drivers and saw this:

H5FD_CORE: This driver performs I/O directly to memory and can be used to create small temporary files that never exist on permanent storage.

Does this mean if I create a memory mapped file I can configure hdf5 to use the memory mapped file for the storage?

If so how do I do this?

3

Timothy,

Thanks for the response. I read it the same way you did in that the library will create the "in memory" file. But is it a memory mapped file or an array of sorts? Because of my lack of understanding here I asked the question about if I created a memory mapped file then could I have the hdf5 library use that.

Andrew

4

My understanding is that this is NOT using mmap calls to implement the CORE driver.

The core driver uses HDF5 library internal functionality to capture all write calls in a malloc'd (and realloc'd) buffer (of sorts).

When the file is closed, if you set backing store, that buffer gets written to disk in a single I/O request. Thats because the buffer itself is a bit-for-bit image of an actual on-disk HDF5 file.

On read, if you use core VFD, then the whole file gets read into memory in a single I/O request and then all operations on the "file" there after just interacting with the buffer in memory.

Again, no mmap calls here. There is no memory mapping of the file going on.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Ryland, Andrew" <Andrew.Ryland@keithley.com<mailto:Andrew.Ryland@keithley.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Thursday, March 10, 2016 8:27 AM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] HDF5 and memory mapped files on Windows

Timothy,

Thanks for the response. I read it the same way you did in that the library will create the “in memory” file. But is it a memory mapped file or an array of sorts? Because of my lack of understanding here I asked the question about if I created a memory mapped file then could I have the hdf5 library use that.

Andrew

5

I neglected to mention, you could implement a new HDF5 Virtual File Driver (VFD), that uses mmap. That would be a great contribution to HDF5 I think.

Mark

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Miller, Mark C." <miller86@llnl.gov<mailto:miller86@llnl.gov>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Thursday, March 10, 2016 9:20 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] HDF5 and memory mapped files on Windows

My understanding is that this is NOT using mmap calls to implement the CORE driver.

The core driver uses HDF5 library internal functionality to capture all write calls in a malloc'd (and realloc'd) buffer (of sorts).

When the file is closed, if you set backing store, that buffer gets written to disk in a single I/O request. Thats because the buffer itself is a bit-for-bit image of an actual on-disk HDF5 file.

On read, if you use core VFD, then the whole file gets read into memory in a single I/O request and then all operations on the "file" there after just interacting with the buffer in memory.

Again, no mmap calls here. There is no memory mapping of the file going on.

Mark

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org<mailto:hdf-forum-bounces@lists.hdfgroup.org>> on behalf of "Ryland, Andrew" <Andrew.Ryland@keithley.com<mailto:Andrew.Ryland@keithley.com>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Thursday, March 10, 2016 8:27 AM
To: "hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>" <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] HDF5 and memory mapped files on Windows

Timothy,

Thanks for the response. I read it the same way you did in that the library will create the “in memory” file. But is it a memory mapped file or an array of sorts? Because of my lack of understanding here I asked the question about if I created a memory mapped file then could I have the hdf5 library use that.

Andrew