Create HDF5 file in memory

Hi,
I have my custom format where I keep all kinds of data: strings, arrays etc. I'd also like to keep chunk of data which is basically whole hdf file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file. When reading just read all that data to memory and read from it with hdf5 functions. Problem is that haven't found functions which will allow me to do that in hdf5 docs.

hope you can help me
regards
Adam

If I am understanding you correctly, you can do that using the 'core'
virtual file driver (VFD). Have a look at...

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

You need to create a file-access or file-creation properties list object
with appropriate values set for the core VFD and it should behave as you
describe.

Good luck.

Mark

···

On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:

Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.

hope you can help me
regards
Adam

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)

regards
Adam

···

If I am understanding you correctly, you can do that using the 'core'
virtual file driver (VFD). Have a look at...

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

You need to create a file-access or file-creation properties list object
with appropriate values set for the core VFD and it should behave as you
describe.

Good luck.

Mark

On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:

Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.

hope you can help me
regards
Adam

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...

Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.

Mark

···

On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:

Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)

regards
Adam

> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...

> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.

> Good luck.

> Mark

> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
>> Hi,
>> I have my custom format where I keep all kinds of data: strings, arrays
>> etc. I'd also like to keep chunk of data which is basically whole hdf
>> file. Is it possible?
>> Idea is to keep hdf5 file in memory and dump all that data to my file.
>> When reading just read all that data to memory and read from it with
>> hdf5 functions. Problem is that haven't found functions which will allow
>> me to do that in hdf5 docs.
>>
>> hope you can help me
>> regards
>> Adam
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

Hi Mark/Adam,
  The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:

  http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/

  Quincey

···

On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:

Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...

https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf

Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.

Mark

On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:

Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)

regards
Adam

> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...

> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.

> Good luck.

> Mark

> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:

Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.

hope you can help me
regards
Adam

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Excellent. Where can I find more information on this. I don't see anything like that in hdf5-1.8.9-pre1-RELEASE.txt.

regards
Adam

···

On 2012-04-24 22:34, Quincey Koziol wrote:

Hi Mark/Adam,
  The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:

  http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/

  Quincey

On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:

Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...

https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf

Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.

Mark

On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:

Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)

regards
Adam

> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...

> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.

> Good luck.

> Mark

> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:

Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.

hope you can help me
regards
Adam

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Adam,

User documentation for this feature is nearly complete and is currently in review. Note that the address below is NOT permanent; the docs will be in the usual place at www.hdfgroup.org/HDF5/doc/ once HDF5 Release 1.8.9 is final.

In the meantime, the docs are TEMPORARILY available here:

Overview:
    http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/Advanced/FileImageOperations/HDF5FileImageOperations.pdf

Reference manual entries:
The relevant reference manual entries are all cross-referenced from "See Also" in the H5PFget_file_image entry:
    http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/RM/RM_H5F.html#File-GetFileImage

Note that there is one high-level function that may suit your needs, H5LTopen_file_iamge. Several low-level H5F and H5P functions are provided for situations requiring more customization.

Please let me know if you experience any difficulty with these documents. Thanks!

-- Frank
   HDF Documentation Team
   fbaker@hdfgroup.org

···

On 24 Apr 2012, at 15:38 PM, Adam Wierzchowski wrote:

Excellent. Where can I find more information on this. I don't see anything like that in hdf5-1.8.9-pre1-RELEASE.txt.

regards
Adam

On 2012-04-24 22:34, Quincey Koziol wrote:

Hi Mark/Adam,
  The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:

  http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/

  Quincey

On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:

Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...

https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf

Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.

Mark

On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:

Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)

regards
Adam

> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...

> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.

> Good luck.

> Mark

> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:

Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.

hope you can help me
regards
Adam

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi.
I did a quick test:

////////////////////////////////////////////////////
  //hdf5
  /*file_id = H5Fopen(sHD5, H5F_ACC_RDWR, H5P_DEFAULT);
  char buf[256];
  start = clock();
  for (int i=0;i<iNumFrames;++i)
  {
    int iFrame = i+1;
    sprintf(buf, "/dset%d", iFrame);
    hid_t dataset_id = H5Dopen(file_id, buf, H5P_DEFAULT);
    H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, fPoints);
    H5Dclose(dataset_id);
  }
  printf("hdf read in %d ticks\n", clock()-start);
  H5Fclose(file_id);*/
  ////////////////////////////////////////////////////
  //hdf5 - memory
  struct stat filestatus;
  stat(sHD5, &filestatus);
  char *filebuf = new char[filestatus.st_size];
  f = fopen(sHD5, "rb");
  fread(filebuf, filestatus.st_size, 1, f);
  fclose(f);
  file_id = H5LTopen_file_image(filebuf, filestatus.st_size, 0);
  char buf[256];
  start = clock();
  for (int i=0;i<iNumFrames;++i)
  {
    int iFrame = i+1;
    sprintf(buf, "/dset%d", iFrame);
    hid_t dataset_id = H5Dopen(file_id, buf, H5P_DEFAULT);
    H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, fPoints);
    H5Dclose(dataset_id);
  }
  printf("hdf read in %d ticks\n", clock()-start);
  H5Fclose(file_id);
  delete [] filebuf;

First part (commented out) opens and reads file from disk. Second loads whole file to filebuf and opens it from there with H5LTopen_file_image. In both cases measured speed is almost exactly same.

Reading data from data sets is extremely slow. If I read same amount of data like this:

f = fopen(sMB, "rb");
  start = clock();
  for (int i=0;i<iNumFrames;++i)
  {
    fseek(f, 0, SEEK_SET);
    fseek(f, sizeof(float) * i * iNumPoints, SEEK_SET);
    fread(fPoints, iNumPoints * sizeof(float), 1, f);
  }
  printf("mb read in %d ticks\n", clock()-start);
  fclose(f);

it is almost two orders of magnitude faster. 30 (fread) versus 1444 (hdf5). I did similar test for BDB and results were near 70. I understand that reading data directly form disk will always be faster but results for HDF5 are extremely disappointing. I'm not sure if I'm doing this right.

regards
Adam

···

On 2012-04-24 23:56, Frank Baker wrote:

Adam,

User documentation for this feature is nearly complete and is currently in review. Note that the address below is NOT permanent; the docs will be in the usual place at www.hdfgroup.org/HDF5/doc/ once HDF5 Release 1.8.9 is final.

In the meantime, the docs are TEMPORARILY available here:

Overview:
     http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/Advanced/FileImageOperations/HDF5FileImageOperations.pdf

Reference manual entries:
The relevant reference manual entries are all cross-referenced from "See Also" in the H5PFget_file_image entry:
     http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/RM/RM_H5F.html#File-GetFileImage

Note that there is one high-level function that may suit your needs, H5LTopen_file_iamge. Several low-level H5F and H5P functions are provided for situations requiring more customization.

Please let me know if you experience any difficulty with these documents. Thanks!

-- Frank
    HDF Documentation Team
    fbaker@hdfgroup.org

On 24 Apr 2012, at 15:38 PM, Adam Wierzchowski wrote:

Excellent. Where can I find more information on this. I don't see anything like that in hdf5-1.8.9-pre1-RELEASE.txt.

regards
Adam

On 2012-04-24 22:34, Quincey Koziol wrote:

Hi Mark/Adam,
  The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:

  http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/

  Quincey

On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:

Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...

https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf

Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.

Mark

On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:

Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)

regards
Adam

> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...

> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore

> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.

> Good luck.

> Mark

> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:

Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.

hope you can help me
regards
Adam

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org