Hi,
I have my custom format where I keep all kinds of data: strings, arrays etc. I'd also like to keep chunk of data which is basically whole hdf file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file. When reading just read all that data to memory and read from it with hdf5 functions. Problem is that haven't found functions which will allow me to do that in hdf5 docs.
hope you can help me
regards
Adam
If I am understanding you correctly, you can do that using the 'core'
virtual file driver (VFD). Have a look at...
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
You need to create a file-access or file-creation properties list object
with appropriate values set for the core VFD and it should behave as you
describe.
Good luck.
Mark
On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.hope you can help me
regards
Adam_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)
regards
Adam
If I am understanding you correctly, you can do that using the 'core'
virtual file driver (VFD). Have a look at...
http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
You need to create a file-access or file-creation properties list object
with appropriate values set for the core VFD and it should behave as you
describe.
Good luck.
Mark
On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.hope you can help me
regards
Adam_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...
Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.
Mark
On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)regards
Adam> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.> Good luck.
> Mark
> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
>> Hi,
>> I have my custom format where I keep all kinds of data: strings, arrays
>> etc. I'd also like to keep chunk of data which is basically whole hdf
>> file. Is it possible?
>> Idea is to keep hdf5 file in memory and dump all that data to my file.
>> When reading just read all that data to memory and read from it with
>> hdf5 functions. Problem is that haven't found functions which will allow
>> me to do that in hdf5 docs.
>>
>> hope you can help me
>> regards
>> Adam
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511
Hi Mark/Adam,
The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:
http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/
Quincey
On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:
Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf
Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.Mark
On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)regards
Adam> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.> Good luck.
> Mark
> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.hope you can help me
regards
Adam_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Excellent. Where can I find more information on this. I don't see anything like that in hdf5-1.8.9-pre1-RELEASE.txt.
regards
Adam
On 2012-04-24 22:34, Quincey Koziol wrote:
Hi Mark/Adam,
The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/
Quincey
On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:
Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf
Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.Mark
On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)regards
Adam> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.> Good luck.
> Mark
> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.hope you can help me
regards
Adam_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Adam,
User documentation for this feature is nearly complete and is currently in review. Note that the address below is NOT permanent; the docs will be in the usual place at www.hdfgroup.org/HDF5/doc/ once HDF5 Release 1.8.9 is final.
In the meantime, the docs are TEMPORARILY available here:
Reference manual entries:
The relevant reference manual entries are all cross-referenced from "See Also" in the H5PFget_file_image entry:
http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/RM/RM_H5F.html#File-GetFileImage
Note that there is one high-level function that may suit your needs, H5LTopen_file_iamge. Several low-level H5F and H5P functions are provided for situations requiring more customization.
Please let me know if you experience any difficulty with these documents. Thanks!
-- Frank
HDF Documentation Team
fbaker@hdfgroup.org
On 24 Apr 2012, at 15:38 PM, Adam Wierzchowski wrote:
Excellent. Where can I find more information on this. I don't see anything like that in hdf5-1.8.9-pre1-RELEASE.txt.
regards
AdamOn 2012-04-24 22:34, Quincey Koziol wrote:
Hi Mark/Adam,
The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/
Quincey
On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:
Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf
Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.Mark
On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)regards
Adam> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.> Good luck.
> Mark
> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.hope you can help me
regards
Adam_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi.
I did a quick test:
////////////////////////////////////////////////////
//hdf5
/*file_id = H5Fopen(sHD5, H5F_ACC_RDWR, H5P_DEFAULT);
char buf[256];
start = clock();
for (int i=0;i<iNumFrames;++i)
{
int iFrame = i+1;
sprintf(buf, "/dset%d", iFrame);
hid_t dataset_id = H5Dopen(file_id, buf, H5P_DEFAULT);
H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, fPoints);
H5Dclose(dataset_id);
}
printf("hdf read in %d ticks\n", clock()-start);
H5Fclose(file_id);*/
////////////////////////////////////////////////////
//hdf5 - memory
struct stat filestatus;
stat(sHD5, &filestatus);
char *filebuf = new char[filestatus.st_size];
f = fopen(sHD5, "rb");
fread(filebuf, filestatus.st_size, 1, f);
fclose(f);
file_id = H5LTopen_file_image(filebuf, filestatus.st_size, 0);
char buf[256];
start = clock();
for (int i=0;i<iNumFrames;++i)
{
int iFrame = i+1;
sprintf(buf, "/dset%d", iFrame);
hid_t dataset_id = H5Dopen(file_id, buf, H5P_DEFAULT);
H5Dread(dataset_id, H5T_NATIVE_FLOAT, H5S_ALL, H5S_ALL, H5P_DEFAULT, fPoints);
H5Dclose(dataset_id);
}
printf("hdf read in %d ticks\n", clock()-start);
H5Fclose(file_id);
delete [] filebuf;
First part (commented out) opens and reads file from disk. Second loads whole file to filebuf and opens it from there with H5LTopen_file_image. In both cases measured speed is almost exactly same.
Reading data from data sets is extremely slow. If I read same amount of data like this:
f = fopen(sMB, "rb");
start = clock();
for (int i=0;i<iNumFrames;++i)
{
fseek(f, 0, SEEK_SET);
fseek(f, sizeof(float) * i * iNumPoints, SEEK_SET);
fread(fPoints, iNumPoints * sizeof(float), 1, f);
}
printf("mb read in %d ticks\n", clock()-start);
fclose(f);
it is almost two orders of magnitude faster. 30 (fread) versus 1444 (hdf5). I did similar test for BDB and results were near 70. I understand that reading data directly form disk will always be faster but results for HDF5 are extremely disappointing. I'm not sure if I'm doing this right.
regards
Adam
On 2012-04-24 23:56, Frank Baker wrote:
Adam,
User documentation for this feature is nearly complete and is currently in review. Note that the address below is NOT permanent; the docs will be in the usual place at www.hdfgroup.org/HDF5/doc/ once HDF5 Release 1.8.9 is final.
In the meantime, the docs are TEMPORARILY available here:
Reference manual entries:
The relevant reference manual entries are all cross-referenced from "See Also" in the H5PFget_file_image entry:
http://www.hdfgroup.org/HDF5/doc_test/H51.8docs/RM/RM_H5F.html#File-GetFileImageNote that there is one high-level function that may suit your needs, H5LTopen_file_iamge. Several low-level H5F and H5P functions are provided for situations requiring more customization.
Please let me know if you experience any difficulty with these documents. Thanks!
-- Frank
HDF Documentation Team
fbaker@hdfgroup.orgOn 24 Apr 2012, at 15:38 PM, Adam Wierzchowski wrote:
Excellent. Where can I find more information on this. I don't see anything like that in hdf5-1.8.9-pre1-RELEASE.txt.
regards
AdamOn 2012-04-24 22:34, Quincey Koziol wrote:
Hi Mark/Adam,
The "file image" feature is available in the upcoming 1.8.9 release, which has a prerelease candidate available here:http://www.hdfgroup.uiuc.edu/ftp/pub/outgoing/hdf5/hdf5-1.8.9-pre1/
Quincey
On Apr 24, 2012, at 2:53 PM, Mark Miller wrote:
Ah, that capability also exists but not in a currently released version
of HDF5. I think it is available in HDF5-1.9 series. Maybe someone from
HDF Group could point you in the right direction. Also, have a looksee
here...https://visitbugs.ornl.gov/attachments/66/load_core_file_driver_from_image_RFC_v04.pdf
Hopefully, Quincy Koziol might be able point you to actual HDF5 version
tarball supporting this.Mark
On Tue, 2012-04-24 at 12:45 -0700, Adam Wierzchowski wrote:
Hi,
thank you. Exactly what I needed. But what about reading these data. I
need something like:
H5Fcreate...
H5InitializeWithRawBytesFromMemory(void* dataPointer, int
dataLengthInBytes)regards
Adam> If I am understanding you correctly, you can do that using the 'core'
> virtual file driver (VFD). Have a look at...> http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetFaplCore
> You need to create a file-access or file-creation properties list object
> with appropriate values set for the core VFD and it should behave as you
> describe.> Good luck.
> Mark
> On Tue, 2012-04-24 at 06:55 -0700, Adam Wierzchowski wrote:
Hi,
I have my custom format where I keep all kinds of data: strings, arrays
etc. I'd also like to keep chunk of data which is basically whole hdf
file. Is it possible?
Idea is to keep hdf5 file in memory and dump all that data to my file.
When reading just read all that data to memory and read from it with
hdf5 functions. Problem is that haven't found functions which will allow
me to do that in hdf5 docs.hope you can help me
regards
Adam_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
