I've begun writing a library in c++ with the purpose of easing use of hdf5
in c++ with the stl. The source is available at:
My reasons for doing this over the current hdf5 c++ bindings are thus:
- Lack of support for parallel MPI-IO (not currently implemented but will
be shortly!)
- Removal of necessity to construct manually dataspaces / datatypes when
the C++ type system can do the heavy lifting for you.
- Lack of support for hyperslabs
I'm interested in seeing what interest there is in helping out developing
this library as it is very much in its infancy and will largely be driven by
my own requirements but want to be able to make it available to others to
use. It is available under the Boost software license and I'd like to see it
in the future either be integrated into the hdfgroup's source tree or the
boost libraries.
I look forward to seeing the interest around this library.
Look for the H5Lite.h and H5Lite.cpp. I ended up doing what you did and used C++ type system to "pick the right type" using C++ templates. Pretty basic but seems to work.
···
___________________________________________________________
Mike Jackson www.bluequartz.net
Principal Software Engineer mike.jackson@bluequartz.net
BlueQuartz Software Dayton, Ohio
On Mar 29, 2010, at 3:08 PM, James Sharpe wrote:
I've begun writing a library in c++ with the purpose of easing use of hdf5 in c++ with the stl. The source is available at:
My reasons for doing this over the current hdf5 c++ bindings are thus:
• Lack of support for parallel MPI-IO (not currently implemented but will be shortly!)
• Removal of necessity to construct manually dataspaces / datatypes when the C++ type system can do the heavy lifting for you.
• Lack of support for hyperslabs
I'm interested in seeing what interest there is in helping out developing this library as it is very much in its infancy and will largely be driven by my own requirements but want to be able to make it available to others to use. It is available under the Boost software license and I'd like to see it in the future either be integrated into the hdfgroup's source tree or the boost libraries.
I look forward to seeing the interest around this library.
My reasons for doing this over the current hdf5 c++ bindings are thus:
Lack of support for parallel MPI-IO (not currently implemented but will be shortly!)
Removal of necessity to construct manually dataspaces / datatypes when the C++ type system can do the heavy lifting for you.
Lack of support for hyperslabs
Could you please send us an example of "lack of support for hyperslabs" in C++?
I'm interested in seeing what interest there is in helping out developing this library as it is very much in its infancy and will largely be driven by my own requirements but want to be able to make it available to others to use. It is available under the Boost software license and I'd like to see it in the future either be integrated into the hdfgroup's source tree or the boost libraries.
At this point we do not have funding to do any further development work on the HDF5 C++ library. Easing use of HDF5 C++ library with stl was one of the tasks we would like to do, but couldn't do due to the lack of resources. Our group will be glad to provide some support for this work, especially if it comes as an improvement/extension to the existing library since many of the HDF5 C++ users will benefit from this work.
Thank you!
Elena
···
On Mar 29, 2010, at 2:08 PM, James Sharpe wrote:
I look forward to seeing the interest around this library.
My reasons for doing this over the current hdf5 c++ bindings are thus:
- Lack of support for parallel MPI-IO (not currently implemented but
will be shortly!)
- Removal of necessity to construct manually dataspaces / datatypes
when the C++ type system can do the heavy lifting for you.
- Lack of support for hyperslabs
Could you please send us an example of "lack of support for hyperslabs" in
C++?
I guess when I mean a lack of support for them I mean lack of a proper
abstraction. When you have a container of structures in C++ and you want to
fill one of the fields of the structure from a dataset in the file I want to
be able to say fill field x of this structure and create the appropriate
dataspace for me. I admit though that I haven't yet come up with a nice way
of easily specifying this in C++.
At this point we do not have funding to do any further development work on
the HDF5 C++ library. Easing use of HDF5 C++ library with stl was one of the
tasks we would like to do, but couldn't do due to the lack of resources. Our
group will be glad to provide some support for this work, especially if it
comes as an improvement/extension to the existing library since many of the
HDF5 C++ users will benefit from this work.
My feelings are that the current level of abstraction provided by the
library is too low level. The MPI C++ bindings are kind of a good example of
this; the latest MPI standard has deprecated them because they are of little
use - they provided nothing above using the C api. The c++ bindings that
most people use is the boost library which provides a higher level
abstraction of the API that is more useful for interacting with C++ data
structures.
The kind of things that I'd like to see support for in HDF5 is at a layer
which interacts closely with the stl and associated algorithms. An
interesting application might be to wrap a HDF5 dataset with the stl
container concept to enable streaming of processing of data in a similar
vain to the stxxl library but obviously with the benefits of the on disk
format being hdf5. Being able to overlap I/O and compute transparently
without having to do the chunking myself is a very desirable feature, and of
course one of the compelling reasons to use c++ because it lends its self to
these kinds of abstractions very well.
I think the fact that people like Michael claim that people using c++ roll
their own functions around the hdf5 library shows that the c++ abstractions
are insufficient as they currently stand and could definitely be made more
useful.
My dynamic builds are all good (win/linux), but my static ones fail as some libs have the suffix -static on the libs themselves, but the link line doesn't. I'll be working on this tomorrow.
Question : Is this what we want (I'm trying to understand where the -static and lib and _D prefix/suffixes should go). If someone compiles static, then make install, then compiles dynamic and make install into the same location. We have a problem with the FindHDF5 stuff. I don't understand quite what naming convention you want. Do we want to allow the user to build static and dynamic together (Boost does this yes?) if so perhaps then we can manage it. Either way I would like to know what naming convention. Please correct below. thanks JB
Win32 :
Dynamic build
hdf5.dll (or do you want libhdf5.dll)
hdf5.lib (or do you want libhdf5.lib)
Dynamic Debug build
hdf5_D.dll (or libhdf5_D.dll)
hdf5_D.lib (or libhdf5_D.lib)
Static build
libhdf5.lib
Static Debug build
libhdf5_D.lib
I've begun writing a library in c++ with the purpose of easing use of hdf5 in c++ with the stl. The source is available at:
My reasons for doing this over the current hdf5 c++ bindings are thus:
* Lack of support for parallel MPI-IO (not currently implemented but will be shortly!)
* Removal of necessity to construct manually dataspaces / datatypes when the C++ type system can do the heavy lifting for you.
* Lack of support for hyperslabs
Could you please send us an example of "lack of support for hyperslabs" in C++?
I guess when I mean a lack of support for them I mean lack of a proper abstraction. When you have a container of structures in C++ and you want to fill one of the fields of the structure from a dataset in the file I want to be able to say fill field x of this structure and create the appropriate dataspace for me. I admit though that I haven't yet come up with a nice way of easily specifying this in C++.
At this point we do not have funding to do any further development work on the HDF5 C++ library. Easing use of HDF5 C++ library with stl was one of the tasks we would like to do, but couldn't do due to the lack of resources. Our group will be glad to provide some support for this work, especially if it comes as an improvement/extension to the existing library since many of the HDF5 C++ users will benefit from this work.
My feelings are that the current level of abstraction provided by the library is too low level. The MPI C++ bindings are kind of a good example of this; the latest MPI standard has deprecated them because they are of little use - they provided nothing above using the C api. The c++ bindings that most people use is the boost library which provides a higher level abstraction of the API that is more useful for interacting with C++ data structures.
The kind of things that I'd like to see support for in HDF5 is at a layer which interacts closely with the stl and associated algorithms. An interesting application might be to wrap a HDF5 dataset with the stl container concept to enable streaming of processing of data in a similar vain to the stxxl library but obviously with the benefits of the on disk format being hdf5. Being able to overlap I/O and compute transparently without having to do the chunking myself is a very desirable feature, and of course one of the compelling reasons to use c++ because it lends its self to these kinds of abstractions very well.
I think the fact that people like Michael claim that people using c++ roll their own functions around the hdf5 library shows that the c++ abstractions are insufficient as they currently stand and could definitely be made more useful.
Sorry. I hit reply to the wrong message and typed without looking, this email was intended for the Cmake + HDF5 people and not the HDF5 users list. Apologies for the noise.
JB
···
From: Biddiscombe, John A.
Sent: 31 March 2010 22:56
To: HDF Users Discussion List
Subject: RE: [Hdf-forum] HDF5 C++ library
My dynamic builds are all good (win/linux), but my static ones fail as some libs have the suffix -static on the libs themselves, but the link line doesn't. I'll be working on this tomorrow.
Question : Is this what we want (I'm trying to understand where the -static and lib and _D prefix/suffixes should go). If someone compiles static, then make install, then compiles dynamic and make install into the same location. We have a problem with the FindHDF5 stuff. I don't understand quite what naming convention you want. Do we want to allow the user to build static and dynamic together (Boost does this yes?) if so perhaps then we can manage it. Either way I would like to know what naming convention. Please correct below. thanks JB
Win32 :
Dynamic build
hdf5.dll (or do you want libhdf5.dll)
hdf5.lib (or do you want libhdf5.lib)
Dynamic Debug build
hdf5_D.dll (or libhdf5_D.dll)
hdf5_D.lib (or libhdf5_D.lib)
Static build
libhdf5.lib
Static Debug build
libhdf5_D.lib
I've begun writing a library in c++ with the purpose of easing use of hdf5 in c++ with the stl. The source is available at:
My reasons for doing this over the current hdf5 c++ bindings are thus:
* Lack of support for parallel MPI-IO (not currently implemented but will be shortly!)
* Removal of necessity to construct manually dataspaces / datatypes when the C++ type system can do the heavy lifting for you.
* Lack of support for hyperslabs
Could you please send us an example of "lack of support for hyperslabs" in C++?
I guess when I mean a lack of support for them I mean lack of a proper abstraction. When you have a container of structures in C++ and you want to fill one of the fields of the structure from a dataset in the file I want to be able to say fill field x of this structure and create the appropriate dataspace for me. I admit though that I haven't yet come up with a nice way of easily specifying this in C++.
At this point we do not have funding to do any further development work on the HDF5 C++ library. Easing use of HDF5 C++ library with stl was one of the tasks we would like to do, but couldn't do due to the lack of resources. Our group will be glad to provide some support for this work, especially if it comes as an improvement/extension to the existing library since many of the HDF5 C++ users will benefit from this work.
My feelings are that the current level of abstraction provided by the library is too low level. The MPI C++ bindings are kind of a good example of this; the latest MPI standard has deprecated them because they are of little use - they provided nothing above using the C api. The c++ bindings that most people use is the boost library which provides a higher level abstraction of the API that is more useful for interacting with C++ data structures.
The kind of things that I'd like to see support for in HDF5 is at a layer which interacts closely with the stl and associated algorithms. An interesting application might be to wrap a HDF5 dataset with the stl container concept to enable streaming of processing of data in a similar vain to the stxxl library but obviously with the benefits of the on disk format being hdf5. Being able to overlap I/O and compute transparently without having to do the chunking myself is a very desirable feature, and of course one of the compelling reasons to use c++ because it lends its self to these kinds of abstractions very well.
I think the fact that people like Michael claim that people using c++ roll their own functions around the hdf5 library shows that the c++ abstractions are insufficient as they currently stand and could definitely be made more useful.
Hi All,
I am trying
hid_t dataset_id = H5Dcreate(group_frame_id, "mass (GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
And the returned dataset_id = -1. If I changed the string to "mass (GeV_(c^2))", I got a valid dataset_id.
Is "/" a special/reserved character in HDF5? Is there such a reserved character list for HDF5?
Thanks so much.
Yes, I am pretty sure the slash character is used by HDF5 library to
break up dataset names into group paths much like a unix filesystem
does.
I don't know if there is a way to tell HDF5 to use a different separator
character for this purpose for a given file.
Mark
···
On Thu, 2010-04-01 at 14:42, Dr. X wrote:
Hi All,
I am trying
hid_t dataset_id = H5Dcreate(group_frame_id, "mass
(GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
And the returned dataset_id = -1. If I changed the string to "mass
(GeV_(c^2))", I got a valid dataset_id.
Is "/" a special/reserved character in HDF5? Is there such a reserved
character list for HDF5?
Thanks so much.
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================ miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
Thanks, Mark. I thought the field name (const char *) is packaged differently than the data tree. I am bumping my head to find an alternative character to represent division intuitively. Any suggestions?
Best,
xunlei
···
On 4/1/2010 5:48 PM, Mark Miller wrote:
Yes, I am pretty sure the slash character is used by HDF5 library to
break up dataset names into group paths much like a unix filesystem
does.
I don't know if there is a way to tell HDF5 to use a different separator
character for this purpose for a given file.
Mark
On Thu, 2010-04-01 at 14:42, Dr. X wrote:
Hi All,
I am trying
hid_t dataset_id = H5Dcreate(group_frame_id, "mass
(GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
And the returned dataset_id = -1. If I changed the string to "mass
(GeV_(c^2))", I got a valid dataset_id.
Is "/" a special/reserved character in HDF5? Is there such a reserved
character list for HDF5?
Thanks so much.
Hi All,
I am trying
hid_t dataset_id = H5Dcreate(group_frame_id, "mass (GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
And the returned dataset_id = -1. If I changed the string to "mass (GeV_(c^2))", I got a valid dataset_id.
Is "/" a special/reserved character in HDF5? Is there such a reserved character list for HDF5?
Hi Xunlei,
If your goal is to indicate the units of mass used then a more common approach is to specify those as a bit of metadata, for example as an HDF5 attribute called "Units" which a value of "GeV/(c^2)".
Cheers,
--dan
···
--
Daniel Kahn
Science Systems and Applications Inc.
301-867-2162
Hmm. '|' (better), '\' (backwards) '%' (integral mod) or ':' (ratio)
or if you want to get verbose '.|.' which kinda looks like the ole'
character you find on most calculator keyboards but turned 90 degress.
Best I can think of.
Beware another developer's suggestion; you can store strings containing
slashes as the 'values' (and maybe even names) of attributes on your
dataset(s).
Mark
···
On Thu, 2010-04-01 at 14:56, Dr. X wrote:
Thanks, Mark. I thought the field name (const char *) is packaged
differently than the data tree. I am bumping my head to find an
alternative character to represent division intuitively. Any suggestions?
Best,
xunlei
On 4/1/2010 5:48 PM, Mark Miller wrote:
> Yes, I am pretty sure the slash character is used by HDF5 library to
> break up dataset names into group paths much like a unix filesystem
> does.
>
> I don't know if there is a way to tell HDF5 to use a different separator
> character for this purpose for a given file.
>
> Mark
>
> On Thu, 2010-04-01 at 14:42, Dr. X wrote:
>
>> Hi All,
>> I am trying
>> hid_t dataset_id = H5Dcreate(group_frame_id, "mass
>> (GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
>> And the returned dataset_id = -1. If I changed the string to "mass
>> (GeV_(c^2))", I got a valid dataset_id.
>> Is "/" a special/reserved character in HDF5? Is there such a reserved
>> character list for HDF5?
>> Thanks so much.
>>
>> Best,
>> xunlei
>>
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://**mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================ miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-851
An alternative is going to unicode and use the appropriate math character:
Though actually I'd prefer to have HDF5 names of groups and datasets
to be free of reserver characters, though changing this established
behavior in the library now would probably break too many existing codes.
Werner
···
On Thu, 01 Apr 2010 18:24:56 -0400, Mark Miller <miller86@llnl.gov> wrote:
Hmm. '|' (better), '\' (backwards) '%' (integral mod) or ':' (ratio)
or if you want to get verbose '.|.' which kinda looks like the ole'
character you find on most calculator keyboards but turned 90 degress.
Best I can think of.
Beware another developer's suggestion; you can store strings containing
slashes as the 'values' (and maybe even names) of attributes on your
dataset(s).
Mark
On Thu, 2010-04-01 at 14:56, Dr. X wrote:
Thanks, Mark. I thought the field name (const char *) is packaged
differently than the data tree. I am bumping my head to find an
alternative character to represent division intuitively. Any suggestions?
Best,
xunlei
On 4/1/2010 5:48 PM, Mark Miller wrote:
> Yes, I am pretty sure the slash character is used by HDF5 library to
> break up dataset names into group paths much like a unix filesystem
> does.
>
> I don't know if there is a way to tell HDF5 to use a different separator
> character for this purpose for a given file.
>
> Mark
>
> On Thu, 2010-04-01 at 14:42, Dr. X wrote:
>
>> Hi All,
>> I am trying
>> hid_t dataset_id = H5Dcreate(group_frame_id, "mass
>> (GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
>> And the returned dataset_id = -1. If I changed the string to "mass
>> (GeV_(c^2))", I got a valid dataset_id.
>> Is "/" a special/reserved character in HDF5? Is there such a reserved
>> character list for HDF5?
>> Thanks so much.
>>
>> Best,
>> xunlei
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://**mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
Hi Dan, Mark, Werner,
Thank you all for your help. I in the end implemented what Dan suggested by using Attribute to store units. It is the cleanest way, I think.
In a long run, Unicode integration would be great though it is a minor issue.
Best,
x
···
On 4/1/2010 6:00 PM, Daniel Kahn wrote:
Dr. X wrote:
Hi All,
I am trying
hid_t dataset_id = H5Dcreate(group_frame_id, "mass
(GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
And the returned dataset_id = -1. If I changed the string to "mass
(GeV_(c^2))", I got a valid dataset_id.
Is "/" a special/reserved character in HDF5? Is there such a reserved
character list for HDF5?
Hi Xunlei,
If your goal is to indicate the units of mass used then a more common
approach is to specify those as a bit of metadata, for example as an
HDF5 attribute called "Units" which a value of "GeV/(c^2)".
Oh btw., you could also add the units attribute to the data *type* instead
of the data *set*.
That has the advantage that automatically all datasets of the same data type
share the same units and you don't need to add the same attribute for each
data set. For doing so, you need to create a named data type that is stored
in the file.
cheers,
Werner
···
On Fri, 02 Apr 2010 09:20:22 -0400, Dr. X <xunlei@renci.org> wrote:
Hi Dan, Mark, Werner,
Thank you all for your help. I in the end implemented what Dan suggested
by using Attribute to store units. It is the cleanest way, I think.
In a long run, Unicode integration would be great though it is a minor
issue.
Best,
x
On 4/1/2010 6:00 PM, Daniel Kahn wrote:
Dr. X wrote:
Hi All,
I am trying
hid_t dataset_id = H5Dcreate(group_frame_id, "mass
(GeV/(c^2))", std_type_id, space_id, H5P_DEFAULT, H5P_DEFAULT,
H5P_DEFAULT);
And the returned dataset_id = -1. If I changed the string to "mass
(GeV_(c^2))", I got a valid dataset_id.
Is "/" a special/reserved character in HDF5? Is there such a reserved
character list for HDF5?
Hi Xunlei,
If your goal is to indicate the units of mass used then a more common
approach is to specify those as a bit of metadata, for example as an
HDF5 attribute called "Units" which a value of "GeV/(c^2)".
--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
