Different data types

Hdf-forum archives
1

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

Thanks for the help!

2

Hello, Cecilia

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot >pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library >of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

It's not clear to me, from your description, the nature of the problem you're trying to solve.
If you send further details I'll be glad to help.

Regarding the use of H5NX, this page

http://www.space-research.org/nexus/h5nx.html

has both links for

      H5NX User's Guide
      Detailed guide covering critical H5NX features
      H5NX Reference Manual
      Detailed C++ interface descriptions

but I deal with 2D matrices and different data types.

The H5NX function H5NXmake_dataset_vector deals with any rank, that is, you can use it to define a 1D array, a 2D array, etc.

Just put your data values into a std::vector of any type and define the dimensions in a std::vector<hsize_t>

If the size of this dimension vector is, say 2, then you have a matrix, or a 2D array.
Since this is C++, there is no need for an extra parameter to the function with the size of the dimension vector (the rank)

int H5NXmake_dataset_vector(const std::string &group_path,

const std::string &dataset_name,

const std::vector<NumT> &vec,

const std::vector<hsize_t> &dim_vec);

The only function that currently deals exclusively with a 1D array is

H5NXappend_slab

Regarding the types, I am currently defining it for other types, I'll upload a new version when done.

H5NX uses the HDF5 native types corresponding to the C99 fixed width integer types plus the HDF5 string type, used in the "string" functions

int8_t
uint8_t
int16_t
uint16_t
int32_t
uint32_t
int64_t
uint64_t
float
double

but not all API functions have templates for all types.

What are your needs regarding these types and functions ?

Pedro

···

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

  ----- Original Message -----
  From: Cecilia Herrera
  To: 'hdf-forum@hdfgroup.org'
  Sent: Monday, March 11, 2013 2:26 AM
  Subject: [Hdf-forum] Different data types

  I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

  Thanks for the help!

------------------------------------------------------------------------------

  _______________________________________________
  Hdf-forum is for HDF software users discussion.
  Hdf-forum@hdfgroup.org
  http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

3

Hello Pedro,

Thanks for your response.

I have a program where the user can choose one or more variables with different data types to export. For example, name and age, which have string and integer datatypes can be chosen. I do not know beforehand which variables, what data types, and how many variables will be chosen. I have looked at possibilities in HDF5 to do this, and think that the use of compound data type is appropriate. The examples I found defined the structure of the data (e.g. whether to have int or string in the first column, etc.) using "struct". For example, in http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/cmpclass.c, the "struct" is predefined at the start of the program (from line 15...). However, for my application, I cannot predefine the structure, because it depends on which variable the user will choose (can be a combination of different types). Another option is to use strings for all variables (eg in my example transforming the integer data type for age in a string), but the data will take longer to export, and I would like to make the export speed as fast as possible.

Would you know how I can attain what I want using compound data type or any other way?

Thanks,
Cecilia

···

From: Pedro Vicente [mailto:pvicente@uci.edu]
Sent: Wednesday, April 17, 2013 6:43 AM
To: HDF Users Discussion List
Cc: Cecilia Herrera
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello, Cecilia

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot >pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library >of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

It's not clear to me, from your description, the nature of the problem you're trying to solve.
If you send further details I'll be glad to help.

Regarding the use of H5NX, this page

http://www.space-research.org/nexus/h5nx.html

has both links for

H5NX User's Guide<http://www.space-research.org/nexus/h5nx_user.html>
Detailed guide covering critical H5NX features

H5NX Reference Manual<http://www.space-research.org/nexus/h5nx_ref.html>
Detailed C++ interface descriptions

but I deal with 2D matrices and different data types.

The H5NX function H5NXmake_dataset_vector deals with any rank, that is, you can use it to define a 1D array, a 2D array, etc.

Just put your data values into a std::vector of any type and define the dimensions in a std::vector<hsize_t>

If the size of this dimension vector is, say 2, then you have a matrix, or a 2D array.
Since this is C++, there is no need for an extra parameter to the function with the size of the dimension vector (the rank)

int H5NXmake_dataset_vector(const std::string &group_path,

const std::string &dataset_name,

const std::vector<NumT> &vec,

const std::vector<hsize_t> &dim_vec);

The only function that currently deals exclusively with a 1D array is

H5NXappend_slab

Regarding the types, I am currently defining it for other types, I'll upload a new version when done.

H5NX uses the HDF5 native types corresponding to the C99 fixed width integer types plus the HDF5 string type, used in the "string" functions

int8_t
uint8_t
int16_t
uint16_t
int32_t
uint32_t
int64_t
uint64_t
float
double

but not all API functions have templates for all types.

What are your needs regarding these types and functions ?

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: Cecilia Herrera<mailto:C.Herrera@Noldus.NL>
To: 'hdf-forum@hdfgroup.org'<mailto:'hdf-forum@hdfgroup.org'>
Sent: Monday, March 11, 2013 2:26 AM
Subject: [Hdf-forum] Different data types

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

Thanks for the help!

________________________________
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org<mailto:Hdf-forum@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

You can dynamically build a compound HDF type. I'm playing in the C#/.NET world and through reflection, I can dynamically build the struct too. It's a bit ugly, but is certainly do-able. I'm guessing it can be done on the C side using untyped buffers and a lot of work figuring out appropriate indexes.

S

···

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Cecilia Herrera
Sent: Wednesday, April 17, 2013 4:12 AM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello Pedro,

Thanks for your response.

I have a program where the user can choose one or more variables with different data types to export. For example, name and age, which have string and integer datatypes can be chosen. I do not know beforehand which variables, what data types, and how many variables will be chosen. I have looked at possibilities in HDF5 to do this, and think that the use of compound data type is appropriate. The examples I found defined the structure of the data (e.g. whether to have int or string in the first column, etc.) using "struct". For example, in http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/cmpclass.c, the "struct" is predefined at the start of the program (from line 15...). However, for my application, I cannot predefine the structure, because it depends on which variable the user will choose (can be a combination of different types). Another option is to use strings for all variables (eg in my example transforming the integer data type for age in a string), but the data will take longer to export, and I would like to make the export speed as fast as possible.

Would you know how I can attain what I want using compound data type or any other way?

Thanks,
Cecilia

From: Pedro Vicente [mailto:pvicente@uci.edu]
Sent: Wednesday, April 17, 2013 6:43 AM
To: HDF Users Discussion List
Cc: Cecilia Herrera
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello, Cecilia

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot >pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library >of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

It's not clear to me, from your description, the nature of the problem you're trying to solve.
If you send further details I'll be glad to help.

Regarding the use of H5NX, this page

http://www.space-research.org/nexus/h5nx.html

has both links for

H5NX User's Guide<http://www.space-research.org/nexus/h5nx_user.html>
Detailed guide covering critical H5NX features

H5NX Reference Manual<http://www.space-research.org/nexus/h5nx_ref.html>
Detailed C++ interface descriptions

but I deal with 2D matrices and different data types.

The H5NX function H5NXmake_dataset_vector deals with any rank, that is, you can use it to define a 1D array, a 2D array, etc.

Just put your data values into a std::vector of any type and define the dimensions in a std::vector<hsize_t>

If the size of this dimension vector is, say 2, then you have a matrix, or a 2D array.
Since this is C++, there is no need for an extra parameter to the function with the size of the dimension vector (the rank)

int H5NXmake_dataset_vector(const std::string &group_path,

const std::string &dataset_name,

const std::vector<NumT> &vec,

const std::vector<hsize_t> &dim_vec);

The only function that currently deals exclusively with a 1D array is

H5NXappend_slab

Regarding the types, I am currently defining it for other types, I'll upload a new version when done.

H5NX uses the HDF5 native types corresponding to the C99 fixed width integer types plus the HDF5 string type, used in the "string" functions

int8_t
uint8_t
int16_t
uint16_t
int32_t
uint32_t
int64_t
uint64_t
float
double

but not all API functions have templates for all types.

What are your needs regarding these types and functions ?

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: Cecilia Herrera<mailto:C.Herrera@Noldus.NL>
To: 'hdf-forum@hdfgroup.org'<mailto:'hdf-forum@hdfgroup.org'>
Sent: Monday, March 11, 2013 2:26 AM
Subject: [Hdf-forum] Different data types

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

Thanks for the help!

________________________________
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org<mailto:Hdf-forum@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

________________________________

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

5

Hi Cecilia

I am not sure about dynamically building a data type, but often the best solution for a problem is the simplest.

I seems the user will choose an option for a data type to have his data , in some input,

for example

in standard input by pressing a key 1 for integer or key 2 for float

in pseudo code

if key == 1 then
export_mydata_as_integer()
else if key == 2 then
export_mydata_as_float()

In H5NX, to define these 2 functions , you would use the following 2 vector declarations

for integers

std::vector<int32_t> vec_data;

for floats
std::vector<float> vec_data;
then in each function call to save your vectorif (h5nx.H5NXmake_dataset_vector("/", "dset", vec_data, vec_dim) < 0) { };
A complete example is here

http://www.space-research.org/nexus/h5nx_user.html

You can also ask in this forum , since this is not really a HDF5 issue, here's a search for this issue

···

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

  ----- Original Message -----
  From: Cecilia Herrera
  To: hdf-forum@hdfgroup.org
  Cc: 'Pedro Vicente'
  Sent: Wednesday, April 17, 2013 1:11 AM
  Subject: RE: [Hdf-forum] Different data types --- H5NX

  Hello Pedro,

  Thanks for your response.

  I have a program where the user can choose one or more variables with different data types to export. For example, name and age, which have string and integer datatypes can be chosen. I do not know beforehand which variables, what data types, and how many variables will be chosen. I have looked at possibilities in HDF5 to do this, and think that the use of compound data type is appropriate. The examples I found defined the structure of the data (e.g. whether to have int or string in the first column, etc.) using "struct". For example, in http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/cmpclass.c, the "struct" is predefined at the start of the program (from line 15.). However, for my application, I cannot predefine the structure, because it depends on which variable the user will choose (can be a combination of different types). Another option is to use strings for all variables (eg in my example transforming the integer data type for age in a string), but the data will take longer to export, and I would like to make the export speed as fast as possible.

  Would you know how I can attain what I want using compound data type or any other way?

  Thanks,

  Cecilia

  From: Pedro Vicente [mailto:pvicente@uci.edu]
  Sent: Wednesday, April 17, 2013 6:43 AM
  To: HDF Users Discussion List
  Cc: Cecilia Herrera
  Subject: Re: [Hdf-forum] Different data types --- H5NX

  Hello, Cecilia

  >I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot >pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library >of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

  It's not clear to me, from your description, the nature of the problem you're trying to solve.

  If you send further details I'll be glad to help.

  Regarding the use of H5NX, this page

  http://www.space-research.org/nexus/h5nx.html

  has both links for

        H5NX User's Guide
        Detailed guide covering critical H5NX features
       
        H5NX Reference Manual
        Detailed C++ interface descriptions
       
  >> but I deal with 2D matrices and different data types.

  The H5NX function H5NXmake_dataset_vector deals with any rank, that is, you can use it to define a 1D array, a 2D array, etc.

  Just put your data values into a std::vector of any type and define the dimensions in a std::vector<hsize_t>

  If the size of this dimension vector is, say 2, then you have a matrix, or a 2D array.

  Since this is C++, there is no need for an extra parameter to the function with the size of the dimension vector (the rank)

  int H5NXmake_dataset_vector(const std::string &group_path,

  const std::string &dataset_name,

  const std::vector<NumT> &vec,

  const std::vector<hsize_t> &dim_vec);

  The only function that currently deals exclusively with a 1D array is

  H5NXappend_slab

  Regarding the types, I am currently defining it for other types, I'll upload a new version when done.

  H5NX uses the HDF5 native types corresponding to the C99 fixed width integer types plus the HDF5 string type, used in the "string" functions

  int8_t
  uint8_t
  int16_t
  uint16_t
  int32_t
  uint32_t
  int64_t
  uint64_t
  float
  double

  but not all API functions have templates for all types.

  What are your needs regarding these types and functions ?

  Pedro

  ------
  Pedro Vicente, Earth System Science
  University of California, Irvine
  http://www.ess.uci.edu/

    ----- Original Message -----

    From: Cecilia Herrera

    To: 'hdf-forum@hdfgroup.org'

    Sent: Monday, March 11, 2013 2:26 AM

    Subject: [Hdf-forum] Different data types

    I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

    Thanks for the help!

----------------------------------------------------------------------------

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@hdfgroup.org
    http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

6

I am also trying to use HDF5 compound datatypes in the C# world, with great
difficulty. All the sample programs are in C and trying to translate into
C# is nearly impossible. Even using the VS object browser still leaves a
lot to be desired. How far have you gotten. Have you been able to create
the datatype and have been able to write a file containing that data.

···

On Wed, Apr 17, 2013 at 8:50 AM, Mitchell, Scott - IS < Scott.Mitchell@exelisinc.com> wrote:

You can dynamically build a compound HDF type. I’m playing in the
C#/.NET world and through reflection, I can dynamically build the struct
too. It’s a bit ugly, but is certainly do-able. I’m guessing it can be done
on the C side using untyped buffers and a lot of work figuring out
appropriate indexes.

S

*From:* Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Cecilia
Herrera
*Sent:* Wednesday, April 17, 2013 4:12 AM
*To:* hdf-forum@hdfgroup.org

*Subject:* Re: [Hdf-forum] Different data types --- H5NX

Hello Pedro,

Thanks for your response.

I have a program where the user can choose one or more variables with
different data types to export. For example, name and age, which have
string and integer datatypes can be chosen. I do not know beforehand which
variables, what data types, and how many variables will be chosen. I have
looked at possibilities in HDF5 to do this, and think that the use of
compound data type is appropriate. The examples I found defined the
structure of the data (e.g. whether to have int or string in the first
column, etc.) using “struct”. For example, in
http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/cmpclass.c, the
“struct” is predefined at the start of the program (from line 15…).
However, for my application, I cannot predefine the structure, because it
depends on which variable the user will choose (can be a combination of
different types). Another option is to use strings for all variables (eg
in my example transforming the integer data type for age in a string), but
the data will take longer to export, and I would like to make the export
speed as fast as possible.

Would you know how I can attain what I want using compound data type or
any other way?

Thanks,

Cecilia

*From:* Pedro Vicente [mailto:pvicente@uci.edu <pvicente@uci.edu>]
*Sent:* Wednesday, April 17, 2013 6:43 AM
*To:* HDF Users Discussion List
*Cc:* Cecilia Herrera
*Subject:* Re: [Hdf-forum] Different data types --- H5NX

Hello, Cecilia

>I would like to export large amounts of data in one file. The data
variables can be of different types; but it is not known beforehand what
they are. I saw examples where the Compound data type was used with a
pre-defined “struct”. In this case, I cannot >pre-define my struct. I can
ofcourse make everything string, but I want to take advantage of the faster
speed of writing other types. How should I approach the problem? Do you
have available examples? It would also be nice to make use of the H5NX
library >of Pedro Vicente for faster speed, but I deal with 2D matrices and
different data types.

It's not clear to me, from your description, the nature of the problem
you're trying to solve.

If you send further details I'll be glad to help.

Regarding the use of H5NX, this page

http://www.space-research.org/nexus/h5nx.html

has both links for

   H5NX User’s Guide <http://www.space-research.org/nexus/h5nx_user.html>

Detailed guide covering critical H5NX features
  H5NX Reference Manual<http://www.space-research.org/nexus/h5nx_ref.html>

Detailed C++ interface descriptions

>> but I deal with 2D matrices and different data types.

The H5NX function H5NXmake_dataset_vector deals with any rank, that is,
you can use it to define a 1D array, a 2D array, etc.

Just put your data values into a std::vector of any type and define the
dimensions in a std::vector<hsize_t>

If the size of this dimension vector is, say 2, then you have a matrix, or
a 2D array.

Since this is C++, there is no need for an extra parameter to the function
with the size of the dimension vector (the rank)

int H5NXmake_dataset_vector(const std::string &group_path,

const std::string &dataset_name,

const std::vector<NumT> &vec,

const std::vector<hsize_t> &dim_vec);

The only function that currently deals exclusively with a 1D array is

H5NXappend_slab

Regarding the types, I am currently defining it for other types, I'll
upload a new version when done.

H5NX uses the HDF5 native types corresponding to the C99 fixed width
integer types plus the HDF5 string type, used in the "string" functions

int8_t
uint8_t
int16_t
uint16_t
int32_t
uint32_t
int64_t
uint64_t
float
double

but not all API functions have templates for all types.

What are your needs regarding these types and functions ?

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----

*From:* Cecilia Herrera <C.Herrera@Noldus.NL>

*To:* 'hdf-forum@hdfgroup.org'

*Sent:* Monday, March 11, 2013 2:26 AM

*Subject:* [Hdf-forum] Different data types

I would like to export large amounts of data in one file. The data
variables can be of different types; but it is not known beforehand what
they are. I saw examples where the Compound data type was used with a
pre-defined “struct”. In this case, I cannot pre-define my struct. I can
ofcourse make everything string, but I want to take advantage of the faster
speed of writing other types. How should I approach the problem? Do you
have available examples? It would also be nice to make use of the H5NX
library of Pedro Vicente for faster speed, but I deal with 2D matrices and
different data types.

Thanks for the help!

------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

------------------------------

This e-mail and any files transmitted with it may be proprietary and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the
sender. Please note that any views or opinions presented in this e-mail are
solely those of the author and do not necessarily represent those of Exelis
Inc. The recipient should check this e-mail and any attachments for the
presence of viruses. Exelis Inc. accepts no liability for any damage caused
by any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

7

Hi Mitchell,

I program in C++. I am interested in the C# program you have that dynamically builds the struct. Can I have a copy of it?
I will look into untyped buffers and Boost::Variant.

Thanks,
Cecilia

···

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Mitchell, Scott - IS
Sent: Wednesday, April 17, 2013 3:50 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Different data types --- H5NX

You can dynamically build a compound HDF type. I'm playing in the C#/.NET world and through reflection, I can dynamically build the struct too. It's a bit ugly, but is certainly do-able. I'm guessing it can be done on the C side using untyped buffers and a lot of work figuring out appropriate indexes.

S

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Cecilia Herrera
Sent: Wednesday, April 17, 2013 4:12 AM
To: hdf-forum@hdfgroup.org<mailto:hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello Pedro,

Thanks for your response.

I have a program where the user can choose one or more variables with different data types to export. For example, name and age, which have string and integer datatypes can be chosen. I do not know beforehand which variables, what data types, and how many variables will be chosen. I have looked at possibilities in HDF5 to do this, and think that the use of compound data type is appropriate. The examples I found defined the structure of the data (e.g. whether to have int or string in the first column, etc.) using "struct". For example, in http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/cmpclass.c, the "struct" is predefined at the start of the program (from line 15...). However, for my application, I cannot predefine the structure, because it depends on which variable the user will choose (can be a combination of different types). Another option is to use strings for all variables (eg in my example transforming the integer data type for age in a string), but the data will take longer to export, and I would like to make the export speed as fast as possible.

Would you know how I can attain what I want using compound data type or any other way?

Thanks,
Cecilia

From: Pedro Vicente [mailto:pvicente@uci.edu]
Sent: Wednesday, April 17, 2013 6:43 AM
To: HDF Users Discussion List
Cc: Cecilia Herrera
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello, Cecilia

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot >pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library >of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

It's not clear to me, from your description, the nature of the problem you're trying to solve.
If you send further details I'll be glad to help.

Regarding the use of H5NX, this page

http://www.space-research.org/nexus/h5nx.html

has both links for

H5NX User's Guide<http://www.space-research.org/nexus/h5nx_user.html>
Detailed guide covering critical H5NX features

H5NX Reference Manual<http://www.space-research.org/nexus/h5nx_ref.html>
Detailed C++ interface descriptions

but I deal with 2D matrices and different data types.

The H5NX function H5NXmake_dataset_vector deals with any rank, that is, you can use it to define a 1D array, a 2D array, etc.

Just put your data values into a std::vector of any type and define the dimensions in a std::vector<hsize_t>

If the size of this dimension vector is, say 2, then you have a matrix, or a 2D array.
Since this is C++, there is no need for an extra parameter to the function with the size of the dimension vector (the rank)

int H5NXmake_dataset_vector(const std::string &group_path,

const std::string &dataset_name,

const std::vector<NumT> &vec,

const std::vector<hsize_t> &dim_vec);

The only function that currently deals exclusively with a 1D array is

H5NXappend_slab

Regarding the types, I am currently defining it for other types, I'll upload a new version when done.

H5NX uses the HDF5 native types corresponding to the C99 fixed width integer types plus the HDF5 string type, used in the "string" functions

int8_t
uint8_t
int16_t
uint16_t
int32_t
uint32_t
int64_t
uint64_t
float
double

but not all API functions have templates for all types.

What are your needs regarding these types and functions ?

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: Cecilia Herrera<mailto:C.Herrera@Noldus.NL>
To: 'hdf-forum@hdfgroup.org'<mailto:'hdf-forum@hdfgroup.org'>
Sent: Monday, March 11, 2013 2:26 AM
Subject: [Hdf-forum] Different data types

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

Thanks for the help!

________________________________
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org<mailto:Hdf-forum@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

________________________________

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.

8

Unfortunately I cannot pass along the code: company policy. They're paranoid about the whole "open source" thing (and yes I've asked). I have C# code that can change .NET structs to & from HDF types. There are definitely limitations. There is not a complete 1:1 mapping, but I have found it useful. Especially as I have a over dozen compound types (some with many versions) that I need to handle and I don't have to rewrite some code each time something is added or changed.

If someone wants to take it on. I can provide help.

In C# the important classes are in System.Type for type queries and System.Reflection for construction: AssemblyBuilder, ModuleBuilder, etc.. TypeBuilder builds structs as H5T.create() builds compound types. Enums translate cleanly. Most primitive types are easy. The exception is Boolean, which exists in HDF as a 4byte int and is indistinguishable from other 4byte ints through H5T queries. Strings work fine with the Chararray class mentioned before. I also have some special cases for handling .NET DateTimes as specially named ulongs. I haven't finished the handling of arrays within the struct yet, other issues have taken precedence.

Scott

···

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Cecilia Herrera
Sent: Thursday, April 18, 2013 3:45 AM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] FW: Different data types --- H5NX

Hi Mitchell,

I program in C++. I am interested in the C# program you have that dynamically builds the struct. Can I have a copy of it?
I will look into untyped buffers and Boost::Variant.

Thanks,
Cecilia

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Mitchell, Scott - IS
Sent: Wednesday, April 17, 2013 3:50 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Different data types --- H5NX

You can dynamically build a compound HDF type. I'm playing in the C#/.NET world and through reflection, I can dynamically build the struct too. It's a bit ugly, but is certainly do-able. I'm guessing it can be done on the C side using untyped buffers and a lot of work figuring out appropriate indexes.

S

From: Hdf-forum [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Cecilia Herrera
Sent: Wednesday, April 17, 2013 4:12 AM
To: hdf-forum@hdfgroup.org<mailto:hdf-forum@hdfgroup.org>
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello Pedro,

Thanks for your response.

I have a program where the user can choose one or more variables with different data types to export. For example, name and age, which have string and integer datatypes can be chosen. I do not know beforehand which variables, what data types, and how many variables will be chosen. I have looked at possibilities in HDF5 to do this, and think that the use of compound data type is appropriate. The examples I found defined the structure of the data (e.g. whether to have int or string in the first column, etc.) using "struct". For example, in http://www.hdfgroup.org/ftp/HDF5/examples/misc-examples/cmpclass.c, the "struct" is predefined at the start of the program (from line 15...). However, for my application, I cannot predefine the structure, because it depends on which variable the user will choose (can be a combination of different types). Another option is to use strings for all variables (eg in my example transforming the integer data type for age in a string), but the data will take longer to export, and I would like to make the export speed as fast as possible.

Would you know how I can attain what I want using compound data type or any other way?

Thanks,
Cecilia

From: Pedro Vicente [mailto:pvicente@uci.edu]
Sent: Wednesday, April 17, 2013 6:43 AM
To: HDF Users Discussion List
Cc: Cecilia Herrera
Subject: Re: [Hdf-forum] Different data types --- H5NX

Hello, Cecilia

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot >pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library >of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

It's not clear to me, from your description, the nature of the problem you're trying to solve.
If you send further details I'll be glad to help.

Regarding the use of H5NX, this page

http://www.space-research.org/nexus/h5nx.html

has both links for

H5NX User's Guide<http://www.space-research.org/nexus/h5nx_user.html>
Detailed guide covering critical H5NX features

H5NX Reference Manual<http://www.space-research.org/nexus/h5nx_ref.html>
Detailed C++ interface descriptions

but I deal with 2D matrices and different data types.

The H5NX function H5NXmake_dataset_vector deals with any rank, that is, you can use it to define a 1D array, a 2D array, etc.

Just put your data values into a std::vector of any type and define the dimensions in a std::vector<hsize_t>

If the size of this dimension vector is, say 2, then you have a matrix, or a 2D array.
Since this is C++, there is no need for an extra parameter to the function with the size of the dimension vector (the rank)

int H5NXmake_dataset_vector(const std::string &group_path,

const std::string &dataset_name,

const std::vector<NumT> &vec,

const std::vector<hsize_t> &dim_vec);

The only function that currently deals exclusively with a 1D array is

H5NXappend_slab

Regarding the types, I am currently defining it for other types, I'll upload a new version when done.

H5NX uses the HDF5 native types corresponding to the C99 fixed width integer types plus the HDF5 string type, used in the "string" functions

int8_t
uint8_t
int16_t
uint16_t
int32_t
uint32_t
int64_t
uint64_t
float
double

but not all API functions have templates for all types.

What are your needs regarding these types and functions ?

Pedro

------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/

----- Original Message -----
From: Cecilia Herrera<mailto:C.Herrera@Noldus.NL>
To: 'hdf-forum@hdfgroup.org'<mailto:'hdf-forum@hdfgroup.org'>
Sent: Monday, March 11, 2013 2:26 AM
Subject: [Hdf-forum] Different data types

I would like to export large amounts of data in one file. The data variables can be of different types; but it is not known beforehand what they are. I saw examples where the Compound data type was used with a pre-defined "struct". In this case, I cannot pre-define my struct. I can ofcourse make everything string, but I want to take advantage of the faster speed of writing other types. How should I approach the problem? Do you have available examples? It would also be nice to make use of the H5NX library of Pedro Vicente for faster speed, but I deal with 2D matrices and different data types.

Thanks for the help!

________________________________
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org<mailto:Hdf-forum@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

________________________________

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender. Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of Exelis Inc. The recipient should check this e-mail and any attachments for the presence of viruses. Exelis Inc. accepts no liability for any damage caused by any virus transmitted by this e-mail.