Ruby HDF library

Hello
I have come across HDF5 and it seems interesting. I was wondering if any effort is underway on a Ruby library?

Google indicates people have made enquiries: http://www.ruby-forum.com/topic/72805

and there is a ruby netcdf library:
http://ruby.gfd-dennou.org/products/ruby-netcdf/

but nothing more. I started to toy a few weeks ago but before going further I thought I should check if anyone else is interested, I'd planned on putting my efforts here:
http://github.com/hedge-hog/hdf-ruby

At themoment there is nothing more than a skeleton.

Some questions in advance for anyone. FFI seems to be the new Ruby way to interface with other libraries:
http://kenai.com/projects/ruby-ffi/pages/Home

I have noticed for example that the Ruby MPI library (http://github.com/abedra/mpi-ruby/tree/master) does not seem to use this, so I'd appreciate any insights into whether ffi would be a good choice for HDF or is there some 'trap-for-young-players' lurking?

I'm pretty new to both ffi and HDF so temper your expectations, but would welcome any feedback.

best
hh

···

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα [The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)

http://wiki.hedgehogshiatus.com

Hi hh,

We have a high-level language(including Ruby) discussion wiki supported by hdfgroup(http://hdf5dev.pbworks.com/High+level+language+supports+for+HDF5).
You may put your comments there.

Please let me know if you need any assistance.

Thanks,

Kent
hedgehogshiatus@gmail.com wrote:

···

Hello
I have come across HDF5 and it seems interesting. I was wondering if any effort is underway on a Ruby library?

Google indicates people have made enquiries: http://www.ruby-forum.com/topic/72805

and there is a ruby netcdf library:
http://ruby.gfd-dennou.org/products/ruby-netcdf/

but nothing more. I started to toy a few weeks ago but before going further I thought I should check if anyone else is interested, I'd planned on putting my efforts here:
http://github.com/hedge-hog/hdf-ruby

At themoment there is nothing more than a skeleton.

Some questions in advance for anyone. FFI seems to be the new Ruby way to interface with other libraries:
http://kenai.com/projects/ruby-ffi/pages/Home

I have noticed for example that the Ruby MPI library (http://github.com/abedra/mpi-ruby/tree/master) does not seem to use this, so I'd appreciate any insights into whether ffi would be a good choice for HDF or is there some 'trap-for-young-players' lurking?

I'm pretty new to both ffi and HDF so temper your expectations, but would welcome any feedback.

best
hh

--
****************************************
Kent Yang
The HDF Group
1901 South First Street, Suite C-2
Champaign, IL 61820

myang6@hdfgroup.org
(217)265-5129 (office) (217)333-9049 (fax)
****************************************

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi,
I wrote some things on the page Kent has referred to but have not
proceeded beyond that - I have some other work in front of me right
now and for the foreseeable future.
The only other thought I'd add for a Ruby library, rather than HDF's
HLLs in general, is to evaluate Ruby's NArray library. It is used by
the netcdf Ruby library so you may find some insights there?
I really can't say anything useful about Ruby's MPI, the last time I
used MPI was on Windows XP about 6-7 years ago so am well and truly
rusty.
HTH. Good luck.
Cheers
Mark

···

On Fri, Jun 19, 2009 at 1:58 AM, MuQun Yang<myang6@hdfgroup.org> wrote:

Hi hh,

We have a high-level language(including Ruby) discussion wiki supported by
hdfgroup(http://hdf5dev.pbworks.com/High+level+language+supports+for+HDF5).
You may put your comments there.

Please let me know if you need any assistance.

Thanks,

Kent
hedgehogshiatus@gmail.com wrote:

Hello
I have come across HDF5 and it seems interesting. I was wondering if any
effort is underway on a Ruby library?

Google indicates people have made enquiries:
http://www.ruby-forum.com/topic/72805

and there is a ruby netcdf library:
http://ruby.gfd-dennou.org/products/ruby-netcdf/

but nothing more. I started to toy a few weeks ago but before going
further I thought I should check if anyone else is interested, I'd planned
on putting my efforts here:
http://github.com/hedge-hog/hdf-ruby

At themoment there is nothing more than a skeleton.

Some questions in advance for anyone. FFI seems to be the new Ruby way to
interface with other libraries:
http://kenai.com/projects/ruby-ffi/pages/Home

I have noticed for example that the Ruby MPI library
(http://github.com/abedra/mpi-ruby/tree/master) does not seem to use this,
so I'd appreciate any insights into whether ffi would be a good choice for
HDF or is there some 'trap-for-young-players' lurking?

I'm pretty new to both ffi and HDF so temper your expectations, but would
welcome any feedback.

best
hh

--
****************************************
Kent Yang
The HDF Group
1901 South First Street, Suite C-2
Champaign, IL 61820

myang6@hdfgroup.org
(217)265-5129 (office) (217)333-9049 (fax)
****************************************

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi hh,

We have a high-level language(including Ruby) discussion wiki supported by
hdfgroup(http://hdf5dev.pbworks.com/High+level+language+supports+for+HDF5).
You may put your comments there.

Please let me know if you need any assistance.

Hi Kent,
Thanks for the offer. Some questions follow.
The HLL discussion pages points to a PPT presentation on swig, mentioning that the swig interface file could be distributed:
1) Are these swig interface files available?
2) How do they compare to permafrost's swig interface files, see http://code.google.com/p/permafrost.
3) Anyother known sources of hdf5 swig interface files? Any considered to be the reference interface definition?

FYI I'm considering generating the intial ffi definitions from swig and then hand optimizing where needed - may not be a feasible approach - I'll suck it and see :slight_smile:

Best wishes hh

···

On Fri, Jun 19, 2009 at 1:58 AM, MuQun Yang<myang6@hdfgroup.org> wrote:

Thanks,

Kent
hedgehogshiatus@gmail.com wrote:

Hello
I have come across HDF5 and it seems interesting. I was wondering if any
effort is underway on a Ruby library?

Google indicates people have made enquiries:
http://www.ruby-forum.com/topic/72805

and there is a ruby netcdf library:
http://ruby.gfd-dennou.org/products/ruby-netcdf/

but nothing more. I started to toy a few weeks ago but before going
further I thought I should check if anyone else is interested, I'd planned
on putting my efforts here:
http://github.com/hedge-hog/hdf-ruby

At themoment there is nothing more than a skeleton.

Some questions in advance for anyone. FFI seems to be the new Ruby way to
interface with other libraries:
http://kenai.com/projects/ruby-ffi/pages/Home

I have noticed for example that the Ruby MPI library
(http://github.com/abedra/mpi-ruby/tree/master) does not seem to use this,
so I'd appreciate any insights into whether ffi would be a good choice for
HDF or is there some 'trap-for-young-players' lurking?

I'm pretty new to both ffi and HDF so temper your expectations, but would
welcome any feedback.

best
hh

--
****************************************
Kent Yang
The HDF Group
1901 South First Street, Suite C-2
Champaign, IL 61820

myang6@hdfgroup.org
(217)265-5129 (office) (217)333-9049 (fax)
****************************************

--
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα [The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)

http://wiki.hedgehogshiatus.com

Greetings,

wrote:
I wrote the Swig interface referred to here and then abandoned it after
the h5py package became available. My interface was very rudimentary
and rather specific to Python. I was trying to do a minimal amount of
work and let the SWIG do most of the work.
A number of problems cropped up during its development.

  1. The use of macros in the C API of HDF5 causes problems because SWIG
    cannot automatically analyze them. (I would like to see them removed
    though I can’t see how this can be done without an API change.) I got
    around this by cheating, but the right way to to do is to write code to
    wrap the macros by hand.
  2. Later versions SWIG became rather strict about not allowing python
    to convert integers to pointers. The numpy module for Python can
    provide a pointer to array memory as a Python integer. The HDF5 C API
    wants void *, etc. and the Python SWIG wrapper reflects this. I
    started a C conversion library which was nothing more than functions of
    integers which cast the argument into a a pointer, eg:
    When wrapped by SWIG this provided the necessary type conversion
    functions in Python. The opaque object returned to the Python
    interpreter could be passed to the SWIG interface to HDF5.
    If you want to develop a Ruby interface I would recommend starting by
    looking at Andrew Collette’s excellent h5py interface. It was written
    using a Python specific C interface so the code itself probably isn’t
    useful to Ruby, but it will give you a sense of the effort involved.
    If you are familiar with Python you’ll also see the nice way in which
    h5py “shoe horns” the API and HDF5 objects into the syntactic
    structures familiar to Python programmers. I imagine there are similar
    opportunities in developing the Ruby interface.
    Cheers,
    –dan
···

hedgehogshiatus@gmail.com

Hi Kent,

Thanks for the offer. Some questions follow.

The HLL discussion pages points to a PPT presentation on swig,
mentioning that the swig interface file could be distributed:

  1. Are these swig interface files available?

  2. How do they compare to permafrost’s swig interface files, see
    .

  3. Anyother known sources of hdf5 swig interface files? Any considered
    to be the reference interface definition?
    http://code.google.com/p/permafrost

void *Return_void (int NumPyInt) {

return (void *) NumPyInt;

}

-- Daniel Kahn
Science Systems and Applications Inc.
301-867-2162

Greetings,

Hi Kent,
Thanks for the offer. Some questions follow.
The HLL discussion pages points to a PPT presentation on swig, mentioning
that the swig interface file could be distributed:
1) Are these swig interface files available?
2) How do they compare to permafrost's swig interface files, see
http://code.google.com/p/permafrost.
3) Anyother known sources of hdf5 swig interface files? Any considered to
be the reference interface definition?

I wrote the Swig interface referred to here and then abandoned it after the
h5py package became available. My interface was very rudimentary and rather
specific to Python. I was trying to do a minimal amount of work and let the
SWIG do most of the work.

A number of problems cropped up during its development.

1) The use of macros in the C API of HDF5 causes problems because SWIG
cannot automatically analyze them. (I would like to see them removed though
I can't see how this can be done without an API change.) I got around this
by cheating, but the right way to to do is to write code to wrap the macros
by hand.

2) Later versions SWIG became rather strict about not allowing python to
convert integers to pointers. The numpy module for Python can provide a
pointer to array memory as a Python integer. The HDF5 C API wants void *,
etc. and the Python SWIG wrapper reflects this. I started a C conversion
library which was nothing more than functions of integers which cast the
argument into a a pointer, eg:

void *Return_void (int NumPyInt) {

  return (void *) NumPyInt;

}

When wrapped by SWIG this provided the necessary type conversion functions
in Python. The opaque object returned to the Python interpreter could be
passed to the SWIG interface to HDF5.

If you want to develop a Ruby interface I would recommend starting by
looking at Andrew Collette's excellent h5py interface. It was written using
a Python specific C interface so the code itself probably isn't useful to
Ruby, but it will give you a sense of the effort involved. If you are
familiar with Python you'll also see the nice way in which h5py "shoe horns"
the API and HDF5 objects into the syntactic structures familiar to Python
programmers. I imagine there are similar opportunities in developing the
Ruby interface.

Thanks for the feedback. I will look at h5py more closely.
After wrestling with Permafrost's swig interface files (Java centric)
I discovered that it is possible to get a lot of tedious lifting done
with ruby's ffi-swig library and a minimal interface file passed to
swig. Specifically:

# ------------------ start ------------------------
%module Hdf5
%{
require 'rubygems'
require 'ffi'
module Hdf5::Mixin
extend FFI::Library
%}
#include hdf5.h
%{
end
%}
# ------------------ end ------------------------

I generate a xml file with swig as follows:

swig -includeall -Wall -Wallkw
- -I/usr/lib/gcc/i586-suse-linux/4.3/include -I/usr/include
- -I/usr/src/hdf5-1.8.3/src -ruby -xmlout
~/hdf/ruby/hdf/ext/interfaces/hdf5.xml hdf5.i

This is then passed to Ruby's ffi-gen utility and a large part of the
tedious work is done. I just did this last night so how much hand
editing will be required remains to be seen.
Anyway it seems that the xml representation of the swig results is
what helps ffi-swig-generate - that pathway may be of interest to
some.

Best wishes
HH

Cheers,
--dan

--
Daniel Kahn
Science Systems and Applications Inc.
301-867-2162

- --
πόλλ' οἶδ ἀλώπηξ, ἀλλ' ἐχῖνος ἓν μέγα
[The fox knows many things, but the hedgehog knows one big thing.]
  Archilochus, Greek poet (c. 680 BC – c. 645 BC)

http://wiki.hedgehogshiatus.com

···

On Sat, Jun 27, 2009 at 7:34 AM, Daniel Kahn wrote:

hedgehogshiatus@gmail.com wrote:

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.