index methods in HDF5?

I'm new to HDF5 & was wondering if there are any recommended ways to add
indices to datasets in real-time. (e.g. if you have an array of x/y/z
components that's being extended in real time, add an index to speed up
queries for points with certain values of x/y/z.) The only obvious ways
I can think of are to use the groups themselves as discrete index keys,
or to build up your own index datastructures e.g. red-black trees,
b-trees, skip lists, etc., and I'm not sure which methods are
efficient/inefficient to update frequently in an HDF5 file.

Jason Sachs

p.s. is this mailing list archived anywhere? maybe someone's asked this
already.

···

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi Jason,

I'm new to HDF5 & was wondering if there are any recommended ways to add
indices to datasets in real-time. (e.g. if you have an array of x/y/z
components that's being extended in real time, add an index to speed up
queries for points with certain values of x/y/z.) The only obvious ways
I can think of are to use the groups themselves as discrete index keys,
or to build up your own index datastructures e.g. red-black trees,
b-trees, skip lists, etc., and I'm not sure which methods are
efficient/inefficient to update frequently in an HDF5 file.

  There have been several implementations of indexing HDF5 files, including a prototype that we (The HDF Group) did:

  PyTables/PyTables Pro: http://www.pytables.org/moin

  FastBit: http://sdm.lbl.gov/fastbit/

  HDF Group prototype: http://www.hdfgroup.uiuc.edu/RFC/HDF5/hdf5Indexing/index.html

  We do have some funding to implement "production level" indexing features within the HDF5 distribution for some common use cases and will be making more announcements about that work as it proceeds.

Jason Sachs

p.s. is this mailing list archived anywhere? maybe someone's asked this
already.

  Yes, the mailing list is archived, although you'll have to query the archives by e-mail and not the web. Send mail to 'hdf-form-help@hdfgroup.org' for instructions on how to retrieve messages from the archive.

  Quincey Koziol
  The HDF Group

···

On Mar 6, 2008, at 12:40 PM, Jason Sachs wrote:

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Yes, the mailing list is archived, although you'll have to query

the

archives by e-mail and not the web. Send mail to

'hdf-form-help@hdfgroup.org

' for instructions on how to retrieve messages from the archive.

hmmm... is there a way to temporarily unsubscribe, or subscribe in some
kind of "daily digest" mode?

Have you thought of creating an archive in Google Groups? (GG gives you
an email address that you can just add to an email mailing list, and you
get an instant archive from that point forward)

···

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

  Yes, the mailing list is archived, although you'll have to query

the

archives by e-mail and not the web. Send mail to

'hdf-form-help@hdfgroup.org

' for instructions on how to retrieve messages from the archive.

hmmm... is there a way to temporarily unsubscribe, or subscribe in some
kind of "daily digest" mode?

  Not as far as I know.

Have you thought of creating an archive in Google Groups? (GG gives you
an email address that you can just add to an email mailing list, and you
get an instant archive from that point forward)

  That sounds like a great idea - I'll check into it.

  Thanks,
    Quincey

···

On Mar 7, 2008, at 6:42 PM, Jason Sachs wrote:

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi all,

···

On Mar 10, 2008, at 2:57 PM, Quincey Koziol wrote:

On Mar 7, 2008, at 6:42 PM, Jason Sachs wrote:

  Yes, the mailing list is archived, although you'll have to query

the

archives by e-mail and not the web. Send mail to

'hdf-form-help@hdfgroup.org

' for instructions on how to retrieve messages from the archive.

hmmm... is there a way to temporarily unsubscribe, or subscribe in some
kind of "daily digest" mode?

  Not as far as I know.

Have you thought of creating an archive in Google Groups? (GG gives you
an email address that you can just add to an email mailing list, and you
get an instant archive from that point forward)

  That sounds like a great idea - I'll check into it.

  OK, I've created a Google Group for archiving the hdf-forum@hdfgroup.org mailing list here:

http://groups.google.com/group/hdf-forum/

  Quincey

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

Hi Elena,

Hi Quincey,

Does it mean that all emails that go to hdf-forum could be seen on-line now?

  Yes.

What about previous emails?

  Those are not archived by Google.

I joined the group, but I am not sure how it will work :wink:

  You don't need to join the Google group, it is just there to archive the discussions on the hdf-forum@hdfgroup.org mailing list - send all the messages to the @hdfgroup.org mailing list.

  Quincey

···

On Mar 10, 2008, at 6:51 PM, Elena Pourmal wrote:

Elena

At 5:12 PM -0500 3/10/08, Quincey Koziol wrote:

Hi all,

On Mar 10, 2008, at 2:57 PM, Quincey Koziol wrote:

On Mar 7, 2008, at 6:42 PM, Jason Sachs wrote:

  Yes, the mailing list is archived, although you'll have to query

the

archives by e-mail and not the web. Send mail to

'hdf-form-help@hdfgroup.org

' for instructions on how to retrieve messages from the archive.

hmmm... is there a way to temporarily unsubscribe, or subscribe in some
kind of "daily digest" mode?

  Not as far as I know.

Have you thought of creating an archive in Google Groups? (GG gives you
an email address that you can just add to an email mailing list, and you
get an instant archive from that point forward)

  That sounds like a great idea - I'll check into it.

  OK, I've created a Google Group for archiving the hdf-forum@hdfgroup.org mailing list here:

http://groups.google.com/group/hdf-forum/

  Quincey

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.

--

------------------------------------------------------------
Elena Pourmal
The HDF Group
1901 So First ST.
Suite C-2
Champaign, IL 61820

epourmal@hdfgroup.org
(217)333-0238 (office)
(217)333-9049 (fax)
------------------------------------------------------------

----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.