Select query options

Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

Thanks

···

Sent from my iPhone

Hi,

But that approach would be slow right? assuming that we are dealing with the
worst possible volume of data.

···

________________________________
From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>

Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab
coz i wont know which region to look for. What is the expected result of my
query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your
complete table. Just use hyperslabs for reading your dataset in small chunks
(my suggestion is that these chunks should fit your CPU cache, for improved
performance).

--
Francesc Alted

If you don't know which region you should look, then you should look into
your complete table. Just use hyperslabs for reading your dataset in small
chunks (my suggestion is that these chunks should fit your CPU cache, for
improved performance).

···

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>

Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use
hyperslab coz i wont know which region to look for. What is the expected
result of my query is only 10 records out 12 million.

--
Francesc Alted

Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for.

What if the expected result of my query is only 10 records out 12 million.

Would you be kind enough to prove some code snippet in C.

Sorry newbie here

···

Thanks

Sent from my iPhone

Hi,

  HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

     Werner

···

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

  But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>

To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>

Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).
--Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi all,

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

  We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

  Quincey

···

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

    Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

  But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

May be it would help you looking at the general algorithm used by PyTables.
It is written in Pyrex (a mix of Python and C), but it is easy enough for
you to get an idea. You can see it at:

http://www.pytables.org/trac/browser/trunk/tables/tableExtension.pyx
** *
*(look at method `__next__inKernel`)

Hope it helps,

···

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>

Would you be kind enough to prove some code snippet in C.

--
Francesc Alted

Hi,

Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5

···

Sent from my iPhone

On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi all,

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

  We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

  Quincey

    Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

  But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?

Thanks,

-Roy M.

···

On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:

Hi,

Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5

Sent from my iPhone

On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi all,

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

  We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

  Quincey

    Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

  But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

https://codeforge.lbl.gov/frs/?group_id=44

should do?

  Werner

···

On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:

On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?

Thanks,

-Roy M.

On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:

Hi,

Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5

Sent from my iPhone

On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi all,

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

  We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

  Quincey

    Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

  But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

The link that you've given is fastbit and not for hdf5- fastquery. I guess fastquery wont be out that soon.

···

Sent from my iPhone

On Aug 14, 2010, at 6:26 PM, "Werner Benger" <werner@cct.lsu.edu> wrote:

https://codeforge.lbl.gov/frs/?group_id=44

should do?

   Werner

On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:

On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?

Thanks,

-Roy M.

On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:

Hi,

Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5

Sent from my iPhone

On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi all,

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

   We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

   Quincey

   Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

   But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Yes, that was my reaction also. I can find fastbits but not hdf5-fastquery. If the FastBits API can be used as is to create HDF5-FastQuery it is unclear to me.

-Roy

···

On Aug 14, 2010, at 7:33 AM, Rodel Piano wrote:

The link that you've given is fastbit and not for hdf5- fastquery. I guess fastquery wont be out that soon.

Sent from my iPhone

On Aug 14, 2010, at 6:26 PM, "Werner Benger" <werner@cct.lsu.edu> wrote:

https://codeforge.lbl.gov/frs/?group_id=44

should do?

Werner

On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:

On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?

Thanks,

-Roy M.

On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:

Hi,

Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5

Sent from my iPhone

On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi all,

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

Quincey

Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

hi,

this morning I'm curious about HDF5-Fast Query and try to search if there is a free trial for that API, and when I look into this thread, it is fit to what I'm looking for. Hope that someone got the information about this API.

thanks

···

--- On Sat, 8/14/10, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:

From: Roy Mendelssohn <Roy.Mendelssohn@noaa.gov>
Subject: Re: [Hdf-forum] Select query options
To: "Rodel Piano" <rodel_piano@yahoo.com>
Cc: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Saturday, August 14, 2010, 11:16 AM

Yes, that was my reaction also. I can find fastbits but not hdf5-fastquery. If the FastBits API can be used as is to create HDF5-FastQuery it is unclear to me.

-Roy

On Aug 14, 2010, at 7:33 AM, Rodel Piano wrote:

The link that you've given is fastbit and not for hdf5- fastquery. I guess fastquery wont be out that soon.

Sent from my iPhone

On Aug 14, 2010, at 6:26 PM, "Werner Benger" <werner@cct.lsu.edu> wrote:

https://codeforge.lbl.gov/frs/?group_id=44

should do?

Werner

On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:

On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?

Thanks,

-Roy M.

On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:

Hi,

Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5

Sent from my iPhone

On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:

Hi all,

On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:

Hi,

HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:

http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
http://crd.lbl.gov/~kewu/fastbit/doc/index.html

This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.

We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...

Quincey

Werner

On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:

Hi,

But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.

From: Francesc Alted <faltet@pytables.org>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Sent: Fri, August 13, 2010 10:30:56 PM
Subject: Re: [Hdf-forum] Select query options

2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
Thanks for thr reply Francesc.

My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.

If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).

--
Francesc Alted

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

You and me both :slight_smile:

···

Sent from my iPhone

On Aug 18, 2010, at 12:35 PM, elisa sibarani <internisi@yahoo.com> wrote:

hi,

this morning I'm curious about HDF5-Fast Query and try to search if there is a free trial for that API, and when I look into this thread, it is fit to what I'm looking for. Hope that someone got the information about this API.

thanks

--- On Sat, 8/14/10, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:

From: Roy Mendelssohn <Roy.Mendelssohn@noaa.gov>
Subject: Re: [Hdf-forum] Select query options
To: "Rodel Piano" <rodel_piano@yahoo.com>
Cc: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Saturday, August 14, 2010, 11:16 AM

Yes, that was my reaction also. I can find fastbits but not hdf5-fastquery. If the FastBits API can be used as is to create HDF5-FastQuery it is unclear to me.

-Roy

On Aug 14, 2010, at 7:33 AM, Rodel Piano wrote:

> The link that you've given is fastbit and not for hdf5- fastquery. I guess fastquery wont be out that soon.
>
> Sent from my iPhone
>
> On Aug 14, 2010, at 6:26 PM, "Werner Benger" <werner@cct.lsu.edu> wrote:
>
>> https://codeforge.lbl.gov/frs/?group_id=44
>>
>> should do?
>>
>>
>> Werner
>>
>>
>> On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:
>>
>>> On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?
>>>
>>> Thanks,
>>>
>>> -Roy M.
>>>
>>> On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:
>>>
>>>> Hi,
>>>>
>>>> Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:
>>>>>>
>>>>>> http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
>>>>>> http://crd.lbl.gov/~kewu/fastbit/doc/index.html
>>>>>>
>>>>>> This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.
>>>>>
>>>>> We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...
>>>>>
>>>>> Quincey
>>>>>
>>>>>> Werner
>>>>>>
>>>>>>
>>>>>> On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.
>>>>>>
>>>>>> From: Francesc Alted <faltet@pytables.org>
>>>>>> To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
>>>>>> Sent: Fri, August 13, 2010 10:30:56 PM
>>>>>> Subject: Re: [Hdf-forum] Select query options
>>>>>>
>>>>>> 2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
>>>>>> Thanks for thr reply Francesc.
>>>>>>
>>>>>> My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.
>>>>>>
>>>>>> If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).
>>>>>>
>>>>>> --
>>>>>> Francesc Alted
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ___________________________________________________________________________
>>>>>> Dr. Werner Benger Visualization Research
>>>>>> Laboratory for Creative Arts and Technology (LCAT)
>>>>>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>>>>>> 211 Johnston Hall, Baton Rouge, Louisiana 70803
>>>>>> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
>>>>>> _______________________________________________
>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>> Hdf-forum@hdfgroup.org
>>>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> Hdf-forum@hdfgroup.org
>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> 1352 Lighthouse Avenue
>>> Pacific Grove, CA 93950-2097
>>>
>>> e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
>>> voice: (831)-648-9029
>>> fax: (831)-648-8440
>>> www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum@hdfgroup.org
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>>
>>
>>
>> --
>> ___________________________________________________________________________
>> Dr. Werner Benger Visualization Research
>> Laboratory for Creative Arts and Technology (LCAT)
>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>> 211 Johnston Hall, Baton Rouge, Louisiana 70803
>> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org