hi,
this morning I'm curious about HDF5-Fast Query and try to search if there is a free trial for that API, and when I look into this thread, it is fit to what I'm looking for. Hope that someone got the information about this API.
thanks
--- On Sat, 8/14/10, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:
From: Roy Mendelssohn <Roy.Mendelssohn@noaa.gov>
Subject: Re: [Hdf-forum] Select query options
To: "Rodel Piano" <rodel_piano@yahoo.com>
Cc: "HDF Users Discussion List" <hdf-forum@hdfgroup.org>
Date: Saturday, August 14, 2010, 11:16 AM
Yes, that was my reaction also. I can find fastbits but not hdf5-fastquery. If the FastBits API can be used as is to create HDF5-FastQuery it is unclear to me.
-Roy
On Aug 14, 2010, at 7:33 AM, Rodel Piano wrote:
> The link that you've given is fastbit and not for hdf5- fastquery. I guess fastquery wont be out that soon.
>
> Sent from my iPhone
>
> On Aug 14, 2010, at 6:26 PM, "Werner Benger" <werner@cct.lsu.edu> wrote:
>
>> https://codeforge.lbl.gov/frs/?group_id=44
>>
>> should do?
>>
>>
>> Werner
>>
>>
>> On Sat, 14 Aug 2010 07:00:57 +0200, Roy Mendelssohn <Roy.Mendelssohn@noaa.gov> wrote:
>>
>>> On the internet there is a reference to HDF5-FastQuery that uses FastBits within HDF5 files. But finding the actual API has eluded me. Does anyone know how to actually get this API?
>>>
>>> Thanks,
>>>
>>> -Roy M.
>>>
>>> On Aug 13, 2010, at 9:01 PM, Rodel Piano wrote:
>>>
>>>> Hi,
>>>>
>>>> Thats glad to know. I have explored fastbit, but fastbit requires that the file be converted from hdf5 to binary file. With that in mind, would it be ideal to just create the file in binary in the first place to reduce overhead. And use fastbit itself instead of hdf5
>>>>
>>>> Sent from my iPhone
>>>>
>>>> On Aug 14, 2010, at 4:44 AM, Quincey Koziol <koziol@hdfgroup.org> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> On Aug 13, 2010, at 12:57 PM, Werner Benger wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> HDF5 doesn't support searching within datasets. You might want to look into something like Fastbit:
>>>>>>
>>>>>> http://highscalability.com/blog/2009/5/1/fastbit-an-efficient-compressed-bitmap-index-technology.html
>>>>>> http://crd.lbl.gov/~kewu/fastbit/doc/index.html
>>>>>>
>>>>>> This is an addon-library which builds indices on HDF5 datasets allowing to select certain regions based on indexed properties.
>>>>>
>>>>> We (The HDF Group) have just had a proposal funded that will integrate FastBit [style] indexing more tightly with HDF5. The work is scheduled for a little while from now (year 2 or 3 of the proposal, I believe), but I just wanted to let people know that this functionality will be coming...
>>>>>
>>>>> Quincey
>>>>>
>>>>>> Werner
>>>>>>
>>>>>>
>>>>>> On Fri, 13 Aug 2010 19:40:12 +0200, Rodel Piano <rodel_piano@yahoo.com> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> But that approach would be slow right? assuming that we are dealing with the worst possible volume of data.
>>>>>>
>>>>>> From: Francesc Alted <faltet@pytables.org>
>>>>>> To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
>>>>>> Sent: Fri, August 13, 2010 10:30:56 PM
>>>>>> Subject: Re: [Hdf-forum] Select query options
>>>>>>
>>>>>> 2010/8/13 Rodel Piano <rodel_piano@yahoo.com>
>>>>>> Thanks for thr reply Francesc.
>>>>>>
>>>>>> My codes are in C. Lets say my records are about 12million. I cant use hyperslab coz i wont know which region to look for. What is the expected result of my query is only 10 records out 12 million.
>>>>>>
>>>>>> If you don't know which region you should look, then you should look into your complete table. Just use hyperslabs for reading your dataset in small chunks (my suggestion is that these chunks should fit your CPU cache, for improved performance).
>>>>>>
>>>>>> --
>>>>>> Francesc Alted
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> ___________________________________________________________________________
>>>>>> Dr. Werner Benger Visualization Research
>>>>>> Laboratory for Creative Arts and Technology (LCAT)
>>>>>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>>>>>> 211 Johnston Hall, Baton Rouge, Louisiana 70803
>>>>>> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
>>>>>> _______________________________________________
>>>>>> Hdf-forum is for HDF software users discussion.
>>>>>> Hdf-forum@hdfgroup.org
>>>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>>>
>>>> _______________________________________________
>>>> Hdf-forum is for HDF software users discussion.
>>>> Hdf-forum@hdfgroup.org
>>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>> **********************
>>> "The contents of this message do not reflect any position of the U.S. Government or NOAA."
>>> **********************
>>> Roy Mendelssohn
>>> Supervisory Operations Research Analyst
>>> NOAA/NMFS
>>> Environmental Research Division
>>> Southwest Fisheries Science Center
>>> 1352 Lighthouse Avenue
>>> Pacific Grove, CA 93950-2097
>>>
>>> e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
>>> voice: (831)-648-9029
>>> fax: (831)-648-8440
>>> www: http://www.pfeg.noaa.gov/
>>>
>>> "Old age and treachery will overcome youth and skill."
>>> "From those who have been given much, much will be expected"
>>>
>>>
>>> _______________________________________________
>>> Hdf-forum is for HDF software users discussion.
>>> Hdf-forum@hdfgroup.org
>>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>>>
>>>
>>
>>
>> --
>> ___________________________________________________________________________
>> Dr. Werner Benger Visualization Research
>> Laboratory for Creative Arts and Technology (LCAT)
>> Center for Computation & Technology at Louisiana State University (CCT/LSU)
>> 211 Johnston Hall, Baton Rouge, Louisiana 70803
>> Tel.: +1 225 578 4809 Fax.: +1 225 578-5362
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097
e-mail: Roy.Mendelssohn@noaa.gov (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org