hdf5 fastbit and fastquery

myang · April 5, 2016, 9:54pm

Hi,
Can someone tell me if fastbit & fastquery is part of hdf5 official release?
Any doc, tutorial, examples ?

Thanks,

This e-mail and its attachments are intended only for the individual or entity to whom it is addressed and may contain information that is confidential, privileged, inside information, or subject to other restrictions on use or disclosure. Any unauthorized use, dissemination or copying of this transmission or the information in it is prohibited and may be unlawful. If you have received this transmission in error, please notify the sender immediately by return e-mail, and permanently delete or destroy this e-mail, any attachments, and all copies (digital or paper). Unless expressly stated in this e-mail, nothing in this message should be construed as a digital or electronic signature. For additional important disclaimers and disclosures regarding KCG's products and services, please click on the following link:

http://www.kcg.com/legal/global-disclosures

epourmal · April 6, 2016, 1:31pm

Mike,

Fastbit and fastquery are not part of the HDF5 software developed and maintained by The HDF Group. It comes from NERSC; see https://sdm.lbl.gov/fastbit/

Elena

···

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 5, 2016, at 4:54 PM, Mike Yang <myang@kcg.com<mailto:myang@kcg.com>> wrote:

Hi,
Can someone tell me if fastbit & fastquery is part of hdf5 official release?
Any doc, tutorial, examples ?

Thanks,

This e-mail and its attachments are intended only for the individual or entity to whom it is addressed and may contain information that is confidential, privileged, inside information, or subject to other restrictions on use or disclosure. Any unauthorized use, dissemination or copying of this transmission or the information in it is prohibited and may be unlawful. If you have received this transmission in error, please notify the sender immediately by return e-mail, and permanently delete or destroy this e-mail, any attachments, and all copies (digital or paper). Unless expressly stated in this e-mail, nothing in this message should be construed as a digital or electronic signature. For additional important disclaimers and disclosures regarding KCG’s products and services, please click on the following link:

http://www.kcg.com/legal/global-disclosures _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

roy.mendelssohn · April 6, 2016, 2:55pm

Hi Elena:

A couple of years ago the HDF Group had adding this capability to HDF5 as a long term project. Is this still ongoing or has it been scrapped?

Thanks,

-Roy

···

On Apr 6, 2016, at 6:31 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

Mike,

Fastbit and fastquery are not part of the HDF5 software developed and maintained by The HDF Group. It comes from NERSC; see https://sdm.lbl.gov/fastbit/

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 5, 2016, at 4:54 PM, Mike Yang <myang@kcg.com> wrote:

Hi,
Can someone tell me if fastbit & fastquery is part of hdf5 official release?
Any doc, tutorial, examples ?

Thanks,

This e-mail and its attachments are intended only for the individual or entity to whom it is addressed and may contain information that is confidential, privileged, inside information, or subject to other restrictions on use or disclosure. Any unauthorized use, dissemination or copying of this transmission or the information in it is prohibited and may be unlawful. If you have received this transmission in error, please notify the sender immediately by return e-mail, and permanently delete or destroy this e-mail, any attachments, and all copies (digital or paper). Unless expressly stated in this e-mail, nothing in this message should be construed as a digital or electronic signature. For additional important disclaimers and disclosures regarding KCG’s products and services, please click on the following link:

http://www.kcg.com/legal/global-disclosures _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

epourmal · April 6, 2016, 3:46pm

Hi Roy,

···

On Apr 6, 2016, at 9:55 AM, Roy Mendelssohn - NOAA Federal <roy.mendelssohn@noaa.gov> wrote:

Hi Elena:

A couple of years ago the HDF Group had adding this capability to HDF5 as a long term project. Is this still ongoing or has it been scrapped?

At this point no decision was made. We have several features that were prototyped but were never productized, and indexing is one of them. It would take a lot of work to integrate indexing/querying feature into HDF5. It is also not clear if this is a right approach since we would be adding a lot of complexity to HDF5 increasing its maintenance cost. We need to evaluate other solutions that involve existing technologies. On another hand, it would be great to have indexing and querying in HDF5, wouldn’t it?

During this year we have to release promised HPC features (see https://hdfgroup.org/wp/2015/05/whats-coming-in-the-hdf5-1-10-0-release/), and we will be working on the roadmap for HDF5. Stay tuned. Input from this FORUM would be highly appreciated as always.

Thank you!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks,

-Roy

On Apr 6, 2016, at 6:31 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

Mike,

Fastbit and fastquery are not part of the HDF5 software developed and maintained by The HDF Group. It comes from NERSC; see https://sdm.lbl.gov/fastbit/

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 5, 2016, at 4:54 PM, Mike Yang <myang@kcg.com> wrote:

Hi,
Can someone tell me if fastbit & fastquery is part of hdf5 official release?
Any doc, tutorial, examples ?

Thanks,

This e-mail and its attachments are intended only for the individual or entity to whom it is addressed and may contain information that is confidential, privileged, inside information, or subject to other restrictions on use or disclosure. Any unauthorized use, dissemination or copying of this transmission or the information in it is prohibited and may be unlawful. If you have received this transmission in error, please notify the sender immediately by return e-mail, and permanently delete or destroy this e-mail, any attachments, and all copies (digital or paper). Unless expressly stated in this e-mail, nothing in this message should be construed as a digital or electronic signature. For additional important disclaimers and disclosures regarding KCG’s products and services, please click on the following link:

http://www.kcg.com/legal/global-disclosures _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

roy.mendelssohn · April 6, 2016, 4:20pm

Thanks. I do not know enough about the format of HDF5, nor the pros and cons of different indexing schemes. But for certain applications, some form of indexing would be nice. In the examples given in what I think they called “FastHDF” the range of queries of the data were much larger, and the speed of access was improved.

I also realize that a lot of what you put your resources to is what someone is willing to fund, and all I can offer is buying Ted a beer the next time I see him at a meeting.

-Roy

···

On Apr 6, 2016, at 8:46 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

Hi Roy,

On Apr 6, 2016, at 9:55 AM, Roy Mendelssohn - NOAA Federal <roy.mendelssohn@noaa.gov> wrote:

Hi Elena:

A couple of years ago the HDF Group had adding this capability to HDF5 as a long term project. Is this still ongoing or has it been scrapped?

At this point no decision was made. We have several features that were prototyped but were never productized, and indexing is one of them. It would take a lot of work to integrate indexing/querying feature into HDF5. It is also not clear if this is a right approach since we would be adding a lot of complexity to HDF5 increasing its maintenance cost. We need to evaluate other solutions that involve existing technologies. On another hand, it would be great to have indexing and querying in HDF5, wouldn’t it?

During this year we have to release promised HPC features (see https://hdfgroup.org/wp/2015/05/whats-coming-in-the-hdf5-1-10-0-release/), and we will be working on the roadmap for HDF5. Stay tuned. Input from this FORUM would be highly appreciated as always.

Thank you!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks,

-Roy

On Apr 6, 2016, at 6:31 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

Mike,

Fastbit and fastquery are not part of the HDF5 software developed and maintained by The HDF Group. It comes from NERSC; see https://sdm.lbl.gov/fastbit/

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 5, 2016, at 4:54 PM, Mike Yang <myang@kcg.com> wrote:

Hi,
Can someone tell me if fastbit & fastquery is part of hdf5 official release?
Any doc, tutorial, examples ?

Thanks,

This e-mail and its attachments are intended only for the individual or entity to whom it is addressed and may contain information that is confidential, privileged, inside information, or subject to other restrictions on use or disclosure. Any unauthorized use, dissemination or copying of this transmission or the information in it is prohibited and may be unlawful. If you have received this transmission in error, please notify the sender immediately by return e-mail, and permanently delete or destroy this e-mail, any attachments, and all copies (digital or paper). Unless expressly stated in this e-mail, nothing in this message should be construed as a digital or electronic signature. For additional important disclaimers and disclosures regarding KCG’s products and services, please click on the following link:

http://www.kcg.com/legal/global-disclosures _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

epourmal · April 6, 2016, 4:46pm

Thanks. I do not know enough about the format of HDF5, nor the pros and cons of different indexing schemes. But for certain applications, some form of indexing would be nice. In the examples given in what I think they called “FastHDF” the range of queries of the data were much larger, and the speed of access was improved.

I also realize that a lot of what you put your resources to is what someone is willing to fund, and all I can offer is buying Ted a beer the next time I see him at a meeting.

and help us with convincing NOAA leadership to contribute to the HDF5 support and development

Elena

-Roy

···

On Apr 6, 2016, at 11:20 AM, Roy Mendelssohn - NOAA Federal <roy.mendelssohn@noaa.gov<mailto:roy.mendelssohn@noaa.gov>> wrote:

On Apr 6, 2016, at 8:46 AM, Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>> wrote:

Hi Roy,

On Apr 6, 2016, at 9:55 AM, Roy Mendelssohn - NOAA Federal <roy.mendelssohn@noaa.gov<mailto:roy.mendelssohn@noaa.gov>> wrote:

Hi Elena:

A couple of years ago the HDF Group had adding this capability to HDF5 as a long term project. Is this still ongoing or has it been scrapped?

At this point no decision was made. We have several features that were prototyped but were never productized, and indexing is one of them. It would take a lot of work to integrate indexing/querying feature into HDF5. It is also not clear if this is a right approach since we would be adding a lot of complexity to HDF5 increasing its maintenance cost. We need to evaluate other solutions that involve existing technologies. On another hand, it would be great to have indexing and querying in HDF5, wouldn’t it?

During this year we have to release promised HPC features (see https://hdfgroup.org/wp/2015/05/whats-coming-in-the-hdf5-1-10-0-release/), and we will be working on the roadmap for HDF5. Stay tuned. Input from this FORUM would be highly appreciated as always.

Thank you!

Elena

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Thanks,

-Roy

On Apr 6, 2016, at 6:31 AM, Elena Pourmal <epourmal@hdfgroup.org<mailto:epourmal@hdfgroup.org>> wrote:

Mike,

Fastbit and fastquery are not part of the HDF5 software developed and maintained by The HDF Group. It comes from NERSC; see https://sdm.lbl.gov/fastbit/

Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal The HDF Group http://hdfgroup.org
1800 So. Oak St., Suite 203, Champaign IL 61820
217.531.6112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

On Apr 5, 2016, at 4:54 PM, Mike Yang <myang@kcg.com<mailto:myang@kcg.com>> wrote:

Hi,
Can someone tell me if fastbit & fastquery is part of hdf5 official release?
Any doc, tutorial, examples ?

Thanks,

This e-mail and its attachments are intended only for the individual or entity to whom it is addressed and may contain information that is confidential, privileged, inside information, or subject to other restrictions on use or disclosure. Any unauthorized use, dissemination or copying of this transmission or the information in it is prohibited and may be unlawful. If you have received this transmission in error, please notify the sender immediately by return e-mail, and permanently delete or destroy this e-mail, any attachments, and all copies (digital or paper). Unless expressly stated in this e-mail, nothing in this message should be construed as a digital or electronic signature. For additional important disclaimers and disclosures regarding KCG’s products and services, please click on the following link:

http://www.kcg.com/legal/global-disclosures _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov<mailto:Roy.Mendelssohn@noaa.gov> www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov<mailto:Roy.Mendelssohn@noaa.gov> www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

roy.mendelssohn · April 6, 2016, 5:08pm

hahahahaha - I thought April Fool’s Day was a few days ago.

ROTFLMFAO,

-Roy

PS - I am sure Ted can give you good odds on that.

···

On Apr 6, 2016, at 9:46 AM, Elena Pourmal <epourmal@hdfgroup.org> wrote:

and help us with convincing NOAA leadership to contribute to the HDF5 support and development

Elena

**********************
"The contents of this message do not reflect any position of the U.S. Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@noaa.gov www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.