check if dataset exists

Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

Please try H5Lexists.

Elena

···

On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:

Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I was there in the docu, but concluded it is meant only for links. It indeed works for regular datasets as well, maybe a small clarification in the documentation would help.
Many thanks,
Dominik

Elena Pourmal wrote:

···

Please try H5Lexists.

Elena
On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:

Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Just to pile on a little bit. For such an important little function, there isn't a H5L::exists in the HDF5DotNet wrapper...

Scott

···

-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Dominik Szczerba
Sent: Monday, October 19, 2009 3:33 PM
To: hdf-forum@hdfgroup.org
Subject: Re: [Hdf-forum] check if dataset exists

I was there in the docu, but concluded it is meant only for links. It
indeed works for regular datasets as well, maybe a small clarification
in the documentation would help.
Many thanks,
Dominik

Elena Pourmal wrote:
> Please try H5Lexists.
>
> Elena
> On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:
>
>> Is there a clean way to check if a given dataset exists?
>> I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors
>> are contaminating my output. I can not find any H5Dexist....
>> regards,
>> Dominik
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@hdfgroup.org
>> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@hdfgroup.org
> http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
>

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.

Hi Dominik,

I was there in the docu, but concluded it is meant only for links. It indeed works for regular datasets as well, maybe a small clarification in the documentation would help.

  Objects in the file (including datasets) don't have names. The links to the objects have names. Imagine you have two hard links to the same file in UNIX - each link can have a different name, so the actual file the links point to doesn't have a name.

  Quincey

···

On Oct 19, 2009, at 2:32 PM, Dominik Szczerba wrote:

Many thanks,
Dominik

Elena Pourmal wrote:

Please try H5Lexists.
Elena
On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:

Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi Quincey,

you're right but I think Dominik is not wrong either. If you go through the
user's guide, you'll notice that the link functions are under the group
description, somehow unrelated to datasets. Maybe a rearangement or
crosslink could help here.

HTH

-- dimitris

···

2009/10/20 Quincey Koziol <koziol@hdfgroup.org>

Hi Dominik,

On Oct 19, 2009, at 2:32 PM, Dominik Szczerba wrote:

I was there in the docu, but concluded it is meant only for links. It

indeed works for regular datasets as well, maybe a small clarification in
the documentation would help.

       Objects in the file (including datasets) don't have names. The
links to the objects have names. Imagine you have two hard links to the
same file in UNIX - each link can have a different name, so the actual file
the links point to doesn't have a name.

       Quincey

Many thanks,

Dominik

Elena Pourmal wrote:

Please try H5Lexists.
Elena
On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:

Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors
are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Hi,

On another note: I am wondering if it would be helpful to add high-level
APIs that will be more intuitive, for example, H5Dexists will find all
datasets in the file with the specified "name", e.g., if provided "A",
H5Dexists, will return all paths to existing datasets pointed by a link
(soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/D/A" (or something like
this)?

If you're interested in adding to the API, I would suggest starting
with something like "H5Oexists", which would take an arbitrary
(relative or absolute) path. Right now you have to (1) manually spit
your path up and call H5Lexists on each component, or (2) temporarily
disable errors while calling H5Gget_info or H5Oget_info. This is the
source of a lot of confusion; I've seen this question on the list many
times and it was the source of one bug in my application. In
principle, asking if a path is valid and receiving a "no" answer
should not be handled via error logic. In high-level languages this
sort of container testing can be quite frequent.

Your high-level API proposal sounds interesting. An alternative would
be to publish a "standard" code snippet showing how to do this with
H5Ovisit/visit_by_name, which shouldn't be too hard. The callback
would simply have to check the "type" field of the H5O_type_t
structure for each kind of object.

Andrew Collette

All,

There is no doubt our UG needs some substantial work (it is still oriented toward 1.6 data model and APIs ;-). We are trying to bump priority of this work.

On another note: I am wondering if it would be helpful to add high-level APIs that will be more intuitive, for example, H5Dexists will find all datasets in the file with the specified "name", e.g., if provided "A", H5Dexists, will return all paths to existing datasets pointed by a link (soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/D/A" (or something like this)?

Any use cases for such high-level APIs will be highly appreciated.

Thank you!

Elena

···

On Oct 19, 2009, at 5:10 PM, Dimitris Servis wrote:

Hi Quincey,

you're right but I think Dominik is not wrong either. If you go through the user's guide, you'll notice that the link functions are under the group description, somehow unrelated to datasets. Maybe a rearangement or crosslink could help here.

HTH

-- dimitris

2009/10/20 Quincey Koziol <koziol@hdfgroup.org>
Hi Dominik,

On Oct 19, 2009, at 2:32 PM, Dominik Szczerba wrote:

I was there in the docu, but concluded it is meant only for links. It indeed works for regular datasets as well, maybe a small clarification in the documentation would help.

       Objects in the file (including datasets) don't have names. The links to the objects have names. Imagine you have two hard links to the same file in UNIX - each link can have a different name, so the actual file the links point to doesn't have a name.

       Quincey

Many thanks,
Dominik

Elena Pourmal wrote:
Please try H5Lexists.
Elena
On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:
Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG errors are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

All,

There is no doubt our UG needs some substantial work (it is still
oriented toward 1.6 data model and APIs ;-). We are trying to bump
priority of this work.

On another note: I am wondering if it would be helpful to add high-
level APIs that will be more intuitive, for example, H5Dexists will
find all datasets in the file with the specified "name", e.g., if
provided "A", H5Dexists, will return all paths to existing datasets
pointed by a link (soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/
D/A" (or something like this)?

Any use cases for such high-level APIs will be highly appreciated.

Hm, would such a functionality be efficiently fast? I'm using something
like this, but via some kind of reverse lookup.

The use case is a scenario where data are stored like this:

/T=0.0/alpha
/T=0.0/beta
/T=1.0/alpha
/T=2.0/alpha
/T=2.0/beta

And your H5Dexists("alpha") function would return the paths
of 0.0,1.0,2.0 , respectively.

Such a function would be pretty useful, but if it's done
via iteration over the entire file content, then it would
be awfully slow and not practical.

  Werner

···

On Mon, 19 Oct 2009 18:30:38 -0500, Elena Pourmal <epourmal@hdfgroup.org> wrote:

Thank you!

Elena
On Oct 19, 2009, at 5:10 PM, Dimitris Servis wrote:

Hi Quincey,

you're right but I think Dominik is not wrong either. If you go
through the user's guide, you'll notice that the link functions are
under the group description, somehow unrelated to datasets. Maybe a
rearangement or crosslink could help here.

HTH

-- dimitris

2009/10/20 Quincey Koziol <koziol@hdfgroup.org>
Hi Dominik,

On Oct 19, 2009, at 2:32 PM, Dominik Szczerba wrote:

I was there in the docu, but concluded it is meant only for links.
It indeed works for regular datasets as well, maybe a small
clarification in the documentation would help.

       Objects in the file (including datasets) don't have names.
The links to the objects have names. Imagine you have two hard
links to the same file in UNIX - each link can have a different
name, so the actual file the links point to doesn't have a name.

       Quincey

Many thanks,
Dominik

Elena Pourmal wrote:
Please try H5Lexists.
Elena
On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:
Is there a clean way to check if a given dataset exists?
I do H5Dopen2 and see what it returns, but a lot of HDF5-DIAG
errors are contaminating my output. I can not find any H5Dexist....
regards,
Dominik

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

Hi Elena

···

2009/10/20 Elena Pourmal <epourmal@hdfgroup.org>

On another note: I am wondering if it would be helpful to add high-level
APIs that will be more intuitive, for example, H5Dexists will find all
datasets in the file with the specified "name", e.g., if provided "A",
H5Dexists, will return all paths to existing datasets pointed by a link
(soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/D/A" (or something like
this)?

Any use cases for such high-level APIs will be highly appreciated.

Sure, a relational database :wink: seriously, maybe it will help some, but will
confuse the many, and already the HDF5 interface is more cluttered than most
people can handle.

HTH

-- dimitris

Elena Pourmal wrote:

All,

There is no doubt our UG needs some substantial work (it is still oriented toward 1.6 data model and APIs ;-). We are trying to bump priority of this work.

On another note: I am wondering if it would be helpful to add high-level APIs that will be more intuitive, for example, H5Dexists will find all datasets in the file with the specified "name", e.g., if provided "A", H5Dexists, will return all paths to existing datasets pointed by a link (soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/D/A" (or something like this)?

I would separate (say) H5Dexist from (say) H5Dfind. And yes, along with compression, those would be one of my favorite high level additions.

with regards,
Dominik

···

Any use cases for such high-level APIs will be highly appreciated.

Thank you!

Elena
On Oct 19, 2009, at 5:10 PM, Dimitris Servis wrote:

Hi Quincey,

you're right but I think Dominik is not wrong either. If you go through the user's guide, you'll notice that the link functions are under the group description, somehow unrelated to datasets. Maybe a rearangement or crosslink could help here.

HTH

-- dimitris

2009/10/20 Quincey Koziol <koziol@hdfgroup.org <mailto:koziol@hdfgroup.org>>

    Hi Dominik,

    On Oct 19, 2009, at 2:32 PM, Dominik Szczerba wrote:

        I was there in the docu, but concluded it is meant only for
        links. It indeed works for regular datasets as well, maybe a
        small clarification in the documentation would help.

           Objects in the file (including datasets) don't have names.
     The links to the objects have names. Imagine you have two hard
    links to the same file in UNIX - each link can have a different
    name, so the actual file the links point to doesn't have a name.

           Quincey

        Many thanks,
        Dominik

        Elena Pourmal wrote:

            Please try H5Lexists.
            Elena
            On Oct 19, 2009, at 1:53 PM, Dominik Szczerba wrote:

                Is there a clean way to check if a given dataset exists?
                I do H5Dopen2 and see what it returns, but a lot of
                HDF5-DIAG errors are contaminating my output. I can
                not find any H5Dexist....
                regards,
                Dominik

                _______________________________________________
                Hdf-forum is for HDF software users discussion.
                Hdf-forum@hdfgroup.org <mailto:Hdf-forum@hdfgroup.org>
                http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

            _______________________________________________
            Hdf-forum is for HDF software users discussion.
            Hdf-forum@hdfgroup.org <mailto:Hdf-forum@hdfgroup.org>
            http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

        _______________________________________________
        Hdf-forum is for HDF software users discussion.
        Hdf-forum@hdfgroup.org <mailto:Hdf-forum@hdfgroup.org>
        http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

    _______________________________________________
    Hdf-forum is for HDF software users discussion.
    Hdf-forum@hdfgroup.org <mailto:Hdf-forum@hdfgroup.org>
    http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org <mailto:Hdf-forum@hdfgroup.org>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

------------------------------------------------------------------------

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

I agree with the suggestion of Dominik that it would be nice to have two functions H5Dexists and H5Dfind:
* H5Dexists should simple test the existence of a dataset, relative to the location "loc_id". If "loc_id" refers to /group1 and name is "A" or "/A" then the test is if dataset (or link to dataset ) /group1/A exists
* H5Dfind will find all datasets in the file with the specified "name" (which can indeed take a long time...)

It would be nice to have also H5Aexists, H5Gexists and H5Gfind(?).

Best regards, Richard

Dominik Szczerba wrote:

···

Elena Pourmal wrote:

All,

There is no doubt our UG needs some substantial work (it is still oriented toward 1.6 data model and APIs ;-). We are trying to bump priority of this work.

On another note: I am wondering if it would be helpful to add high-level APIs that will be more intuitive, for example, H5Dexists will find all datasets in the file with the specified "name", e.g., if provided "A", H5Dexists, will return all paths to existing datasets pointed by a link (soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/D/A" (or something like this)?

I would separate (say) H5Dexist from (say) H5Dfind. And yes, along with compression, those would be one of my favorite high level additions.

with regards,
Dominik

Any use cases for such high-level APIs will be highly appreciated.

Thank you!

All,

Thank you for your responses!

To address the clutter of the current HDF5 APIs: we created HDF5 C library to provide functionality; high-level APIs are convenience APIs, i.e., two sets address different needs and target different kind of users. Performance of the HL APIs shouldn't be worse than performance of a well written piece of the HDF5 code.

Thanks again and have a great day!

Elena

···

On Oct 20, 2009, at 3:08 AM, Richard van Hees wrote:

I agree with the suggestion of Dominik that it would be nice to have two functions H5Dexists and H5Dfind:
* H5Dexists should simple test the existence of a dataset, relative to the location "loc_id". If "loc_id" refers to /group1 and name is "A" or "/A" then the test is if dataset (or link to dataset ) /group1/A exists
* H5Dfind will find all datasets in the file with the specified "name" (which can indeed take a long time...)

It would be nice to have also H5Aexists, H5Gexists and H5Gfind(?).

Best regards, Richard

Dominik Szczerba wrote:

Elena Pourmal wrote:

All,

There is no doubt our UG needs some substantial work (it is still oriented toward 1.6 data model and APIs ;-). We are trying to bump priority of this work.

On another note: I am wondering if it would be helpful to add high-level APIs that will be more intuitive, for example, H5Dexists will find all datasets in the file with the specified "name", e.g., if provided "A", H5Dexists, will return all paths to existing datasets pointed by a link (soft, hard, TBD) "A", such as "/A", "/A/A", "/B/C/D/A" (or something like this)?

I would separate (say) H5Dexist from (say) H5Dfind. And yes, along with compression, those would be one of my favorite high level additions.

with regards,
Dominik

Any use cases for such high-level APIs will be highly appreciated.

Thank you!

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

H5Dfind() might become quite a complex function. I would assume it would
make use of some callback function that allows stopping the iteration.
It might also need parameters to control how the search is performed,
such as how many levels deep in the group hierarchy it would go; if
it should first traverse deep into the hierarchy, or first iterate
over the topmost groups; if it should inspect mounted external files
as well or just do it locally on one file; maybe want to have regular
pattern matching here as well... maybe want to have a generic find()
function that looks for datasets and groups together.

I have the impression that the wishes one could impose onto a find()
function would rather deserve creation of its own library as extension
to the core library. That one could then also provide means to speed
up the find process by saving lookup tables, like arrays that map
dataset names to links to the various datasets, or similar.

  Werner

···

On Tue, 20 Oct 2009 03:08:55 -0500, Richard van Hees <r.m.van.hees@sron.nl> wrote:

I agree with the suggestion of Dominik that it would be nice to have two
functions H5Dexists and H5Dfind:
* H5Dexists should simple test the existence of a dataset, relative to
the location "loc_id". If "loc_id" refers to /group1 and name is "A" or
"/A" then the test is if dataset (or link to dataset ) /group1/A exists
* H5Dfind will find all datasets in the file with the specified "name"
(which can indeed take a long time...)

It would be nice to have also H5Aexists, H5Gexists and H5Gfind(?).

Best regards, Richard

Dominik Szczerba wrote:

Elena Pourmal wrote:

All,

There is no doubt our UG needs some substantial work (it is still
oriented toward 1.6 data model and APIs ;-). We are trying to bump
priority of this work.

On another note: I am wondering if it would be helpful to add
high-level APIs that will be more intuitive, for example, H5Dexists
will find all datasets in the file with the specified "name", e.g.,
if provided "A", H5Dexists, will return all paths to existing
datasets pointed by a link (soft, hard, TBD) "A", such as "/A",
"/A/A", "/B/C/D/A" (or something like this)?

I would separate (say) H5Dexist from (say) H5Dfind. And yes, along
with compression, those would be one of my favorite high level additions.

with regards,
Dominik

Any use cases for such high-level APIs will be highly appreciated.

Thank you!

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Laboratory for Creative Arts and Technology (LCAT)
Center for Computation & Technology at Louisiana State University (CCT/LSU)
211 Johnston Hall, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362