Question about domain name in h5 server


#1

Hi,

  I'm a new user of h5 server. On my test server, I installed and ran the h5serv docker container. From a remote browser, I could get response by sending http://128.XXX.XXX.XXX:5000/ to the server. However, when I tried to use curl-based request, such as "curl --header "Host: dp.h5_store.hdfgroup.org" http://128.XXX. XXX. XXX:5000/", an error "tornado.web.HTTPError: HTTP 404: Not Found" occurred. The h5 files I want the server to serve is under "/data/h5_store". Because the h5 server is installed on a machine without domain name (e.g. hdfgroup.org), how could I set up a valid HTTP header so that the curl command could be used to send the request?

  Thanks.

  * Yawei

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Computer Science Research Group
Computer Science and Mathematics Division
Oak Ridge National Laboratory
PO BOX 2008 MS6164
OAK RIDGE TN 37831-6164


#2

Hi Yawei,

   For curl, always use the dns name of the server the container is running on, i.e. the http://128… address you have.

   To address files, the pseudo-DNS name is created by taking the “domain” config value (in h5serv/server/config.py), and combining with the reverse path of the file relative to the data directory. There’s another config value: “datapath” that gives the location to the data directory. So in your case you should set this to: /data/h5_store. After this (and restarting the server), if you have “myfile.h5” in the h5_store directory, the h5serv host path to that file would be: myfile.hdfgroup.org.

   If you’ve setup a DNS entry for your server, say h5serv.ornl.gov, then you can update the domain config value to match. In that case you’d have myfile.h5serv.ornl.gov. The various config entries are documented here: http://h5serv.readthedocs.io/en/latest/Installation/ServerSetup.html#server-configuration.

   Hope this helps! Configuration is always the challenging aspect. You can email me directly (jreadey AT hdfgroup DOT org) if that doesn’t work.

Cheers,
John

···

From: Hdf-forum <hdf-forum-bounces@lists.hdfgroup.org> on behalf of "Hui, Yawei" <huiy@ornl.gov>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Date: Tuesday, October 24, 2017 at 9:20 AM
To: "hdf-forum@lists.hdfgroup.org" <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Question about domain name in h5 server

Hi,

  I’m a new user of h5 server. On my test server, I installed and ran the h5serv docker container. From a remote browser, I could get response by sending http://128.XXX.XXX.XXX:5000/ to the server. However, when I tried to use curl-based request, such as “curl --header "Host: dp.h5_store.hdfgroup.org" http://128.XXX. XXX. XXX:5000/”, an error “tornado.web.HTTPError: HTTP 404: Not Found” occurred. The h5 files I want the server to serve is under “/data/h5_store”. Because the h5 server is installed on a machine without domain name (e.g. hdfgroup.org), how could I set up a valid HTTP header so that the curl command could be used to send the request?

  Thanks.

- Yawei

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Computer Science Research Group
Computer Science and Mathematics Division
Oak Ridge National Laboratory
PO BOX 2008 MS6164
OAK RIDGE TN 37831-6164


#3

Hi All,

I've just discovered that when I create a dataset with one dimension which has a size = 0, then I cannot properly open such dataset - H5Dopen() returns -1.
But when I create similar dataset with one dimension which has a size at least 1 (even without writing data to such dataset), then I can properly open such dataset.

Is this a bug or a feature in the HDF5 library?

I think I should be able to open 0-size dimension dataset, just to create an attribute inside such dataset - and that was my intension - no data inside (thus dimension size = 0), but write some attributes to such dataset...

Best regards,
Rafal


#4

Rafal, this might be a bug. However, if you don't intend to store
data there's a cheaper and maybe less contorted option: You can
create a dataset with a dataspace of class H5S_NULL.
Think of that as a dataset whose dataspace is an empty set.
(Likewise you'd use H5S_SCALAR for a singleton.)

Best, G.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Rafal Lichwala
Sent: Thursday, October 26, 2017 3:00 AM
To: hdf-forum@lists.hdfgroup.org
Subject: [Hdf-forum] HDF5 library bug or feature?

Hi All,

I've just discovered that when I create a dataset with one dimension which has a size = 0, then I cannot properly open such dataset -
H5Dopen() returns -1.
But when I create similar dataset with one dimension which has a size at least 1 (even without writing data to such dataset), then I can properly open such dataset.

Is this a bug or a feature in the HDF5 library?

I think I should be able to open 0-size dimension dataset, just to create an attribute inside such dataset - and that was my intension - no data inside (thus dimension size = 0), but write some attributes to such dataset...

Best regards,
Rafal

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org


Twitter: https://twitter.com/hdf5


#5

Hi Gerd,

Thank you for an answer!

Rafal, this might be a bug.

Is there any way to record/track this issue?

However, if you don't intend to store
data there's a cheaper and maybe less contorted option: You can
create a dataset with a dataspace of class H5S_NULL.

Good point - I will remember about that possibility in the future - thank you.
For now I've just moved all attributes level up - to the parent group - works fine.

Best regards,
Rafal


#6

Rafal, this might be a bug.

Is there any way to record/track this issue?

Not sure. Helpdesk will check. (It's still early morning here...)

G.


#7

Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:

Rafal, this might be a bug. However, if you don't intend to store
data there's a cheaper and maybe less contorted option: You can
create a dataset with a dataspace of class H5S_NULL.
Think of that as a dataset whose dataspace is an empty set.
(Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider XML attributes as example). I tend to use empty groups which only serve as container of attributes. What do you think: would empty datasets (H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


#8

Andrey, the experts should chime in on the storage trade-offs.
(I imagine the null dataset is stored with compact layout, but in
newer versions of the file format an empty group is very "compact"
as well.) At the conceptual level, the two are very different and
that might tip the balance in either direction.
A group leaves the door open for other descendants.
A dataset is a terminal node and, despite being empty, is nevertheless
typed. (A null dataset has an element type!) Of course, you could
"type" a group with additional attributes, but it would be a very
different kind of variable.

This might be a case where there're no right or wrong choices
as far as HDF5 is concerned, and users will have to decide what
makes it right for them.

Best, G.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Thursday, October 26, 2017 6:52 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:

Rafal, this might be a bug. However, if you don't intend to store data
there's a cheaper and maybe less contorted option: You can create a
dataset with a dataspace of class H5S_NULL.
Think of that as a dataset whose dataspace is an empty set.
(Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider XML attributes as example). I tend to use empty groups which only serve as container of attributes. What do you think: would empty datasets
(H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org


Twitter: https://twitter.com/hdf5


#9

A choice consideration might be that the number of attributes has a
limit.

I think HDF5 should support empty N-dim datasets. Being empty also
conveys information.
Note that in any language it would be strange if you could not have an
empty sequence.

Ger

Gerd Heber <gheber@hdfgroup.org> 10/26/17 2:27 PM >>>

Andrey, the experts should chime in on the storage trade-offs.
(I imagine the null dataset is stored with compact layout, but in
newer versions of the file format an empty group is very "compact"
as well.) At the conceptual level, the two are very different and
that might tip the balance in either direction.
A group leaves the door open for other descendants.
A dataset is a terminal node and, despite being empty, is nevertheless
typed. (A null dataset has an element type!) Of course, you could
"type" a group with additional attributes, but it would be a very
different kind of variable.

This might be a case where there're no right or wrong choices
as far as HDF5 is concerned, and users will have to decide what
makes it right for them.

Best, G.

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf
Of ??? ???
Sent: Thursday, October 26, 2017 6:52 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:

Rafal, this might be a bug. However, if you don't intend to store data

there's a cheaper and maybe less contorted option: You can create a
dataset with a dataspace of class H5S_NULL.
Think of that as a dataset whose dataspace is an empty set.
(Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider
XML attributes as example). I tend to use empty groups which only serve
as container of attributes. What do you think: would empty datasets
(H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by
MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org


Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


#10

Hi Rafal,

I am able to create a dataset with zero dimensions, close it, and re-open. (See attached).
Can you send us an example of what you are trying that fails?
Please send it to the helpdesk: help@hdfgroup.org

Thanks!
-Barbara
help@hdfgroup.org

rafal.c (3.27 KB)

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Gerd Heber
Sent: Thursday, October 26, 2017 6:49 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Rafal, this might be a bug.

Is there any way to record/track this issue?

Not sure. Helpdesk will check. (It's still early morning here...)

G.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org


Twitter: https://twitter.com/hdf5


#11

26.10.2017 18:40, Ger van Diepen пишет:

A choice consideration might be that the number of attributes has a limit.

Hmm, do I understand correctly that this limit is different for a group and (an empty) dataset?

Best wishes,
Andrey Paramonov

···

I think HDF5 should support empty N-dim datasets. Being empty also conveys information.
Note that in any language it would be strange if you could not have an empty sequence.

Ger

Gerd Heber <gheber@hdfgroup.org> 10/26/17 2:27 PM >>>

Andrey, the experts should chime in on the storage trade-offs.
(I imagine the null dataset is stored with compact layout, but in
newer versions of the file format an empty group is very "compact"
as well.) At the conceptual level, the two are very different and
that might tip the balance in either direction.
A group leaves the door open for other descendants.
A dataset is a terminal node and, despite being empty, is nevertheless
typed. (A null dataset has an element type!) Of course, you could
"type" a group with additional attributes, but it would be a very
different kind of variable.

This might be a case where there're no right or wrong choices
as far as HDF5 is concerned, and users will have to decide what
makes it right for them.

Best, G.

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Thursday, October 26, 2017 6:52 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Gerd!

26.10.2017 14:31, Gerd Heber пишет:
> Rafal, this might be a bug. However, if you don't intend to store data
> there's a cheaper and maybe less contorted option: You can create a
> dataset with a dataspace of class H5S_NULL.
> Think of that as a dataset whose dataspace is an empty set.
> (Likewise you'd use H5S_SCALAR for a singleton.)

A common desire is to store non-uniform sparse data in HDF5 (consider XML attributes as example). I tend to use empty groups which only serve as container of attributes. What do you think: would empty datasets
(H5S_NULL) with attributes be more efficient (introduce less overhead)?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
This message has been scanned for viruses and
dangerous content by *MailScanner* <http://www.mailscanner.info/>, and is
believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.


#12

Hi Barbara,

Thank you for an answer!
Yes, your code also works fine on my side, BUT...
when I try to set some custom H5P_FILE_ACCESS by call:

H5Pset_fapl_mpio(plist_id, comm, info)

(so running this example under OpenMPI - even for 1 process only)
it DOES NOT work any more.
In that case (see attached slightly modified your code) I've got the following output:

2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fcreate: OK
2: H5Screate_simple: OK
2: H5Dcreate2: OK
2: H5Dclose: OK
2: H5Sclose: OK
2: H5Pclose: OK
2: H5Fclose: OK
2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fopen: OK
2: H5Dopen2 Failed!
2: H5Dget_space Failed!
2: H5Dclose Failed!
2: H5Sclose Failed!
2: H5Pclose: OK
2: H5Fclose: OK

HDF5 library also reports some error messages in that case:
  unable to open dataset
  unable to initialize file storage
  unable to mark dataspace as dirty
  unable to update layout info
  unable to write object header message
  unable to modify constant message

The most interesting thing is (which possibly means it's a bug), that when in the same code I just change dims definition to:

    dims[0] = 1;
    dims[1] = 1;

everything WORKS FINE now! :slight_smile:

Best regards,
Rafal

W dniu 2017-10-26 o 17:53, Barbara Jones pisze:

test.cpp (3.24 KB)

···

Hi Rafal,

I am able to create a dataset with zero dimensions, close it, and re-open. (See attached).
Can you send us an example of what you are trying that fails?
Please send it to the helpdesk: help@hdfgroup.org

Thanks!
-Barbara
help@hdfgroup.org

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Gerd Heber
Sent: Thursday, October 26, 2017 6:49 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Rafal, this might be a bug.

Is there any way to record/track this issue?

Not sure. Helpdesk will check. (It's still early morning here...)

G.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


#13

Hi Rafal,

Ahh! I found a bug report for this same issue (HDFFV-10001).
I added your information to the bug report.

Thanks!
-Barbara
help@hdfgroup.org

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Rafal Lichwala
Sent: Friday, October 27, 2017 4:06 AM
To: hdf-forum@lists.hdfgroup.org
Cc: HDF Helpdesk
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Hi Barbara,

Thank you for an answer!
Yes, your code also works fine on my side, BUT...
when I try to set some custom H5P_FILE_ACCESS by call:

H5Pset_fapl_mpio(plist_id, comm, info)

(so running this example under OpenMPI - even for 1 process only) it DOES NOT work any more.
In that case (see attached slightly modified your code) I've got the following output:

2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fcreate: OK
2: H5Screate_simple: OK
2: H5Dcreate2: OK
2: H5Dclose: OK
2: H5Sclose: OK
2: H5Pclose: OK
2: H5Fclose: OK
2: H5Pcreate: OK
2: H5Pset_fapl_mpio: OK
2: H5Fopen: OK
2: H5Dopen2 Failed!
2: H5Dget_space Failed!
2: H5Dclose Failed!
2: H5Sclose Failed!
2: H5Pclose: OK
2: H5Fclose: OK

HDF5 library also reports some error messages in that case:
  unable to open dataset
  unable to initialize file storage
  unable to mark dataspace as dirty
  unable to update layout info
  unable to write object header message
  unable to modify constant message

The most interesting thing is (which possibly means it's a bug), that when in the same code I just change dims definition to:

    dims[0] = 1;
    dims[1] = 1;

everything WORKS FINE now! :slight_smile:

Best regards,
Rafal

W dniu 2017-10-26 o 17:53, Barbara Jones pisze:

Hi Rafal,

I am able to create a dataset with zero dimensions, close it, and re-open. (See attached).
Can you send us an example of what you are trying that fails?
Please send it to the helpdesk: help@hdfgroup.org

Thanks!
-Barbara
help@hdfgroup.org

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On
Behalf Of Gerd Heber
Sent: Thursday, October 26, 2017 6:49 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF5 library bug or feature?

Rafal, this might be a bug.

Is there any way to record/track this issue?

Not sure. Helpdesk will check. (It's still early morning here...)

G.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
g
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
g
Twitter: https://twitter.com/hdf5


#14

This server is not working properly on Windows 10. I get useful ideas from https://errorcode0x.com/windows-10-preparing-automatic-repair-error-fixed/ page.