I have a file with two external links that at are both at the top level:
HDF5 "DATA.h5" {
GROUP "/" {
EXTERNAL_LINK "Mesh" {
TARGETFILE "Mesh.h5"
TARGETPATH "/Mesh"
GROUP "/Mesh" {
}
}
EXTERNAL_LINK "Step#0" {
TARGETFILE "step_0.h5"
TARGETPATH "/Step#0"
GROUP "/Step#0" {
}
}
}
}
When I open the main file, DATA.h5 as RDONLY I am able to traverse both links as one would expect.
However, when I open the main file, DATA.h5 as RDWR the group "/Mesh" succeeds but the "Step#0" fails. Both links are created in the exact same manner.
if( H5Lcreate_external(MESH_NAME, "/Mesh",
file_id, "Mesh",
H5P_DEFAULT, H5P_DEFAULT) < 0 ) {
}
So I am at a loss as to why the access is different. I tried to follow through and understand more on the Property list that is set to teh default. ie this var:
hid_t lapl_id IN: Link access property list identifier.
but I did not get far. Should I be setting something here???
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
One followup could this problem be related to perhaps already having the one link open ??? and then trying to open it again???
Cheers,
Allen
···
On Dec 18, 2009, at 6:02 PM, Allen Sanderson wrote:
I have a file with two external links that at are both at the top level:
HDF5 "DATA.h5" {
GROUP "/" {
EXTERNAL_LINK "Mesh" {
TARGETFILE "Mesh.h5"
TARGETPATH "/Mesh"
GROUP "/Mesh" {
}
}
EXTERNAL_LINK "Step#0" {
TARGETFILE "step_0.h5"
TARGETPATH "/Step#0"
GROUP "/Step#0" {
}
}
}
}
When I open the main file, DATA.h5 as RDONLY I am able to traverse both links as one would expect.
However, when I open the main file, DATA.h5 as RDWR the group "/Mesh" succeeds but the "Step#0" fails. Both links are created in the exact same manner.
if( H5Lcreate_external(MESH_NAME, "/Mesh",
file_id, "Mesh",
H5P_DEFAULT, H5P_DEFAULT) < 0 ) {
}
So I am at a loss as to why the access is different. I tried to follow through and understand more on the Property list that is set to teh default. ie this var:
hid_t lapl_id IN: Link access property list identifier.
but I did not get far. Should I be setting something here???
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Allen,
One followup could this problem be related to perhaps already having the one link open ??? and then trying to open it again???
Hmm, are you using the latest release (1.8.4)?
Quincey
···
On Dec 18, 2009, at 7:07 PM, Allen Sanderson wrote:
Cheers,
Allen
On Dec 18, 2009, at 6:02 PM, Allen Sanderson wrote:
I have a file with two external links that at are both at the top level:
HDF5 "DATA.h5" {
GROUP "/" {
EXTERNAL_LINK "Mesh" {
TARGETFILE "Mesh.h5"
TARGETPATH "/Mesh"
GROUP "/Mesh" {
}
}
EXTERNAL_LINK "Step#0" {
TARGETFILE "step_0.h5"
TARGETPATH "/Step#0"
GROUP "/Step#0" {
}
}
}
}
When I open the main file, DATA.h5 as RDONLY I am able to traverse both links as one would expect.
However, when I open the main file, DATA.h5 as RDWR the group "/Mesh" succeeds but the "Step#0" fails. Both links are created in the exact same manner.
if( H5Lcreate_external(MESH_NAME, "/Mesh",
file_id, "Mesh",
H5P_DEFAULT, H5P_DEFAULT) < 0 ) {
}
So I am at a loss as to why the access is different. I tried to follow through and understand more on the Property list that is set to teh default. ie this var:
hid_t lapl_id IN: Link access property list identifier.
but I did not get far. Should I be setting something here???
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Quincey,
I am currently using 1.8.2.
One other clue, this only happens when I utilize one particular piece of code (a third party lib). If I skip it then the link is followed fine later (I access it in multiple places). So I think the link and the permissions are okay but how this third party code opens the link initially tends to make it inaccessible later on. So my guess is that this particular piece of code pooches something. I have not started looking at it yet. (Probably will be able to as my hard drive on that machine died Friday night).
Some random thoughts if the file was initially opened as RDONLY with that link accessed. If opening the file again as RDWR would that force the access to that link to fail? If there a way to see what groups are currently open?
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
···
On Dec 18, 2009, at 10:22 PM, Quincey Koziol wrote:
Hi Allen,
On Dec 18, 2009, at 7:07 PM, Allen Sanderson wrote:
One followup could this problem be related to perhaps already having the one link open ??? and then trying to open it again???
Hmm, are you using the latest release (1.8.4)?
Quincey
Cheers,
Allen
On Dec 18, 2009, at 6:02 PM, Allen Sanderson wrote:
I have a file with two external links that at are both at the top level:
HDF5 "DATA.h5" {
GROUP "/" {
EXTERNAL_LINK "Mesh" {
TARGETFILE "Mesh.h5"
TARGETPATH "/Mesh"
GROUP "/Mesh" {
}
}
EXTERNAL_LINK "Step#0" {
TARGETFILE "step_0.h5"
TARGETPATH "/Step#0"
GROUP "/Step#0" {
}
}
}
}
When I open the main file, DATA.h5 as RDONLY I am able to traverse both links as one would expect.
However, when I open the main file, DATA.h5 as RDWR the group "/Mesh" succeeds but the "Step#0" fails. Both links are created in the exact same manner.
if( H5Lcreate_external(MESH_NAME, "/Mesh",
file_id, "Mesh",
H5P_DEFAULT, H5P_DEFAULT) < 0 ) {
}
So I am at a loss as to why the access is different. I tried to follow through and understand more on the Property list that is set to teh default. ie this var:
hid_t lapl_id IN: Link access property list identifier.
but I did not get far. Should I be setting something here???
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Allen,
Hi Quincey,
I am currently using 1.8.2.
One other clue, this only happens when I utilize one particular piece of code (a third party lib). If I skip it then the link is followed fine later (I access it in multiple places). So I think the link and the permissions are okay but how this third party code opens the link initially tends to make it inaccessible later on. So my guess is that this particular piece of code pooches something. I have not started looking at it yet. (Probably will be able to as my hard drive on that machine died Friday night).
Some random thoughts if the file was initially opened as RDONLY with that link accessed. If opening the file again as RDWR would that force the access to that link to fail?
Yes, that would cause the second access to fail.
If there a way to see what groups are currently open?
You can use the H5Fget_obj_count/H5Fget_obj_ids routines to investigate this.
Quincey
···
On Dec 19, 2009, at 1:33 PM, allen wrote:
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
On Dec 18, 2009, at 10:22 PM, Quincey Koziol wrote:
Hi Allen,
On Dec 18, 2009, at 7:07 PM, Allen Sanderson wrote:
One followup could this problem be related to perhaps already having the one link open ??? and then trying to open it again???
Hmm, are you using the latest release (1.8.4)?
Quincey
Cheers,
Allen
On Dec 18, 2009, at 6:02 PM, Allen Sanderson wrote:
I have a file with two external links that at are both at the top level:
HDF5 "DATA.h5" {
GROUP "/" {
EXTERNAL_LINK "Mesh" {
TARGETFILE "Mesh.h5"
TARGETPATH "/Mesh"
GROUP "/Mesh" {
}
}
EXTERNAL_LINK "Step#0" {
TARGETFILE "step_0.h5"
TARGETPATH "/Step#0"
GROUP "/Step#0" {
}
}
}
}
When I open the main file, DATA.h5 as RDONLY I am able to traverse both links as one would expect.
However, when I open the main file, DATA.h5 as RDWR the group "/Mesh" succeeds but the "Step#0" fails. Both links are created in the exact same manner.
if( H5Lcreate_external(MESH_NAME, "/Mesh",
file_id, "Mesh",
H5P_DEFAULT, H5P_DEFAULT) < 0 ) {
}
So I am at a loss as to why the access is different. I tried to follow through and understand more on the Property list that is set to teh default. ie this var:
hid_t lapl_id IN: Link access property list identifier.
but I did not get far. Should I be setting something here???
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Thanks Quincy,
Over the holidays I did some checking and I think it is a case of too many objects opened and not properly closed that is leading to the failure. The HDF5 calls were part of a larger API which I decided to scrap because trying to follow it was too difficult and use a simpler API. The simpler API worked as expected so I was able to make everything work for writing - unfortunately the API needs to be used for reading. And has some other issues with the external links. But I think I know what needs to fixed there.
Cheers,
Allen
···
On Jan 4, 2010, at 7:40 AM, Quincey Koziol wrote:
Hi Allen,
On Dec 19, 2009, at 1:33 PM, allen wrote:
Hi Quincey,
I am currently using 1.8.2.
One other clue, this only happens when I utilize one particular piece of code (a third party lib). If I skip it then the link is followed fine later (I access it in multiple places). So I think the link and the permissions are okay but how this third party code opens the link initially tends to make it inaccessible later on. So my guess is that this particular piece of code pooches something. I have not started looking at it yet. (Probably will be able to as my hard drive on that machine died Friday night).
Some random thoughts if the file was initially opened as RDONLY with that link accessed. If opening the file again as RDWR would that force the access to that link to fail?
Yes, that would cause the second access to fail.
If there a way to see what groups are currently open?
You can use the H5Fget_obj_count/H5Fget_obj_ids routines to investigate this.
Quincey
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
On Dec 18, 2009, at 10:22 PM, Quincey Koziol wrote:
Hi Allen,
On Dec 18, 2009, at 7:07 PM, Allen Sanderson wrote:
One followup could this problem be related to perhaps already having the one link open ??? and then trying to open it again???
Hmm, are you using the latest release (1.8.4)?
Quincey
Cheers,
Allen
On Dec 18, 2009, at 6:02 PM, Allen Sanderson wrote:
I have a file with two external links that at are both at the top level:
HDF5 "DATA.h5" {
GROUP "/" {
EXTERNAL_LINK "Mesh" {
TARGETFILE "Mesh.h5"
TARGETPATH "/Mesh"
GROUP "/Mesh" {
}
}
EXTERNAL_LINK "Step#0" {
TARGETFILE "step_0.h5"
TARGETPATH "/Step#0"
GROUP "/Step#0" {
}
}
}
}
When I open the main file, DATA.h5 as RDONLY I am able to traverse both links as one would expect.
However, when I open the main file, DATA.h5 as RDWR the group "/Mesh" succeeds but the "Step#0" fails. Both links are created in the exact same manner.
if( H5Lcreate_external(MESH_NAME, "/Mesh",
file_id, "Mesh",
H5P_DEFAULT, H5P_DEFAULT) < 0 ) {
}
So I am at a loss as to why the access is different. I tried to follow through and understand more on the Property list that is set to teh default. ie this var:
hid_t lapl_id IN: Link access property list identifier.
but I did not get far. Should I be setting something here???
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
HI,
This morning I was able to track down a bug with a file with external links where the file was opened twice simultaneously. The first open was read only. The second open was read write. However, if I create the file with the exact same hierarchy but without links I do not see the bug. I found this odd.
Here is the sequence
fp1 = open file read only
gp1 = open /group using fp1 (note /group is an eternal link)
fp2 = open file read/write
dp2 = open /group/dataset using fp2
the last open fails. I can understand the failure given the gp1 is read only and dp2 which is under it is trying to perhaps do a write which could cause some problems. So it fails.
However, if the file has the same hierarchy but without the link it does not fail. I found this behavior strange as I would have expected both to behave the same.
Obviously, one solution is not to open the file twice which is certainly possible but also having both opens be read/write also works. But I am more concerned with the inconsistency.
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
Allen,
Thank you for your report!
HI,
This morning I was able to track down a bug with a file with external links where the file was opened twice simultaneously. The first open was read only. The second open was read write. However, if I create the file with the exact same hierarchy but without links I do not see the bug. I found this odd.
Here is the sequence
fp1 = open file read only
gp1 = open /group using fp1 (note /group is an eternal link)
fp2 = open file read/write
dp2 = open /group/dataset using fp2
the last open fails. I can understand the failure given the gp1 is read only and dp2 which is under it is trying to perhaps do a write which could cause some problems. So it fails.
However, if the file has the same hierarchy but without the link it does not fail. I found this behavior strange as I would have expected both to behave the same.
Obviously, one solution is not to open the file twice which is certainly possible but also having both opens be read/write also works. But I am more concerned with the inconsistency.
yes, good point. Could you please send us an error stack from the failed program?
In our defense 
RM entry for H5Fopen states "A file can often be opened with a new H5Fopen call without closing an already-open identifier established in a previous H5Fopen or H5Fcreate call. Each such H5Fopen call will return a unique identifier and the file can be accessed through any of these identifiers as long as the identifier remains valid. In such multiply-opened cases, all the open calls should use the same flags argument."
Elena
···
On Jan 14, 2010, at 12:37 PM, Allen Sanderson wrote:
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hello Elena,
Not a problem to send the stack info. Rarely do I gather this info so could you point me to some sample code that will give you the info you need.
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
···
On Jan 14, 2010, at 1:13 PM, Elena Pourmal wrote:
Allen,
Thank you for your report!
On Jan 14, 2010, at 12:37 PM, Allen Sanderson wrote:
HI,
This morning I was able to track down a bug with a file with external links where the file was opened twice simultaneously. The first open was read only. The second open was read write. However, if I create the file with the exact same hierarchy but without links I do not see the bug. I found this odd.
Here is the sequence
fp1 = open file read only
gp1 = open /group using fp1 (note /group is an eternal link)
fp2 = open file read/write
dp2 = open /group/dataset using fp2
the last open fails. I can understand the failure given the gp1 is read only and dp2 which is under it is trying to perhaps do a write which could cause some problems. So it fails.
However, if the file has the same hierarchy but without the link it does not fail. I found this behavior strange as I would have expected both to behave the same.
Obviously, one solution is not to open the file twice which is certainly possible but also having both opens be read/write also works. But I am more concerned with the inconsistency.
yes, good point. Could you please send us an error stack from the failed program?
In our defense RM entry for H5Fopen states "A file can often be opened with a new H5Fopen call without closing an already-open identifier established in a previous H5Fopen or H5Fcreate call. Each such H5Fopen call will return a unique identifier and the file can be accessed through any of these identifiers as long as the identifier remains valid. In such multiply-opened cases, all the open calls should use the same flags argument."
Elena
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Allen,
You said that your program failed. Did it print out errors from the HDF5 library? This what we would like to see to assess the problem (a bug or a feature 
Having source code of the failing program would be even more helpful.
Thank you!
Elena
···
On Jan 14, 2010, at 3:39 PM, Allen Sanderson wrote:
Hello Elena,
Not a problem to send the stack info. Rarely do I gather this info so could you point me to some sample code that will give you the info you need.
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
On Jan 14, 2010, at 1:13 PM, Elena Pourmal wrote:
Allen,
Thank you for your report!
On Jan 14, 2010, at 12:37 PM, Allen Sanderson wrote:
HI,
This morning I was able to track down a bug with a file with external links where the file was opened twice simultaneously. The first open was read only. The second open was read write. However, if I create the file with the exact same hierarchy but without links I do not see the bug. I found this odd.
Here is the sequence
fp1 = open file read only
gp1 = open /group using fp1 (note /group is an eternal link)
fp2 = open file read/write
dp2 = open /group/dataset using fp2
the last open fails. I can understand the failure given the gp1 is read only and dp2 which is under it is trying to perhaps do a write which could cause some problems. So it fails.
However, if the file has the same hierarchy but without the link it does not fail. I found this behavior strange as I would have expected both to behave the same.
Obviously, one solution is not to open the file twice which is certainly possible but also having both opens be read/write also works. But I am more concerned with the inconsistency.
yes, good point. Could you please send us an error stack from the failed program?
In our defense RM entry for H5Fopen states "A file can often be opened with a new H5Fopen call without closing an already-open identifier established in a previous H5Fopen or H5Fcreate call. Each such H5Fopen call will return a unique identifier and the file can be accessed through any of these identifiers as long as the identifier remains valid. In such multiply-opened cases, all the open calls should use the same flags argument."
Elena
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Elena,
HDF5 did not dump out any errors, the call to H5Dopen returned a negative pointer. That was all. Let me see if I can put together some sample code that will make this failure happen.
Cheers,
Allen
···
On Jan 15, 2010, at 6:47 PM, Elena Pourmal wrote:
Allen,
You said that your program failed. Did it print out errors from the HDF5 library? This what we would like to see to assess the problem (a bug or a feature
Having source code of the failing program would be even more helpful.
Thank you!
Elena
On Jan 14, 2010, at 3:39 PM, Allen Sanderson wrote:
Hello Elena,
Not a problem to send the stack info. Rarely do I gather this info so could you point me to some sample code that will give you the info you need.
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
On Jan 14, 2010, at 1:13 PM, Elena Pourmal wrote:
Allen,
Thank you for your report!
On Jan 14, 2010, at 12:37 PM, Allen Sanderson wrote:
HI,
This morning I was able to track down a bug with a file with external links where the file was opened twice simultaneously. The first open was read only. The second open was read write. However, if I create the file with the exact same hierarchy but without links I do not see the bug. I found this odd.
Here is the sequence
fp1 = open file read only
gp1 = open /group using fp1 (note /group is an eternal link)
fp2 = open file read/write
dp2 = open /group/dataset using fp2
the last open fails. I can understand the failure given the gp1 is read only and dp2 which is under it is trying to perhaps do a write which could cause some problems. So it fails.
However, if the file has the same hierarchy but without the link it does not fail. I found this behavior strange as I would have expected both to behave the same.
Obviously, one solution is not to open the file twice which is certainly possible but also having both opens be read/write also works. But I am more concerned with the inconsistency.
yes, good point. Could you please send us an error stack from the failed program?
In our defense RM entry for H5Fopen states "A file can often be opened with a new H5Fopen call without closing an already-open identifier established in a previous H5Fopen or H5Fcreate call. Each such H5Fopen call will return a unique identifier and the file can be accessed through any of these identifiers as long as the identifier remains valid. In such multiply-opened cases, all the open calls should use the same flags argument."
Elena
Cheers,
Allen
Allen Sanderson
SCI Institute
University of Utah
www.sci.utah.edu
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
