I was running in to trouble where I was
- opening an hdf5 file with H5Fcreate with H5F_ACC_TRUNC
- writing some data
- closing it (calling flush and and making sure there were no open hdf5
handles)
- and then reopening it with another call to H5Fcreate and H5F_ACC_TRUNC to
write over the file.
This was resulting in corrupt hdf5 files (the various h5 tools would spit
out errors accessing the file). If I don't close and reopen it, the file
looked to be ok.
Is this something expected? Or a result of misuse of some kind? Thanks.
I got this with various hdf5 versions in the 1.8 series.
···
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/reopening-for-writing-after-close-tp3942092p3942092.html
Sent from the hdf-forum mailing list archive at Nabble.com.
Hi!
Can you create a small test program that shows this behavior and send it to: help@hdfgroup.org? We can see what's going on then...
Quincey
···
On Apr 26, 2012, at 1:11 PM, drussel@gmail.com wrote:
I was running in to trouble where I was
- opening an hdf5 file with H5Fcreate with H5F_ACC_TRUNC
- writing some data
- closing it (calling flush and and making sure there were no open hdf5
handles)
- and then reopening it with another call to H5Fcreate and H5F_ACC_TRUNC to
write over the file.
This was resulting in corrupt hdf5 files (the various h5 tools would spit
out errors accessing the file). If I don't close and reopen it, the file
looked to be ok.
Is this something expected? Or a result of misuse of some kind? Thanks.
I got this with various hdf5 versions in the 1.8 series.
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/reopening-for-writing-after-close-tp3942092p3942092.html
Sent from the hdf-forum mailing list archive at Nabble.com.
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Having investigated further, the problem is not actually connected to
reopening of HDF5 files, and occurs, intermittently, even when I don't
reopen the file and instead just re-write the entries in the data sets. I
have made a build of HDF5 with its internal memory management turned off (on
a different machine), and run the whole thing through with valgrind.
Valgrind reports no errors (nor does hdf5), but the resulting files are
(sometimes) reported as having errors when accessed afterwards either by our
code or tools like h5dump.
Any suggestions as to what to try next? We have numerous various simple
tests of our code that don't exhibit the problem and, so far, my attempts to
produce a simpler example have not yielding something useful. Thanks very
much.
···
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/reopening-for-writing-after-close-tp3942092p3971857.html
Sent from the hdf-forum mailing list archive at Nabble.com.
Hi Drussel,
what filesystem are you writing to? We had bad case of non-reproducible
metadata corruption when writing HDF5 to older version of Lustre.
Cheers,
Andy.
···
drussel@gmail.com wrote on 2012-05-08:
Having investigated further, the problem is not actually connected to
reopening of HDF5 files, and occurs, intermittently, even when I don't
reopen the file and instead just re-write the entries in the data sets.
I have made a build of HDF5 with its internal memory management turned
off (on a different machine), and run the whole thing through with
valgrind. Valgrind reports no errors (nor does hdf5), but the resulting
files are (sometimes) reported as having errors when accessed afterwards
either by our code or tools like h5dump.
Any suggestions as to what to try next? We have numerous various simple
tests of our code that don't exhibit the problem and, so far, my
attempts to produce a simpler example have not yielding something
useful. Thanks very much.
ext4 and hfs+, nothing too exotic, I'm afraid.
···
On Wed, May 9, 2012 at 10:29 PM, Salnikov, Andrei A. < salnikov@slac.stanford.edu> wrote:
Hi Drussel,
what filesystem are you writing to? We had bad case of non-reproducible
metadata corruption when writing HDF5 to older version of Lustre.
Cheers,
Andy.
drussel@gmail.com wrote on 2012-05-08:
> Having investigated further, the problem is not actually connected to
> reopening of HDF5 files, and occurs, intermittently, even when I don't
> reopen the file and instead just re-write the entries in the data sets.
> I have made a build of HDF5 with its internal memory management turned
> off (on a different machine), and run the whole thing through with
> valgrind. Valgrind reports no errors (nor does hdf5), but the resulting
> files are (sometimes) reported as having errors when accessed afterwards
> either by our code or tools like h5dump.
>
> Any suggestions as to what to try next? We have numerous various simple
> tests of our code that don't exhibit the problem and, so far, my
> attempts to produce a simpler example have not yielding something
> useful. Thanks very much.
>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
Hi Daniel,
Hmm, can you pin this down to a small sample program that you could send us?
Quincey
···
On May 10, 2012, at 11:30 AM, Daniel Russel wrote:
ext4 and hfs+, nothing too exotic, I'm afraid.
On Wed, May 9, 2012 at 10:29 PM, Salnikov, Andrei A. <salnikov@slac.stanford.edu> wrote:
Hi Drussel,
what filesystem are you writing to? We had bad case of non-reproducible
metadata corruption when writing HDF5 to older version of Lustre.
Cheers,
Andy.
drussel@gmail.com wrote on 2012-05-08:
> Having investigated further, the problem is not actually connected to
> reopening of HDF5 files, and occurs, intermittently, even when I don't
> reopen the file and instead just re-write the entries in the data sets.
> I have made a build of HDF5 with its internal memory management turned
> off (on a different machine), and run the whole thing through with
> valgrind. Valgrind reports no errors (nor does hdf5), but the resulting
> files are (sometimes) reported as having errors when accessed afterwards
> either by our code or tools like h5dump.
>
> Any suggestions as to what to try next? We have numerous various simple
> tests of our code that don't exhibit the problem and, so far, my
> attempts to produce a simpler example have not yielding something
> useful. Thanks very much.
>
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
No luck reproducing it in something simple so far. The code has ~15 data
sets that are written to a mixed manner through a wrapper and I haven't
managed to get bad behavior (other than something I'll post in a separate
email) with direct usage.
···
On Tue, May 15, 2012 at 11:40 AM, Quincey Koziol [via hdf-forum] < ml-node+s184993n3988636h75@n3.nabble.com> wrote:
Hi Daniel,
Hmm, can you pin this down to a small sample program that you could send
us?
Quincey
On May 10, 2012, at 11:30 AM, Daniel Russel wrote:
ext4 and hfs+, nothing too exotic, I'm afraid.
On Wed, May 9, 2012 at 10:29 PM, Salnikov, Andrei A. <[hidden email]<http://user/SendEmail.jtp?type=node&node=3988636&i=0> > > wrote:
Hi Drussel,
what filesystem are you writing to? We had bad case of non-reproducible
metadata corruption when writing HDF5 to older version of Lustre.
Cheers,
Andy.
[hidden email] <http://user/SendEmail.jtp?type=node&node=3988636&i=1>wrote on 2012-05-08:
> Having investigated further, the problem is not actually connected to
> reopening of HDF5 files, and occurs, intermittently, even when I don't
> reopen the file and instead just re-write the entries in the data sets.
> I have made a build of HDF5 with its internal memory management turned
> off (on a different machine), and run the whole thing through with
> valgrind. Valgrind reports no errors (nor does hdf5), but the resulting
> files are (sometimes) reported as having errors when accessed afterwards
> either by our code or tools like h5dump.
>
> Any suggestions as to what to try next? We have numerous various simple
> tests of our code that don't exhibit the problem and, so far, my
> attempts to produce a simpler example have not yielding something
> useful. Thanks very much.
>
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email] <http://user/SendEmail.jtp?type=node&node=3988636&i=2>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email] <http://user/SendEmail.jtp?type=node&node=3988636&i=3>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
[hidden email] <http://user/SendEmail.jtp?type=node&node=3988636&i=4>
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
------------------------------
If you reply to this email, your message will be added to the discussion
below:
http://hdf-forum.184993.n3.nabble.com/reopening-for-writing-after-close-tp3942092p3988636.html
To unsubscribe from reopening for writing after close, click here<http://hdf-forum.184993.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=3942092&code=ZHJ1c3NlbEBnbWFpbC5jb218Mzk0MjA5Mnw2MDMzODI3ODE=>
.
NAML<http://hdf-forum.184993.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml>
--
View this message in context: http://hdf-forum.184993.n3.nabble.com/reopening-for-writing-after-close-tp3942092p3998102.html
Sent from the hdf-forum mailing list archive at Nabble.com.
