Hi all. I'm collecting signals from some electronics instrumentation and
trying to store them in an HDF5 file. Without knowing how long my
collection is going to go in advance, it seems like my best option is to
expand the dataset as I need. But it seems like doing this with a compound
data type of more than about 10 doubles leads to a crash. The crash looks
like this:
C:\Users\awaltman\Documents\Visual Studio
2010\Projects\ConsoleApplication3\Cons
oleApplication3\bin\Debug>ConsoleApplication3.exe
Unhandled Exception: System.AccessViolationException: Attempted to read or
write protected memory. This is often an indication that other memory is
corrupt.
at H5Dwrite(Int32 dataSetId, Int32 memType, Int32 memSpace, Int32
fileSpace,
Int32 xfer, Void* data)
at HDF5DotNet.H5D.write[Type](H5DataSetId dataSetId, H5DataTypeId
memTypeId, H5DataSpaceId memSpaceId, H5DataSpaceId fileSpaceId,
H5PropertyListId xferPropLi stId, H5Array`1 data)
at HDF5DotNet.H5D.write[Type](H5DataSetId dataSetId, H5DataTypeId
memTypeId,
H5Array`1 data)
at ConsoleApplication3.Program.Main(String[] args) in
C:\Users\awaltman\docum ents\visual studio
2010\Projects\ConsoleApplication3\ConsoleApplication3\Program
.cs:line 62
The attached version tries to write 15000 columns, expanding the data set
one at a time for each write, with 32 rows (double values) in each. If you
drop it down to rows = 8, it runs successfully and I can check out the
results in an HDF viewer.
If there's another way I should be doing this, like tables, let me know.
This seems to be what I have available through the HDF5DotNet API though.
Aaron, how are you? You made a (incorrect) dataspace selection, but never
used it in the H5Dwrite call.
You're writing one record at a time(*), that means your selections in memory
and the file must be
singletons. Finally, you have to tell H5Dwrite about those selections.
Attached is a fixed version of your program. I've tested it with HDF5DotNet
1.8.7, .NET4 64-bit.
-----Original Message-----
From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-
bounces@hdfgroup.org] On Behalf Of Aaron Altman
Sent: Thursday, September 01, 2011 5:13 PM
To: hdf-forum@hdfgroup.org
Subject: [Hdf-forum] HDF5DotNet memory leak when expanding with
HD5.setExtent?
Hi all. I'm collecting signals from some electronics instrumentation and
trying to store them in an HDF5 file. Without knowing how long my
collection is going to go in advance, it seems like my best option is to
expand the dataset as I need. But it seems like doing this with a
compound data type of more than about 10 doubles leads to a crash. The
crash looks like this:
C:\Users\awaltman\Documents\Visual Studio
2010\Projects\ConsoleApplication3\Cons
oleApplication3\bin\Debug>ConsoleApplication3.exe
Unhandled Exception: System.AccessViolationException: Attempted to read or
write protected memory. This is often an indication that other memory is
corrupt.
at H5Dwrite(Int32 dataSetId, Int32 memType, Int32 memSpace, Int32
fileSpace,
Int32 xfer, Void* data)
at HDF5DotNet.H5D.write[Type](H5DataSetId dataSetId, H5DataTypeId
memTypeId, H5DataSpaceId memSpaceId, H5DataSpaceId fileSpaceId,
H5PropertyListId xferPropLi stId, H5Array`1 data)
at HDF5DotNet.H5D.write[Type](H5DataSetId dataSetId, H5DataTypeId
memTypeId,
H5Array`1 data)
at ConsoleApplication3.Program.Main(String[] args) in
C:\Users\awaltman\docum ents\visual studio
2010\Projects\ConsoleApplication3\ConsoleApplication3\Program
.cs:line 62
The attached version tries to write 15000 columns, expanding the data set
one at a time for each write, with 32 rows (double values) in each. If
you drop it down to rows = 8, it runs successfully and I can check out the
results in an HDF viewer.
If there's another way I should be doing this, like tables, let me know.
This seems to be what I have available through the HDF5DotNet API though.
Thanks,
Aaron Altman
--
View this message in context: http://hdf-
forum.184993.n3.nabble.com/HDF5DotNet-memory-leak-when-expanding-with-HD5-
setExtent-tp3302724p3302724.html
Sent from the hdf-forum mailing list archive at Nabble.com.
That was an extremely helpful response. I integrated your changes back in
with my code and it worked.
As for performance and efficiency, one line at a time will probably work
well enough. The file is write once/read many so it only needs to be able
to keep up with the max sampling rate our data acquisition device can report
(probably about 10klines/sec). I'd still like to do it right though, just
not sure how to do it in C#. H5PT doesn't seem to be in HDF5DotNet. Does
H5T expose what I need?
You're welcome. Yes, I'm sorry, I forgot to mention that H5PT is currently
not funded/supported in HDF5DotNet. You still might want to check it out.
What if that 10klines/sec turns into 500k/lines in two years?
And, if you really like it, maybe you want to coerce us with a small
donation
into supporting it in HDF5DotNet
That was an extremely helpful response. I integrated your changes back in
with my code and it worked.
As for performance and efficiency, one line at a time will probably work
well enough. The file is write once/read many so it only needs to be able
to keep up with the max sampling rate our data acquisition device can
report (probably about 10klines/sec). I'd still like to do it right
though, just not sure how to do it in C#. H5PT doesn't seem to be in
HDF5DotNet. Does H5T expose what I need?
