memory issues when reading multiple datasets

Donald_Brandon · September 19, 2011, 7:21pm

Hello all:

I have another problem that I could use some help with. In a nutshell, I have multiple datasets that I am trying to read arrays from and then work with those arrays one by one. The structure of the .h5 file kinda look like this:

*Group<root>
         Group<Group1>
                 Group<GroupA>
                         Group<Group_a>
                                 Dataset<1>
                         Group<group_b>
                                 Dataset<1>
                 Group<GroupB>
                         Group<Group_a>
                                 Dataset<1>
                         Group<group_b>
                                 Dataset<1>
         Group<Group2>
                         Group<Group_a>
                                 Dataset<1>
                         Group<group_b>
                                 Dataset<1>
                 Group<GroupB>
                         Group<Group_a>
                                 Dataset<1>
                         Group<group_b>
                                 Dataset<1>*

I am recursively going through each group and then executing code that looks something like this:

* public float[] GetDepthArray(H5DataSetId dataset)
         {
             H5DataTypeId datasetType = H5D.getType(dataset);
             H5T.H5TClass datasetClass = H5T.getClass(datasetType);
             H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType, H5T.Direction.ASCEND);

long storageSZ = H5D.getStorageSize(dataset);

float[] dArray = new float[storageSZ];

H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));

return dArray;
}*

After about the seventh iteration, the code will fail with an AccessViolationException - "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up after each iteration that I am not seeing? My system RAM seems to be un-phased after each iteration so at least it doesn't look like I am losing memory anywhere...

Thanks for the help!
DB

Mitchell_Scott_AES · September 19, 2011, 8:27pm

Hey Donald,

You need to be close()ing your H5T type objects. I'm not sure if that's it, but....

Scott

···

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Donald Brandon
Sent: Monday, September 19, 2011 3:22 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] memory issues when reading multiple datasets

Hello all:

I have another problem that I could use some help with. In a nutshell, I have multiple datasets that I am trying to read arrays from and then work with those arrays one by one. The structure of the .h5 file kinda look like this:

Group<root>
        Group<Group1>
                Group<GroupA>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
        Group<Group2>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>

I am recursively going through each group and then executing code that looks something like this:

        public float[] GetDepthArray(H5DataSetId dataset)
        {
            H5DataTypeId datasetType = H5D.getType(dataset);
            H5T.H5TClass datasetClass = H5T.getClass(datasetType);
            H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType, H5T.Direction.ASCEND);

long storageSZ = H5D.getStorageSize(dataset);

float[] dArray = new float[storageSZ];

H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));

return dArray;
}

After about the seventh iteration, the code will fail with an AccessViolationException - "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up after each iteration that I am not seeing? My system RAM seems to be un-phased after each iteration so at least it doesn't look like I am losing memory anywhere...

Thanks for the help!
DB

________________________________
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.

gheber · September 19, 2011, 9:48pm

Don, just a comment on sizing. H5D.getStorageSize returns the (allocated)
dataset size

in bytes and NOT the number of elements in your dataset. 'storageSZ' in your
code

is not the number of floats in your dataset. Maybe you're doing stuff

in your code that you haven't shown us that makes incorrect assumptions
about sizes.

Best, G.

···

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Donald Brandon
Sent: Monday, September 19, 2011 2:22 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] memory issues when reading multiple datasets

Hello all:

I have another problem that I could use some help with. In a nutshell, I
have multiple datasets that I am trying to read arrays from and then work
with those arrays one by one. The structure of the .h5 file kinda look like
this:

Group<root>
        Group<Group1>
                Group<GroupA>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
        Group<Group2>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>

I am recursively going through each group and then executing code that looks
something like this:

        public float[] GetDepthArray(H5DataSetId dataset)
        {
            H5DataTypeId datasetType = H5D.getType(dataset);
            H5T.H5TClass datasetClass = H5T.getClass(datasetType);
            H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType,
H5T.Direction.ASCEND);

long storageSZ = H5D.getStorageSize(dataset);

float[] dArray = new float[storageSZ];

H5D.read(dataset, datasetNativeType, new
H5Array<float>(dArray));

return dArray;
}

After about the seventh iteration, the code will fail with an
AccessViolationException - "Attempted to read or write protected memory.
This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up
after each iteration that I am not seeing? My system RAM seems to be
un-phased after each iteration so at least it doesn't look like I am losing
memory anywhere...

Thanks for the help!
DB

Donald_Brandon · September 19, 2011, 8:31pm

Hey, Scott. Yea, I caught that after I sent the e-mail. I am writing a test program now. I will let you know if that solved it.

Thanks,
DB

···

On 9/19/2011 3:27 PM, Mitchell, Scott - IS wrote:

Hey Donald,

You need to be close()ing your H5T type objects. I'm not sure if that's it, but....

Scott

*From:*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Donald Brandon
*Sent:* Monday, September 19, 2011 3:22 PM
*To:* HDF Users Discussion List
*Subject:* [Hdf-forum] memory issues when reading multiple datasets

Hello all:

I have another problem that I could use some help with. In a nutshell, I have multiple datasets that I am trying to read arrays from and then work with those arrays one by one. The structure of the .h5 file kinda look like this:

*Group<root>
        Group<Group1>
                Group<GroupA>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
        Group<Group2>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>*

I am recursively going through each group and then executing code that looks something like this:

* public float[] GetDepthArray(H5DataSetId dataset)
        {
            H5DataTypeId datasetType = H5D.getType(dataset);
            H5T.H5TClass datasetClass = H5T.getClass(datasetType);
            H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType, H5T.Direction.ASCEND);

            long storageSZ = H5D.getStorageSize(dataset);

            float[] dArray = new float[storageSZ];

            H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));

            return dArray;
        }*

After about the seventh iteration, the code will fail with an AccessViolationException - "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up after each iteration that I am not seeing? My system RAM seems to be un-phased after each iteration so at least it doesn't look like I am losing memory anywhere...

Thanks for the help!
DB

------------------------------------------------------------------------
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Donald_Brandon · September 19, 2011, 9:22pm

Yea, adding the appropriate close methods did not make a difference. Here is the loop I am running:

*foreach(String curGroup3Name in curGroup3Names)
{
H5FileId hdfFile = H5F.open(pathToHDF, H5F.OpenMode.ACC_RDONLY);
H5DataSetId curDataset = H5D.open(hdfFile, curGroup1Name + "/" + curGroup2Name + "/" + curGroup3Name + "/dataset");

float[] curDatasetArray = GetDatasetArray(curDataset);

H5D.close(curDataset);
H5F.close(hdfFile);

MessageBox.Show("Finished dataset for tile " + curTileName + "...");
}*

Just as in my other code, I get to about the seventh iteration and i hangs on the call to:

*H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));*

...that happens inside the method getDatasetArray(curDataset). That code is listed below...

Any ideas anyone?

Thanks again,
DB

···

On 9/19/2011 3:31 PM, Donald Brandon wrote:

Hey, Scott. Yea, I caught that after I sent the e-mail. I am writing a test program now. I will let you know if that solved it.

Thanks,
DB

On 9/19/2011 3:27 PM, Mitchell, Scott - IS wrote:

Hey Donald,

You need to be close()ing your H5T type objects. I'm not sure if that's it, but....

Scott

*From:*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Donald Brandon
*Sent:* Monday, September 19, 2011 3:22 PM
*To:* HDF Users Discussion List
*Subject:* [Hdf-forum] memory issues when reading multiple datasets

Hello all:

I have another problem that I could use some help with. In a nutshell, I have multiple datasets that I am trying to read arrays from and then work with those arrays one by one. The structure of the .h5 file kinda look like this:

*Group<root>
        Group<Group1>
                Group<GroupA>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
        Group<Group2>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>*

I am recursively going through each group and then executing code that looks something like this:

* public float[] GetDepthArray(H5DataSetId dataset)
        {
            H5DataTypeId datasetType = H5D.getType(dataset);
            H5T.H5TClass datasetClass = H5T.getClass(datasetType);
            H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType, H5T.Direction.ASCEND);

            long storageSZ = H5D.getStorageSize(dataset);

            float[] dArray = new float[storageSZ];

            H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));

            return dArray;
        }*

After about the seventh iteration, the code will fail with an AccessViolationException - "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up after each iteration that I am not seeing? My system RAM seems to be un-phased after each iteration so at least it doesn't look like I am losing memory anywhere...

Thanks for the help!
DB

------------------------------------------------------------------------
This e-mail and any files transmitted with it may be proprietary and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error please notify the sender.
Please note that any views or opinions presented in this e-mail are solely those of the author and do not necessarily represent those of ITT Corporation. The recipient should check this e-mail and any attachments for the presence of viruses. ITT accepts no liability for any damage caused by any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Donald_Brandon · September 19, 2011, 10:33pm

Hey G:

Thanks for your input. Again, you comments made me look at things from a different perspective and I noticed that the value of storageSZ was changing with each iteration and I was not expecting that. My final outputs are two dimensional arrays of a fixed size and knowing that I can predetermine the value I need for the array size so I decided to just go that route for now. I ran through my test code again with this change and every array was successfully created.

Matt - Thanks for your input as well. I think I am OK for now.

I just want to say, this forum is one of the best I have joined as far as getting people willing to help with the questions that I pose. Thanks everyone for your help.

Until next time,
DB

···

On 9/19/2011 4:48 PM, Gerd Heber wrote:

Don, just a comment on sizing. H5D.getStorageSize returns the (allocated) dataset size

in bytes and NOT the number of elements in your dataset. 'storageSZ' in your code

is not the number of floats in your dataset. Maybe you're doing stuff

in your code that you haven't shown us that makes incorrect assumptions about sizes.

Best, G.

*From:*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] *On Behalf Of *Donald Brandon
*Sent:* Monday, September 19, 2011 2:22 PM
*To:* HDF Users Discussion List
*Subject:* [Hdf-forum] memory issues when reading multiple datasets

Hello all:

I have another problem that I could use some help with. In a nutshell, I have multiple datasets that I am trying to read arrays from and then work with those arrays one by one. The structure of the .h5 file kinda look like this:

*Group<root>
        Group<Group1>
                Group<GroupA>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
        Group<Group2>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>
                Group<GroupB>
                        Group<Group_a>
                                Dataset<1>
                        Group<group_b>
                                Dataset<1>*

I am recursively going through each group and then executing code that looks something like this:

* public float[] GetDepthArray(H5DataSetId dataset)
        {
            H5DataTypeId datasetType = H5D.getType(dataset);
            H5T.H5TClass datasetClass = H5T.getClass(datasetType);
            H5DataTypeId datasetNativeType = H5T.getNativeType(datasetType, H5T.Direction.ASCEND);

            long storageSZ = H5D.getStorageSize(dataset);

            float[] dArray = new float[storageSZ];

            H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));

            return dArray;
        }*

After about the seventh iteration, the code will fail with an AccessViolationException - "Attempted to read or write protected memory. This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up after each iteration that I am not seeing? My system RAM seems to be un-phased after each iteration so at least it doesn't look like I am losing memory anywhere...

Thanks for the help!
DB

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Matt_Calder · September 19, 2011, 9:49pm

Donald,

We have our own C# wrapper for hdf. We saw the same error you site,
and we resolved it, maybe the cause is the same for you. We use the
.net code on Ubuntu, Windows, and Mac. This error arose only on
Windows and Mac and could be reliably replicated by doing reads from
multiple threads. Multi-threaded reading on Ubuntu worked without
error. Our fix was to use locks. So, we have a class:

         private class ThreadLock
        {
            private ReaderWriterLockSlim threadLock = new
ReaderWriterLockSlim();

            public void EnterReadLock() {threadLock.EnterWriteLock();}
            public void ExitReadLock() {threadLock.ExitWriteLock();}
            public void EnterWriteLock() {threadLock.EnterWriteLock();}
            public void ExitWriteLock() {threadLock.ExitWriteLock();}

}

Notice, that both read and write locking invokes the write lock, which
for ReaderWriterLockSlim is exclusive. On Ubuntu, the read lock code
looks like:

public void EnterReadLock() {threadLock.EnterReadLock();}
public void ExitReadLock() {threadLock.ExitReadLock();}

which permits multi-threaded reading (read locks are not exclusive).

Then, hdf operations are wrapped as:

        public bool StoreSequence(string path, double [] data)
        {
            bool ans;
            threadLock.EnterWriteLock();
            try {
                ans = hdfFile.StoreVector(path, data);
                hdfFile.Flush();
                if (logger != null) {logger.DebugFormat("{0} =
StoreSequence({1}, double[{2}])", ans, path, data.Length);}
            } finally {
                threadLock.ExitWriteLock();
            }
            return ans;
        }

Here hdfFile is an instance of our in-house hdf wrapper. The
StoreVector call makes all the hdf library calls one would expect and
because of the locks, they function correctly.

If you want any clarification, please feel free to ask. If this is not
helpful in your situation, I'd like to hear, as even though it fixed
our problems, it may not have been addressing the root cause (and we'd
like to get multithreaded reads back on Windows and Mac).

Matt

···

On Mon, Sep 19, 2011 at 5:22 PM, Donald Brandon <dbran@cableone.net> wrote:

Yea, adding the appropriate close methods did not make a difference. Here
is the loop I am running:

foreach(String curGroup3Name in curGroup3Names)
{
H5FileId hdfFile = H5F.open(pathToHDF, H5F.OpenMode.ACC_RDONLY);
H5DataSetId curDataset = H5D.open(hdfFile, curGroup1Name + "/" +
curGroup2Name + "/" + curGroup3Name + "/dataset");
float\[\] curDatasetArray = GetDatasetArray\(curDataset\);

H5D\.close\(curDataset\);
H5F\.close\(hdfFile\);

MessageBox\.Show\(&quot;Finished dataset for tile &quot; \+ curTileName \+ &quot;\.\.\.&quot;\);
}

Just as in my other code, I get to about the seventh iteration and i hangs
on the call to:

H5D.read(dataset, datasetNativeType, new H5Array<float>(dArray));

...that happens inside the method getDatasetArray(curDataset). That code is
listed below...

Any ideas anyone?

Thanks again,
DB

On 9/19/2011 3:31 PM, Donald Brandon wrote:

Hey, Scott. Yea, I caught that after I sent the e-mail. I am writing a
test program now. I will let you know if that solved it.

Thanks,
DB

On 9/19/2011 3:27 PM, Mitchell, Scott - IS wrote:

Hey Donald,

You need to be close()ing your H5T type objects. I’m not sure if that’s it,
but….

Scott

From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org]
On Behalf Of Donald Brandon
Sent: Monday, September 19, 2011 3:22 PM
To: HDF Users Discussion List
Subject: [Hdf-forum] memory issues when reading multiple datasets

Hello all:

I have another problem that I could use some help with. In a nutshell, I
have multiple datasets that I am trying to read arrays from and then work
with those arrays one by one. The structure of the .h5 file kinda look like
this:

Group<root>
Group<Group1>
Group<GroupA>
Group<Group_a>
Dataset<1>
Group<group_b>
Dataset<1>
Group<GroupB>
Group<Group_a>
Dataset<1>
Group<group_b>
Dataset<1>
Group<Group2>
Group<Group_a>
Dataset<1>
Group<group_b>
Dataset<1>
Group<GroupB>
Group<Group_a>
Dataset<1>
Group<group_b>
Dataset<1>

I am recursively going through each group and then executing code that looks
something like this:
    public float\[\] GetDepthArray\(H5DataSetId dataset\)
    \{
        H5DataTypeId datasetType = H5D\.getType\(dataset\);
        H5T\.H5TClass datasetClass = H5T\.getClass\(datasetType\);
        H5DataTypeId datasetNativeType = H5T\.getNativeType\(datasetType,
H5T.Direction.ASCEND);
        long storageSZ = H5D\.getStorageSize\(dataset\);

        float\[\] dArray = new float\[storageSZ\];

        H5D\.read\(dataset, datasetNativeType, new
H5Array<float>(dArray));
        return dArray;
    \}
After about the seventh iteration, the code will fail with an
AccessViolationException - "Attempted to read or write protected memory.
This is often an indication that other memory is corrupt."

Anyone have advice as to whether or not I should be cleaning something up
after each iteration that I am not seeing? My system RAM seems to be
un-phased after each iteration so at least it doesn't look like I am losing
memory anywhere...

Thanks for the help!
DB

________________________________
This e-mail and any files transmitted with it may be proprietary and are
intended solely for the use of the individual or entity to whom they are
addressed. If you have received this e-mail in error please notify the
sender.
Please note that any views or opinions presented in this e-mail are solely
those of the author and do not necessarily represent those of ITT
Corporation. The recipient should check this e-mail and any attachments for
the presence of viruses. ITT accepts no liability for any damage caused by
any virus transmitted by this e-mail.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org