Thank you Dan, I think you are right on target,
The HDF read side is good; however it is combined with:
1) writes to the console;
2) complex and very slow if statements
3) and writes to the geodatabase.
I suggest you do the same read from the HDF file,
storing the data in an array, and in a second
loop, writing the data out to the geodatabase.
In the past I have improved this operation 100X
using this technique. Stop the debugging statements.
I have been working extensively with the HE5 data
and think that the complex 'if' logic could be removed
in your solution. Most of this data is already
within the 180-180 and 90-90 scope.
John Huddleston
···
______
__________________________________
From: Daniel Kahn [mailto:daniel_kahn@ssaihq.com]
Sent: Wednesday, March 25, 2009 4:07 PM
To: Becker, Thomas
Cc: jhuddleston@hughes.net
Subject: Re: [hdf-forum] creating polygons from corner points
Thomas,
I've worked with OMI data and I think that the HDF5 portion of your code
will be very fast compared to the rest of it. In other words, I don't think
the problem is related to HDF5 and so I won't include my response in the
hdf-forum group, but when you find a solution it might be nice for you to
post it there. (I'll include John H. since he did respond to you. John, if
you don't want to be involved in any further discussion let us know.)
I suspect a lot of the problems are in the inner for loop, not just because
of the writes but because for loops can be very slow in interpreted
languages. Are you familiar with array based languages like IDL or matlab?
Numpy supports array operations in a manner very similar to Matlab. You
should be able to create logical arrays which eliminate the conditionals
inside of your loop. You won't get rid of the loop, but it should be
faster.
Also, OMI footprints are 4 sided. If one of the corner points fails a test
the polygon will still be created, but it will not have four sides. Is this
what you want? Your code duplicates the first and last point. If that
point fails the test you'll get a three line segments, not a polygon. I
don't use ArgGIS, but I notice it has a polygon object. Shouldn't you be
using that instead of your own 5 point arrays.
Cheers,
--dan
Becker, Thomas wrote:
Good morning John,
I am not sure if I really understand the way you propose.
Of course I can write all the coordinates and other values into one huge
list or array. But creating the polygons from the corner points is as far as
I understand always done in the way that you have to create an array holding
the coordinates of the four corner points and the first point again to
indicate that it is a closed polygon and not a polyline.
To do so, in all the ESRI examples I found they create a cursor in the
featureclass to iterate thrue the rows. Then the polygon is created by
passing the array of the corner points to the row and insert the row into
the featureclass.
The array object has to be deleted afterwards to ensure a correct geometry
of the following polygon.
I do not understand how do you write all the points at once to the
geodatabase.
Thomas
-----Original Message-----
From: John Huddleston [mailto:jhuddleston@hughes.net]
Sent: Wednesday, March 25, 2009 12:28 PM
To: Becker, Thomas; hdf-forum@hdfgroup.org
Subject: RE: [hdf-forum] creating polygons from corner points
Good morning Thomas
In general, with any code, read and write operations in a do loop are going
to be slower than one single write.
In code you could use a two pass operation. On the first pass find the total
number of records to write, set up the array and read in the data, and then
perform one write to the geodatabase.
John Huddleston
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to
hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
--
Daniel Kahn
Science Systems and Applications Inc.
301-867-2162
----------------------------------------------------------------------
This mailing list is for HDF software users discussion.
To subscribe to this list, send a message to hdf-forum-subscribe@hdfgroup.org.
To unsubscribe, send a message to hdf-forum-unsubscribe@hdfgroup.org.
