Extracting Group Variables from Landsat 7 HDF-EOS

Hi Folks,
I am really struggling to extract certain Group variables from a HDF4 hdf-eos Landsat 7 file.
I am using the netcdf4-python driver [0] in order to achieve this task. I’ve installed the chain of dependencies fine.
The Group Variables which I cannot seem to access are included within the attached .png. E.g. Projection, UpperLeftPointMtrs, LowerRightMtrs and SphereCode.
I can however view them when I read in the HDF4 file as follows

f = Dataset(hdfFile)

print f

<output>

GROUP=GridStructure

GROUP=GRID_1

GridName="Grid"

XDim=8001

YDim=7271

UpperLeftPointMtrs=(310185.000000,4575015.000000)

LowerRightMtrs=(550215.000000,4356885.000000)


</output>

Can someone please advise me as how I can extract the Group Variables as I’ve explained above?
Thank you very much in advance for any assistance.
Best
Lewis

[0] netCDF4 API documentation

Dr. Lewis John McGibbney PhD, B.Sc., MAGU
Engineering Applications Software Engineer Level 2
Computer Science for Data Intensive Systems Group 398M
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel: (+1) (818)-393-7402
Cell: (+1) (626)-487-3476
Fax: (+1) (818)-393-1190
Email: lewis.j.mcgibbney@jpl.nasa.gov

           [cid:0AFCAB22-5194-48A3-9912-0D7CDA76139D]

Dare Mighty Things

Hi, Lewis!

Is screen shot generated from ToolsUI (Netcdf-Java)?

···

--
Save the Earth. Save Earth data in HDF-EOS. Save Big data in HDF.

On Wed, Dec 3, 2014 at 4:40 PM, Mcgibbney, Lewis J (398M) < Lewis.J.Mcgibbney@jpl.nasa.gov> wrote:

Hi Folks,
I am really struggling to extract certain Group variables from a HDF4
hdf-eos Landsat 7 file.
I am using the netcdf4-python driver [0] in order to achieve this task.
I’ve installed the chain of dependencies fine.
The Group Variables which I cannot seem to access are included within the
attached .png. E.g. Projection, UpperLeftPointMtrs, LowerRightMtrs and
SphereCode.
I can however view them when I read in the HDF4 file as follows

f = Dataset(hdfFile)

print f

<output>

GROUP=GridStructure

GROUP=GRID_1

GridName="Grid"

XDim=8001

YDim=7271

UpperLeftPointMtrs=(310185.000000,4575015.000000)

LowerRightMtrs=(550215.000000,4356885.000000)

</output>

Can someone please advise me as how I can extract the Group Variables as
I’ve explained above?
Thank you very much in advance for any assistance.
Best
Lewis

[0] netCDF4 API documentation

  Dr. Lewis John McGibbney PhD, B.Sc., MAGU
Engineering Applications Software Engineer Level 2
Computer Science for Data Intensive Systems Group 398M
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel: (+1) (818)-393-7402
Cell: (+1) (626)-487-3476
Fax: (+1) (818)-393-1190
Email: lewis.j.mcgibbney@jpl.nasa.gov

  Dare Mighty Things

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org

http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi,
No it is generated by Panoply [0], however I can see the variable fields when I print the Dataset object.
I just can’t see how to actually access them cleanly.
I resorted to the following which is horrible, probably inconsistent in it’s operation across various HDF4 from Landsat 7 and just downright nasty

str(f.__dict__['StructMetadata.0']).split()

This above gives me the StructMetadata.0 element of the dataset dictionary as a list. I can then navigate to the index within that list and grab the variable I require e.g. UpperLeftPointMtrs

Do you have any better suggestions. I anticipate that this hack will come back to bite me pretty soon.
Thank you very much for any information.
lewis

[0] NASA GISS: Panoply 5 netCDF, HDF and GRIB Data Viewer

Dr. Lewis John McGibbney PhD, B.Sc., MAGU
Engineering Applications Software Engineer Level 2
Computer Science for Data Intensive Systems Group 398M
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel: (+1) (818)-393-7402
Cell: (+1) (626)-487-3476
Fax: (+1) (818)-393-1190
Email: lewis.j.mcgibbney@jpl.nasa.gov

           [cid:0C8358E3-2D72-49FC-9735-9A71DDE08CED]

Dare Mighty Things

···

From: "H. Joe Lee" <hyoklee@hdfgroup.org<mailto:hyoklee@hdfgroup.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Thursday, December 4, 2014 at 7:09 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] Extracting Group Variables from Landsat 7 HDF-EOS

Hi, Lewis!

Is screen shot generated from ToolsUI (Netcdf-Java)?

--
Save the Earth. Save Earth data in HDF-EOS. Save Big data in HDF.

On Wed, Dec 3, 2014 at 4:40 PM, Mcgibbney, Lewis J (398M) <Lewis.J.Mcgibbney@jpl.nasa.gov<mailto:Lewis.J.Mcgibbney@jpl.nasa.gov>> wrote:
Hi Folks,
I am really struggling to extract certain Group variables from a HDF4 hdf-eos Landsat 7 file.
I am using the netcdf4-python driver [0] in order to achieve this task. I’ve installed the chain of dependencies fine.
The Group Variables which I cannot seem to access are included within the attached .png. E.g. Projection, UpperLeftPointMtrs, LowerRightMtrs and SphereCode.
I can however view them when I read in the HDF4 file as follows

f = Dataset(hdfFile)

print f

<output>

GROUP=GridStructure

GROUP=GRID_1

GridName="Grid"

XDim=8001

YDim=7271

UpperLeftPointMtrs=(310185.000000,4575015.000000)

LowerRightMtrs=(550215.000000,4356885.000000)


</output>

Can someone please advise me as how I can extract the Group Variables as I’ve explained above?
Thank you very much in advance for any assistance.
Best
Lewis

[0] netCDF4 API documentation

Dr. Lewis John McGibbney PhD, B.Sc., MAGU
Engineering Applications Software Engineer Level 2
Computer Science for Data Intensive Systems Group 398M
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel: (+1) (818)-393-7402<tel:%28%2B1%29%20%28818%29-393-7402>
Cell: (+1) (626)-487-3476<tel:%28%2B1%29%20%28626%29-487-3476>
Fax: (+1) (818)-393-1190<tel:%28%2B1%29%20%28818%29-393-1190>
Email: lewis.j.mcgibbney@jpl.nasa.gov<mailto:lewis.j.mcgibbney@jpl.nasa.gov>

           [cid:0AFCAB22-5194-48A3-9912-0D7CDA76139D]

Dare Mighty Things

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi, Lewis!

Panoply uses NetCDF-Java underneath.
The variables you see are constructed by NetCDF-Java.
NetCDF-Java parses StructMetadata.

For Python, which is based on NetCDF-C, you need to parse StructMetadata.0 string to reconstruct coordinate variables by yourself.
We provide some Python examples at Comprehensive Examples for handling the metadata using netcdf4-python.
For example, you can parse the StructMetadata using Python regular expression like below and construct lat/lon

https://raw.githubusercontent.com/hdfeos/zoo_python/master/zoo/lpdaac/myd/MYD17A2_Gpp_1km.py

and visualize it on map.

We'll provide the complete Python LandSat 7 example later at hdfeos.org.

···

From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Mcgibbney, Lewis J (398M)
Sent: Thursday, December 04, 2014 10:51 AM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] Extracting Group Variables from Landsat 7 HDF-EOS

Hi,
No it is generated by Panoply [0], however I can see the variable fields when I print the Dataset object.
I just can't see how to actually access them cleanly.
I resorted to the following which is horrible, probably inconsistent in it's operation across various HDF4 from Landsat 7 and just downright nasty

str(f.__dict__['StructMetadata.0']).split()

This above gives me the StructMetadata.0 element of the dataset dictionary as a list. I can then navigate to the index within that list and grab the variable I require e.g. UpperLeftPointMtrs

Do you have any better suggestions. I anticipate that this hack will come back to bite me pretty soon.
Thank you very much for any information.
lewis

[0] NASA GISS: Panoply 5 netCDF, HDF and GRIB Data Viewer

Dr. Lewis John McGibbney PhD, B.Sc., MAGU
Engineering Applications Software Engineer Level 2
Computer Science for Data Intensive Systems Group 398M
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel: (+1) (818)-393-7402
Cell: (+1) (626)-487-3476
Fax: (+1) (818)-393-1190
Email: lewis.j.mcgibbney@jpl.nasa.gov<mailto:lewis.j.mcgibbney@jpl.nasa.gov>

           [cid:image001.png@01D00FB1.97386D30]

Dare Mighty Things

From: "H. Joe Lee" <hyoklee@hdfgroup.org<mailto:hyoklee@hdfgroup.org>>
Reply-To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Date: Thursday, December 4, 2014 at 7:09 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org<mailto:hdf-forum@lists.hdfgroup.org>>
Subject: Re: [Hdf-forum] Extracting Group Variables from Landsat 7 HDF-EOS

Hi, Lewis!

Is screen shot generated from ToolsUI (Netcdf-Java)?

--
Save the Earth. Save Earth data in HDF-EOS. Save Big data in HDF.

On Wed, Dec 3, 2014 at 4:40 PM, Mcgibbney, Lewis J (398M) <Lewis.J.Mcgibbney@jpl.nasa.gov<mailto:Lewis.J.Mcgibbney@jpl.nasa.gov>> wrote:
Hi Folks,
I am really struggling to extract certain Group variables from a HDF4 hdf-eos Landsat 7 file.
I am using the netcdf4-python driver [0] in order to achieve this task. I've installed the chain of dependencies fine.
The Group Variables which I cannot seem to access are included within the attached .png. E.g. Projection, UpperLeftPointMtrs, LowerRightMtrs and SphereCode.
I can however view them when I read in the HDF4 file as follows

f = Dataset(hdfFile)

print f

<output>
...

GROUP=GridStructure

GROUP=GRID_1

GridName="Grid"

XDim=8001

YDim=7271

UpperLeftPointMtrs=(310185.000000,4575015.000000)

LowerRightMtrs=(550215.000000,4356885.000000)
...
</output>

Can someone please advise me as how I can extract the Group Variables as I've explained above?
Thank you very much in advance for any assistance.
Best
Lewis

[0] netCDF4 API documentation

Dr. Lewis John McGibbney PhD, B.Sc., MAGU
Engineering Applications Software Engineer Level 2
Computer Science for Data Intensive Systems Group 398M
Jet Propulsion Laboratory
California Institute of Technology
4800 Oak Grove Drive
Pasadena, California 91109-8099
Mail Stop : 158-256C
Tel: (+1) (818)-393-7402<tel:%28%2B1%29%20%28818%29-393-7402>
Cell: (+1) (626)-487-3476<tel:%28%2B1%29%20%28626%29-487-3476>
Fax: (+1) (818)-393-1190<tel:%28%2B1%29%20%28818%29-393-1190>
Email: lewis.j.mcgibbney@jpl.nasa.gov<mailto:lewis.j.mcgibbney@jpl.nasa.gov>

           [cid:image001.png@01D00FB1.97386D30]

Dare Mighty Things

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org<mailto:Hdf-forum@lists.hdfgroup.org>
http://mail.lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

Hi Joe,

Panoply uses NetCDF-Java underneath.
The variables you see are constructed by NetCDF-Java.
NetCDF-Java parses StructMetadata.

If it were my choice then I would be using the NetCDF-Java library for this job however my client wants Python.

For Python, which is based on NetCDF-C, you need to parse StructMetadata.0 string to reconstruct coordinate variables by yourself.

This is good to know. I spent a bucket load of time trying to find, manipulate and figure out why on earth I couldn’t grab the correct fields. I don’t feel so angry with myself now.

We provide some Python examples at http://hdfeos.org/zoo for handling the metadata using netcdf4-python.
For example, you can parse the StructMetadata using Python regular expression like below and construct lat/lon

https://raw.githubusercontent.com/hdfeos/zoo_python/master/zoo/lpdaac/myd/MYD17A2_Gpp_1km.py

and visualize it on map.

We’ll provide the complete Python LandSat 7 example later at hdfeos.org.

These are super helpful thank you for sharing them. I don’t know if you guys also develop the netCDF-python driver (documentation) but it would be super helpful if this was made explicit. It is totally cryptic for people to find this out.
Once again thank you for the detail above this is super helpful.
Lewis