# hdf work with hadoop map reduce **URL:** \ **Category:** Hdf-forum archives\ **Created:** [January 12, 2010, 2:56pm UTC](https://forum.hdfgroup.org/t/hdf-work-with-hadoop-map-reduce/1555 "2010-01-12T14:56:00Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![shawn\_shen](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/s/ecc23a/32.png) [@shawn\_shen](https://forum.hdfgroup.org/u/shawn_shen)\ **Post date:** [January 12, 2010, 2:56pm UTC](https://forum.hdfgroup.org/t/hdf-work-with-hadoop-map-reduce/1555/1 "2010-01-12T14:56:00Z")
Hi all, Is there a pure java HDF file reader impletation? We are working on hadoop distributed file system, and only it's specific api can access HDF file stores in it (which cannot be read by hdf native library). So, i need a pure java HDF file reader. Thanks! ---
**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\ **Post date:** [January 12, 2010, 3:30pm UTC](https://forum.hdfgroup.org/t/hdf-work-with-hadoop-map-reduce/1555/2 "2010-01-12T15:30:43Z")
For pure HDF5 Java reader check NetCDF-Java library > **[Unidata | NetCDF Java](https://www.unidata.ucar.edu/software/netcdf-java/)** > > Unidata's The NetCDF Java library implements the Common Data Model (CDM) to interface netCDF files and other types of scientific data formats. Elena > **···** > > On Jan 12, 2010, at 8:56 AM, shensl wrote: > > > Hi all, > > Is there a pure java HDF file reader impletation? We are working on hadoop distributed file system, and only it's specific api > > can access HDF file stores in it (which cannot be read by hdf native library). > > So, i need a pure java HDF file reader. Thanks! > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![shawn\_shen](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/s/ecc23a/32.png) [@shawn\_shen](https://forum.hdfgroup.org/u/shawn_shen)\ **Post date:** [January 13, 2010, 4:23pm UTC](https://forum.hdfgroup.org/t/hdf-work-with-hadoop-map-reduce/1555/3 "2010-01-13T16:23:20Z")
Thanks Elena, I checked out NetCDF-Java library, but i found it also depends on HDF native library. Our problem is that we have about 2000T HDF files. we have to find a distributed computing platform to run our application. Is there anyone worked with Hadoop map/reduce framework with HDF files before ? Thanks. 在2010-01-12?23:30:43，"Elena?Pourmal"?\?写道： > **···** > > > For?pure?HDF5?Java?reader?check?NetCDF-Java?library > > > > [http://www.unidata.ucar.edu/software/netcdf-java/](http://www.unidata.ucar.edu/software/netcdf-java/) > > > > Elena > > On?Jan?12,?2010,?at?8:56?AM,?shensl?wrote: > > > > > ?Hi?all, > > > ?Is?there?a?pure?java?HDF?file?reader?impletation???We?are?working?on?hadoop?distributed?file?system,?and?only?it's?specific?api > > > ?can?access?HDF?file?stores?in?it?(which?cannot?be?read?by?hdf?native?library).? > > > ?So,?i?need?a?pure?java?HDF?file?reader.??Thanks! > > > ? > > > ? > > > ?\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > ?Hdf-forum?is?for?HDF?software?users?discussion. > > > ?Hdf-forum@hdfgroup.org > > > ?[http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum?is?for?HDF?software?users?discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![epourmal](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/e/c67d28/32.png) [@epourmal](https://forum.hdfgroup.org/u/epourmal)\ **Post date:** [January 13, 2010, 11:28pm UTC](https://forum.hdfgroup.org/t/hdf-work-with-hadoop-map-reduce/1555/4 "2010-01-13T23:28:57Z")
Sorry, I am probably missing something. > Thanks Elena, > I checked out NetCDF-Java library, but i found it also depends on HDF native library. Could you please point me to this information (I hope we are talking about HDF5 and not about HDF4, but it may be a wrong assumption 😉 To my knowledge NetCDF HDF5 Java reader doesn't use HDF5 library (compare with the HDF-Java library and tools that come from our group). here is the info from the UCAR's netcdf-java web site Software libraries used by the Netcdf-Java library: Apache Commons HttpClient for HTTP access, which also use commons-logging and commons-codec. Java-OPeNDAP library for reading OPeNDAP. JDOM for parsing XML. JGoodies for UI stuff JPEG2000 library for decoding GRIB2 files. Simple Logging Facade for Java (SLF4J) for logging Spring lightweight application framework. Used in ToolsUI application. VisAD for reading McIDAS AREA files. Thank you! Elena > **···** > > On Jan 13, 2010, at 10:23 AM, shensl wrote: > > > Our problem is that we have about 2000T HDF files. we have to find a distributed computing platform to run our application. Is there anyone worked with Hadoop map/reduce framework with HDF files before ? Thanks. > > 在2010-01-12 23:30:43，"Elena Pourmal" \ 写道： > > \>For pure HDF5 Java reader check NetCDF-Java library > > \> > > \>[http://www.unidata.ucar.edu/software/netcdf-java/](http://www.unidata.ucar.edu/software/netcdf-java/) > > \> > > \>Elena > > \>On Jan 12, 2010, at 8:56 AM, shensl wrote: > > \> > > \>\> Hi all, > > \>\> Is there a pure java HDF file reader impletation? We are working on hadoop distributed file system, and only it's specific api > > \>\> can access HDF file stores in it (which cannot be read by hdf native library). > > \>\> So, i need a pure java HDF file reader. Thanks! > > \>\> > > \>\> > > \>\> \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > \>\> Hdf-forum is for HDF software users discussion. > > \>\> Hdf-forum@hdfgroup.org > > \>\> [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > \> > > \> > > \>\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > \>Hdf-forum is for HDF software users discussion. > > \>Hdf-forum@hdfgroup.org > > \>[http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)