HDF5 -writing to memory in Java question

I'm currently evaluating HDF5 for use with Java to store datasets. Using the examples given, I have no problem in creating an HDF file that represents a simple example Java object. However, instead of writing the data to a file and incurring the costs of File I/O, I'd like to write the data to memory instead (ultimately to a data stream but for test purpose writing the data to memory as an object would be fine). Is there a good Java example for this? Is it possible? Or can I only use Java to write to files?

If anyone has sample code showing this for a simple Java object, I would be more than grateful.

We don't have any Java example. If you have the HDF5 source code, you should be able to find
some C examples at the "test/" directory. If you are using the HDF5 Java wrapper, you can follow
the C examples and write your own Java example. Something like

        int fapl_id = H5.H5Pcreate(HDF5Constants.H5P_FILE_ACCESS);
        H5.H5Pset_fapl_core(fapl_id, 1024, true);

The core driver was not tested in HDF5 Java.

Thanks
--pc

Nigel Pickard wrote:

···

I'm currently evaluating HDF5 for use with Java to store datasets. Using the examples given, I have no problem in creating an HDF file that represents a simple example Java object. However, instead of writing the data to a file and incurring the costs of File I/O, I'd like to write the data to memory instead (ultimately to a data stream but for test purpose writing the data to memory as an object would be fine). Is there a good Java example for this? Is it possible? Or can I only use Java to write to files?

If anyone has sample code showing this for a simple Java object, I would be more than grateful.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Peter:

thanks for your reply. However, I'm still having problems attempting to implement.... E.g.:

             int fapl_id = H5.H5Pcreate(HDF5Constants.H5P_FILE_ACCESS); //create a property list of type file access
             int increment = 1024; //block size of 1024
             boolean backing_store = false; //do not write contents to disk when finished

             //set the core
             int access_id = H5.H5Pset_fapl_core(fapl_id, increment, backing_store);

             String name = "should_not_exist.txt";
             int flags = HDF5Constants.H5P_DEFAULT;
             int create_id = HDF5Constants.H5P_DEFAULT;
             int file_id = H5.H5Fcreate(name, flags, create_id, access_id);

In the code above, the access_id is being returned as 0 -that makes me wonder if it is correct. Assuming that is correct, I get an error anyway when I attempt to call H5FCreate:

ncsa.hdf.hdf5lib.exceptions.HDF5LibraryException
HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:
   #000: H5Edeprec.c line 182 in H5Eget_minor(): not a error message ID
     major: Invalid arguments to routine
     minor: Inappropriate type

Am I going about this the right way? Can you direct me to the C/C++ example you mentioned? (I tried to look for a /test directory and could not find it). Any help is greatly appreciated,

Nigel

Hi

Small correction to your code:

            int error_code = H5.H5Pset_fapl_core(fapl_id, increment, backing_store);
...
            int file_id = H5.H5Fcreate(name, flags, create_id, fapl_id);

Attached is a C example, check test_core() function for details.

Nigel Pickard wrote:

vfd.c (32.2 KB)

···

Peter:

thanks for your reply. However, I'm still having problems attempting to implement.... E.g.:

            int fapl_id = H5.H5Pcreate(HDF5Constants.H5P_FILE_ACCESS); //create a property list of type file access
            int increment = 1024; //block size of 1024
            boolean backing_store = false; //do not write contents to disk when finished

            //set the core
            int access_id = H5.H5Pset_fapl_core(fapl_id, increment, backing_store);

            String name = "should_not_exist.txt";
            int flags = HDF5Constants.H5P_DEFAULT;
            int create_id = HDF5Constants.H5P_DEFAULT;
            int file_id = H5.H5Fcreate(name, flags, create_id, access_id);

In the code above, the access_id is being returned as 0 -that makes me wonder if it is correct. Assuming that is correct, I get an error anyway when I attempt to call H5FCreate:

ncsa.hdf.hdf5lib.exceptions.HDF5LibraryException
HDF5-DIAG: Error detected in HDF5 (1.8.4-patch1) thread 0:
  #000: H5Edeprec.c line 182 in H5Eget_minor(): not a error message ID
    major: Invalid arguments to routine
    minor: Inappropriate type

Am I going about this the right way? Can you direct me to the C/C++ example you mentioned? (I tried to look for a /test directory and could not find it). Any help is greatly appreciated,

Nigel

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Correction: I did find a test directory, and found vfd.c with a function of test_core at around line 360. I'll look into this.

Thanks again, Peter, I think our messages crossed! As soon as I get a working example, I shall post it in this group.

Hi all,

As you may know already, we added over 100 HDF5 1.8 APIs to HDF-Java JNI.

For the functions with a return error code, e.g. "herr_t H5Oclose( hid_t oid)", we
replace them with "void" in Java, e.g. "void H5Oclose( int oid)" because the return
code is useless in java:
1) when there is an error, the Java call will throw an exception before you can reach
    the error code.
2) when the function call succeed (no exception), you know the error code has a
    non-negative value.

My question is what to do with the old functions with a return error code, e.g.
//C function -- herr_t H5Fclose( hid_t fid);
Java function -- int H5Fclose( hid_t fid);

Should we also change the Java function to "void H5Fclose( hid_t fid)" to make it
consistent with the new Java functions? The down side is that it may break users
code if they use the error code even if the error code is useless.

Thanks
--pc

Here's some sample code to hopefully help anybody else who might be struggling with this topic. Basically a car object
with a string model description, int year and double array of top speed values is put into memory as HDF5 format, and
subsequenty retrieved.

Since the model and year member variables are small, I used attributes to describe this data in the HDF5 format. And even
though the top speeds only has 4 entries, I assume that this data might be large in practice so used a data set for it.

Since I wanted to just get this up and running, I freely used HDF5Constants.H5P_DEFAULT whenever I was in doubt, but
appreciate it may not be the best thing, particularly since I did not fully understand all implications. If any one sees anything
terribly wrong, please let me know, thanks.

     private void storeCarAsHDF5(){
         try {

             //first let's create an instance of a car -this is the java object we wish to
             //store in memory as HDF5. The car object will include 2 attributes: a string
             //model description and an integer year. It will also have a dataset of doubles
             //called topSpeeds.
             double[] topSpeeds = {30.25, 44.0, 65.1, 72.6};
             Car car = new Car("Ford Anglia", 1971, topSpeeds);
             int errorCode;

             int faplId = H5.H5Pcreate(HDF5Constants.H5P_FILE_ACCESS); //create a property list of type file access
             int increment = 512; //block size
             boolean backingStore = false; //do not write contents to disk (true = write contents to file after file is closed)

             //set the core
             errorCode = H5.H5Pset_fapl_core(faplId, increment, backingStore);
             if (errorCode<0){
                 System.out.println("Error setting the core driver.");
                 return;
             }

             String name = "should_not_exist.txt"; //after all, we don't want to use expensive I/O right?
             int flags = HDF5Constants.H5P_DEFAULT; //use default
             int createId = HDF5Constants.H5P_DEFAULT;
             int fileId = H5.H5Fcreate(name, flags, createId, faplId); //create a file id -this will be pointing to a virtual file
             // we now have a valid file id.
             int sizeHint = 0; //optional parameter indicating number of bytes to reserve for the names that will appear in the group
             int groupId = H5.H5Gcreate(fileId, car.getClass().getSimpleName(), sizeHint);

             //
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             // Here is the section that will create
             // a string model attribute, and attach it
             // to our car group, also writing in
             // the model description provided by our
             // car object.
             //
             int modelTypeId = HDF5Constants.H5T_NATIVE_CHAR;
             int rank = 2;
             long[] dims = new long[rank];
             dims[0] = 1;
             dims[1] = car.getModel().length();
             long[] maxDims = dims;
             int modelSpaceId = H5.H5Screate_simple(rank, dims, maxDims);

             //now add an attribute
             int modelAttId = H5.H5Acreate(groupId, "model", modelTypeId, modelSpaceId, 0);

             //now write a value
             String modelValue = car.getModel();
             errorCode = H5.H5Awrite(modelAttId, modelTypeId, modelValue.getBytes());
             if (errorCode<0){
                 System.out.println("Error writing car's model attribute.");
                 return;
             }
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             //

             //
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             // Here is the section that will create
             // a year int attribute, and attach it
             // to our car group, also writing in
             // the year provided by our car object.
             //
             int yearTypeId = HDF5Constants.H5T_NATIVE_INT32;
             rank = 2;
             dims = new long[rank];
             dims[0] = 1;
             dims[1] = 1;
             maxDims = dims;
             int yearSpaceId = H5.H5Screate_simple(rank, dims, maxDims);

             //now add an attribute
             int yearAttId = H5.H5Acreate(groupId, "year", yearTypeId, yearSpaceId, 0);

             //now write a value
             int[] yearValueArray = {car.getYear()};
             errorCode = H5.H5Awrite(yearAttId, yearTypeId, yearValueArray);
             if (errorCode<0){
                 System.out.println("Error writing car's year attribute.");
                 return;
             }
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             //

             //
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             // Here is the section that will create
             // a dataset and attach it
             // to our car group, also writing in
             // the top speed values provided by our car object.
             //
             int topSpeedsTypeId = HDF5Constants.H5T_NATIVE_DOUBLE;
             rank = 2;
             dims = new long[rank];
             dims[0] = 1;
             dims[1] = car.getTopSpeeds().length;
             maxDims = dims;
             int topSpeedsSpaceId = H5.H5Screate_simple(rank, dims, maxDims);

             //now create a dataset
             int topSpeedsDataSetId = H5.H5Dcreate(groupId, "top speeds", topSpeedsTypeId,
                     topSpeedsSpaceId, HDF5Constants.H5P_DEFAULT);
             //int propertyList = H5.H5Pcreate(HDF5Constants.H5P_DEFAULT);

             //now write the values
             double[] topSpeedsValueArray = car.getTopSpeeds();
             errorCode = H5.H5Dwrite(topSpeedsDataSetId, topSpeedsTypeId,
                     HDF5Constants.H5P_DEFAULT, HDF5Constants.H5P_DEFAULT,
                     HDF5Constants.H5P_DEFAULT, topSpeedsValueArray);
             if (errorCode<0){
                 System.out.println("Error writing car's top speeds dataset.");
                 return;
             }
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             //

             //OK, let's close the file here....
             H5.H5Fclose(fileId);

             //now let's retrieve it to test to see if we have the
             //same data.....
             fileId = H5.H5Fopen(name, flags, faplId);
             groupId = H5.H5Gopen(fileId, car.getClass().getSimpleName());

             //
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             // Retrieving the name attribute
             //
             modelAttId = H5.H5Aopen_idx(groupId, 0); //the index for the name
                                                     //attribute should be 0 as
                                                     //we added that first
             String[] nameMetaNameArray = {""};
             long nameMetaNameLength = H5.H5Aget_name(modelAttId, increment, nameMetaNameArray);
             if (nameMetaNameLength<1){
                 System.out.println("Attribute meta name for model can not be found.");
                 return;
             }
             System.out.println("The attribute name is: " + nameMetaNameArray[0]);
             byte[] modelByteArray = new byte[car.getModel().length()];
             errorCode = H5.H5Aread(modelAttId, modelTypeId, modelByteArray);
             if (errorCode<0){
                 System.out.println("Attribute name value can not be read.");
                 return;
             }
             System.out.println("The value is: " + new String(modelByteArray).trim());
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             //

             //
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             // Retrieving the age attribute
             //
             yearAttId = H5.H5Aopen_idx(groupId, 1); //note index is 1; was added
                                                     //after name attribute
             String[] yearMetaNameArray = {""};
             long yearMetaNameLength = H5.H5Aget_name(yearAttId, increment, yearMetaNameArray);
             if (yearMetaNameLength<1){
                 System.out.println("Attribute meta name for year can not be found.");
                 return;
             }
             System.out.println("The attribute name is: " + yearMetaNameArray[0]);
             int[] yearIntArray = new int[1];
             errorCode = H5.H5Aread(yearAttId, yearTypeId, yearIntArray);
             if (errorCode<0){
                 System.out.println("Attribute year value can not be read.");
                 return;
             }
             System.out.println("The value is: " + yearIntArray[0]);
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             //

             //
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             // Retrieving the dataset
             //
             topSpeedsDataSetId = H5.H5Dopen(groupId, "top speeds");
             double[] topSpeedsDoubleArray = new double[car.getTopSpeeds().length];
             errorCode = H5.H5Dread(topSpeedsDataSetId, topSpeedsTypeId,
                     HDF5Constants.H5P_DEFAULT, HDF5Constants.H5P_DEFAULT,
                     HDF5Constants.H5P_DEFAULT, topSpeedsDoubleArray);
             if (errorCode<0){
                 System.out.println("Dataset top speeds can not be read.");
                 return;
             }

             for (int count=0; count<topSpeedsDoubleArray.length; count++){
                 System.out.println("top speed: " + topSpeedsDoubleArray[count]);
             }
             //~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
             //
         }catch(HDF5LibraryException e){
             e.printStackTrace();
             System.out.println("Exception is " + e);
         }catch(HDF5Exception e){
             e.printStackTrace();
             System.out.println("Exception is " + e);
         }catch(NullPointerException e){
             e.printStackTrace();
             System.out.println("Exception is " + e);
         } catch (Exception e) {
             e.printStackTrace();
             System.out.println("Exception is " + e);
         }
     }

     //here's our car object
     public class Car{
         private String model;
         private int year;
         private double[] topSpeeds;

         public Car(String model, int year, double[] topSpeeds){
             this.model = model;
             this.year = year;
             this.topSpeeds = topSpeeds;
         }

         public String getModel(){
             return model;
         }

         public int getYear(){
             return year;
         }

         public double[] getTopSpeeds(){
             return topSpeeds;
         }
     }

What about deprecating methods that return useless codes? I can't think it's a good idea to let them live on (certainly with respect to future development and where consistency is compromized). Unfortunately that would mean you would also have to change the method name slightly (can't override with the exact same method name even if the return type might be different).

Ultimately I'd prefer to see return codes that aren't doing anything removed.

···

On 4/13/2010 10:20 AM, Peter Cao wrote:

Hi all,

As you may know already, we added over 100 HDF5 1.8 APIs to HDF-Java JNI.

For the functions with a return error code, e.g. "herr_t H5Oclose(
hid_t oid)", we
replace them with "void" in Java, e.g. "void H5Oclose( int oid)" because
the return
code is useless in java:
1) when there is an error, the Java call will throw an exception before
you can reach
     the error code.
2) when the function call succeed (no exception), you know the error
code has a
     non-negative value.

My question is what to do with the old functions with a return error
code, e.g.
//C function -- herr_t H5Fclose( hid_t fid);
Java function -- int H5Fclose( hid_t fid);

Should we also change the Java function to "void H5Fclose( hid_t fid)"
to make it
consistent with the new Java functions? The down side is that it may
break users
code if they use the error code even if the error code is useless.

Thanks
--pc

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

It would be nice if you can run your program with a large array and get some performance results.
Since you are not writing data to disk at all, I expect that the your read/write speed could as high
as 1GB/s instead of some disk I/O speed, e.g. 100MB/s. I wonder if JVM will create a bottleneck.

Thanks
--pc

Nigel Pickard wrote:

···

Here's some sample code to hopefully help anybody else who might be struggling with this topic. Basically a car object
with a string model description, int year and double array of top speed values is put into memory as HDF5 format, and
subsequenty retrieved.

Since the model and year member variables are small, I used attributes to describe this data in the HDF5 format. And even
though the top speeds only has 4 entries, I assume that this data might be large in practice so used a data set for it.

Since I wanted to just get this up and running, I freely used HDF5Constants.H5P_DEFAULT whenever I was in doubt, but
appreciate it may not be the best thing, particularly since I did not fully understand all implications. If any one sees anything
terribly wrong, please let me know, thanks.