How to create large file of 1dimensional OpaqueData

HDFView / Java HDF Object Package
1

I am quite new to hdf5 and was given a task to fix some code we have that takes in file(s) and wraps them into an hdf5 file. This all worked well until we started getting larger files (> 4GB) which exceeded the amount that java can store in a single byte array. my thought was to read in as much as possible and write it to the hdf5 file and then read in the next chunk and append it. Our code is storing all files as 1 dimensional arrays of opaque data. It seems like I need to use hyperslab to write the files to hdf5 in chunks,so I looked through the various hyperslab examples and tried to incorporate them into our code but to date all of my attempts have met with “HDF5ResourceUnavailableExceptions” or “Write Failed” exceptions.
Some of the more immediate questions I have would be

When creating the Opaque Datatype should I be setting the size to 1 byte or the entire file size?
Is chunking the file optional ?
Does anyone have any examples of the writes large binary files into an HDF5file?

2

Hi Scott,

Attached is a C example that appends to a 1D dataset with an opaque datatype. Is this helpful?

opqext.c (2.8 KB)

I created it from the h5ex_t_opaque.c example found on this page:
Examples by API

You do have to use chunking and specify an unlimited dimension dataspace:

 hsize_t     maxdims[1] = {H5S_UNLIMITED};
 .
 .
 .
prop = H5Pcreate (H5P_DATASET_CREATE);
status = H5Pset_chunk (prop, RANK, chunk_dims);

space = H5Screate_simple (1, dims, maxdims);  
dset = H5Dcreate (file, DATASET, dtype, space,  H5P_DEFAULT, prop,
            H5P_DEFAULT);

Then extend the dataset size:

size[0] = dims[0]+ dimsext[0];
status = H5Dset_extent (dset, size);

… and write data to the extended part:

filespace = H5Dget_space (dset);
memspace = H5Screate_simple (RANK, dimsext, NULL);

offset[0] = 4;
status = H5Sselect_hyperslab (filespace, H5S_SELECT_SET, offset, NULL, dimsext, NULL);
status = H5Dwrite (dset, dtype, memspace, filespace, H5P_DEFAULT, edata);

-Barbara

3

That helped immensely . By following your example I was able to get our code working. Thank you for your assistance.