Balint,
I am not sure whether pre-allocation will help the performance but there is a good chance it may since the default for chunked data sets is to allocate space incrementally (chunk by chunk) as data is written to the data set, especially if the chunks are small and there are a lot of them. If matlab has access to the low level HDF5 APIs (which I beleive it does) you can use the H5Pset_alloc_time and pass alloc_time as H5D_ALLOC_TIME_EARLY to set a dataset creation property list. There should be no need to mess with the fill value or do any filling as far as I can tell.
You will need to create a property list first then set this property then pass it in to H5Dcreate. Also, I think matlab splits the HDF5 API into classes so the function might look like H5P.set_alloc_time or something like that. It might also be worth while to check that matlab is a recent version so that it is compiled/linked against a recent HDF5 version/build.
Documentation for H5Pset_alloc_time may be found here: http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAllocTime
Good luck,
Izaak Beekman
(301)244-9367
Princeton University Doctoral Candidate
Mechanical and Aerospace Engineering
ibeekman@princeton.edu
UMD-CP Visiting Graduate Student
Aerospace Engineering
ibeekman@umiacs.umd.edu
ibeekman@umd.edu
On Tue, Dec 13, 2011 at 12:00 PM, <hdf-forum-request@hdfgroup.org> wrote:
Send Hdf-forum mailing list submissions to
hdf-forum@hdfgroup.org
To subscribe or unsubscribe via the World Wide Web, visit
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
or, via email, send a message with subject or body 'help' to
hdf-forum-request@hdfgroup.org
You can reach the person managing the list at
hdf-forum-owner@hdfgroup.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Hdf-forum digest..."
Today's Topics:
1. Datasets keep old names after parent has been renamed?
(Darren Dale)
2. speeding up write of chunked HDF (Balint Takacs)
---------- Forwarded message ----------
From: Darren Dale <dsdale24@gmail.com>
To: HDF Users Discussion List <hdf-forum@hdfgroup.org>
Cc:
Date: Mon, 12 Dec 2011 12:29:18 -0500
Subject: [Hdf-forum] Datasets keep old names after parent has been renamed?
(Apologies if this gets posted twice)
Someone reported a bug at the h5py issue tracker:
---
import h5py
# test setup
fid = h5py.File('test.hdf5', 'w')
g = fid.create_group('old_loc')
g2 = g.create_group('group')
d = g.create_dataset('dataset', data=0)
print "before move:"
print g2.name
print d.name
# now rename toplevel group
g.parent.id.move('old_loc', 'new_loc')
print "after move:"
# old parent remains in dataset name, group is ok
print g2.name
print d.name
# parent is accessed by name 'g' which does not exist any more
d.parent
fid.close()
---
That script produces the following output:
---
before move:
/old_loc/group
/old_loc/dataset
after move:
/new_loc/group
/old_loc/dataset
Traceback (most recent call last):
File "move_error.py", line 24, in <module>
d.parent
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/h5py/_hl/base.py",
line 144, in parent
return self.file[posixpath.dirname(self.name)]
File "/Users/darren/Library/Python/2.7/lib/python/site-packages/h5py/_hl/group.py",
line 128, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
File "h5o.pyx", line 176, in h5py.h5o.open (h5py/h5o.c:2814)
KeyError: "unable to open object (Symbol table: Can't open object)"
---
g.name and d.name simply return the result of h5i.get_name.
d.parent just splits d.name at the last "/" and returns the the first
part of the split.
g.parent.id.move calls H5Gmove2. I've read the warnings about
corrupting data using H5Gmove at
http://www.hdfgroup.org/HDF5/doc1.6/Groups.html#H5GUnlinkToCorrupt ,
but the situation described there does not appear to be relevant to
the problem we are seeing. Is h5py not performing the move properly,
or could this be a bug in HDF5?
Thanks,
Darren
---------- Forwarded message ----------
From: Balint Takacs <takbal@gmail.com>
To: hdf-forum@hdfgroup.org
Cc:
Date: Tue, 13 Dec 2011 12:37:08 +0000
Subject: [Hdf-forum] speeding up write of chunked HDF
Hi all,
I need to fill a huge 3D array, chunked in its second dimension. My data are coming as slices with a fixed index in the third dimension, so the layout needs to be re-ordered. The chunks are uncompressed. When the data is read, the access pattern sweeps through it in the second dimension, so the chunking layout makes sense. The data is stored on an SSD, so random access should be relatively fast. I cannot manipulate data index order.
In theory, when filling up the array, the data could be continuously written if it were to be stored in a raw file. However, with HDF this becomes painfully slow. The only way I found to speed this up somewhat is to read as much slices I can into memory, and then write together in batches, but I still experience <2MB/sec write transfers on average.
The file is gradually growing as the slices are added. If this expansion requires re-ordering the entire data, this could explain the slow write speed. I was wondering whether pre-allocating the entire file somehow could help with this, and what is the best way to do it. I could not find any related API function. I know the entire data size before the data collection starts.
The only idea I have so far is to fill the array with some dummy value (not the fill one) by sweeping through the chunking dimension before adding the slices. This would probably grow the file to its final size rapidly, but I am not sure that this helps at all, and is definitely ugly.
I am using MATLAB 2007a with the 1.6.5 HDF library it is coming with.
Thank you for you comments in advance.
Regards,
Balint
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
