I have an HDF-5 based system which consists of two applications. The first is a data collector which harvests data from several thousand surface observing stations and stores them in a single HDF-5 file. The second is a web service which provides data to requestors from this same HDF-5 file. These are separate applications which run independently of each other.
What I'm observing is that, if the data collector is running and the web service receives a request, HDF-5 errors often result--sometimes a diagnostic backtrace (see below), but sometimes a segmentation fault. If the data collector is not running the web service performs flawlessly.
I suspect but can't really determine that the errors result when station data is being requested as that station data is being written into the HDF-5. I am using Python 2.6.2, PyTables 2.1.1, Numpy 1.3.0, and HDF 1.8.1.
Below is a sample backtrace. I should mention that this problem did not arise under earlier versions of these modules, but of course it could be that I just got lucky. Any insight is greatly appreciated.
tables.exceptions.HDF5ExtError: Can't open the group: '/'.
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5G.c line 699 in H5Gclose(): not a group
major: Invalid arguments to routine
minor: Inappropriate type
Exception tables.exceptions.HDF5ExtError: HDF5ExtError('Problems closing the Group /',) in ignored
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5F.c line 1560 in H5Fopen(): unable to open file
major: File accessability
minor: Unable to open file
#001: H5F.c line 1337 in H5F_open(): unable to read superblock
major: File accessability
minor: Read failed
#002: H5Fsuper.c line 542 in H5F_super_read(): truncated file
major: File accessability
minor: File has been truncated
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5Gdeprec.c line 293 in H5Gopen1(): not a location
major: Invalid arguments to routine
minor: Inappropriate type
#001: H5Gloc.c line 241 in H5G_loc(): invalid object ID
major: Invalid arguments to routine
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5Adeprec.c line 208 in H5Aopen_name(): not a location
major: Invalid arguments to routine
minor: Inappropriate type
#001: H5Gloc.c line 241 in H5G_loc(): invalid object ID
major: Invalid arguments to routine
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5G.c line 699 in H5Gclose(): not a group
major: Invalid arguments to routine
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5F.c line 2005 in H5Fclose(): not a file ID
major: Invalid arguments to routine
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5F.c line 1560 in H5Fopen(): unable to open file
major: File accessability
minor: Unable to open file
#001: H5F.c line 1337 in H5F_open(): unable to read superblock
major: File accessability
minor: Read failed
#002: H5Fsuper.c line 542 in H5F_super_read(): truncated file
major: File accessability
minor: File has been truncated
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5F.c line 3287 in H5Fget_mdc_config(): not a file ID
major: Invalid arguments to routine
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5F.c line 3346 in H5Fset_mdc_config(): not a file ID
major: Invalid arguments to routine
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5Gdeprec.c line 293 in H5Gopen1(): not a location
major: Invalid arguments to routine
minor: Inappropriate type
#001: H5Gloc.c line 241 in H5G_loc(): invalid object ID
major: Invalid arguments to routine
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5Adeprec.c line 208 in H5Aopen_name(): not a location
major: Invalid arguments to routine
minor: Inappropriate type
#001: H5Gloc.c line 241 in H5G_loc(): invalid object ID
major: Invalid arguments to routine
minor: Bad value
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5G.c line 699 in H5Gclose(): not a group
major: Invalid arguments to routine
minor: Inappropriate type
HDF5-DIAG: Error detected in HDF5 (1.8.1) thread 3086555808:
#000: H5Gdeprec.c line 293 in H5Gopen1(): not a location
major: Invalid arguments to routine
minor: Inappropriate type
#001: H5Gloc.c line 241 in H5G_loc(): invalid object ID
major: Invalid arguments to routine
minor: Bad value
Traceback (most recent call last):
File "/usr/local/lib/python2.6/site-packages/twisted/protocols/basic.py", line 231, in dataReceived
why = self.lineReceived(line)
File "/usr/local/lib/python2.6/site-packages/twisted/web/http.py", line 1325, in lineReceived
self.allContentReceived()
File "/usr/local/lib/python2.6/site-packages/twisted/web/http.py", line 1391, in allContentReceived
req.requestReceived(command, path, version)
File "/usr/local/lib/python2.6/site-packages/twisted/web/http.py", line 714, in requestReceived
self.process()
--- <exception caught here> ---
File "/usr/local/lib/python2.6/site-packages/twisted/web/server.py", line 150, in process
self.render(resrc)
File "/usr/local/lib/python2.6/site-packages/twisted/web/server.py", line 157, in render
body = resrc.render(self)
File "/home/dsallis/sps/sps_syndication/SyndicationFeed.py", line 423, in render
boundingBox=boundingBox).renderItem()))
File "/home/dsallis/sps/sps_syndication/syndicators.py", line 1034, in renderItem
fp = tables.openFile(SPS_DATA_FILE, 'r')
File "/usr/local/lib/python2.6/site-packages/tables/file.py", line 230, in openFile
return File(filename, mode, title, rootUEP, filters, **kwargs)
File "/usr/local/lib/python2.6/site-packages/tables/file.py", line 520, in __init__
self.root = root = self.__getRootGroup(rootUEP, title, filters)
File "/usr/local/lib/python2.6/site-packages/tables/file.py", line 565, in __getRootGroup
return RootGroup(self, rootUEP, title=title, new=new, filters=filters)
File "/usr/local/lib/python2.6/site-packages/tables/group.py", line 1134, in __init__
self._v_objectID = self._g_open()
File "hdf5Extension.pyx", line 615, in tables.hdf5Extension.Group._g_open
···
--
David E. Sallis, Senior Principal Engineer, Software
General Dynamics Information Technology
NOAA Coastal Data Development Center
Stennis Space Center, Mississippi
228.688.3805
david.sallis@gdit.com
david.sallis@noaa.gov
--------------------------------------------
"Better Living Through Software Engineering"
--------------------------------------------