[ZODB-Dev] Very large amounts of output from fsrefs.py

[Marius Gedminas]

On Wed, Aug 27, 2008 at 01:32:53PM +0200, Ingvar Hagelund wrote:
> We have an solution based on a third party zope based framework, with 12
> Zope instances accessing ZODB via ZEO.
...
> Now, writing to the ZEO database is very slow, and the same goes for
> searching, which is a problem for the users that put information into
> the solution.
...
> We want to sort out if there are any obvious data points on tuning ZEO
> that we have missed, or that there are errors or corruption in the database.
...
> Software environment: ZEO from Zope 2.10.3, python-2.4.4, Debian
> GNU/Linux, Data.fs on ext3 filesystem

ZEO had a bug, fixed in rev 73871, where it would spend up to 10ms (your
OSes scheduling timeslice) in idle sleep for every object received from
the ZEO server, resulting in very long request processing times on an
otherwise idle server, if a request had to load hundreds or thousands of
persistent objects.

I do not remember the mapping from Zope 2.x versions to ZODB/ZEO
versions offhand (nor do I remember how to look that up).  It may be
worth checking if this applies to you.

> Average cpu usage on the box running ZEO is some 10%, and never above
> 65% (of one cpu core).

This is what makes me suspect you may be seeing that bug.

> Running fsrefs.py on Data.fs gives 1507079 instances of "refers to
> invalid objects". This might indicate a problem, I would guess.

This is beyond my expertise.  I'd be worried too.

Marius Gedminas
-- 
Just a note: quantum gravity discussions are _clearly_ on-topic in
c.l.py.
	-- Peter Hansen on comp.lang.python
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20080827/4ac6274f/attachment.bin