[ZODB-Dev] zeo client patch in connection.py

[Paul Williams]

I did use the deadlockdebugger.  There were three threads doing folder listing and the last one was in the wait method doing a continuous loop listening for the server.

The server was disconnected.  Somehow the client just never knew.

-----Original Message-----
From: Christian Theune [mailto:ct at gocept.com] 
Sent: Thursday, March 22, 2007 2:45 AM
To: Paul Williams
Cc: zodb-dev at zope.org
Subject: RE: [ZODB-Dev] zeo client patch in connection.py

Hi,

I haven't had a look at your patch earlier.

With your error description and with the patch together, it makes sense
to me. You patched the "wait" method which is described as:

    def wait(self, msgid):
       """Invoke asyncore mainloop and wait for reply."""

This method is used to wait for any reply to function invoked on the
server. However, you limited the response time to a few seconds. That's
why long running tasks that do not respond within this time frame will
disconnect the client.

Your client lockup problem doesn't sound like something ZODB would do on
it's own. I'd ponder checking whether your application code does
something weird.

Try going back to the original state and, when getting lockups, throw in
DeadLockDebugger (remember to install first and test that it shows you
the processes, before getting a lockup) and find out in what part of the
code the threads are hanging.

Christian

Am Mittwoch, den 21.03.2007, 13:06 -0500 schrieb Paul Williams:
> Since we implemented this, any task that takes more than a few seconds (ie.  pack the database)  throws a clientdisconnectederror.  I am hoping someone out there can shed some light on what might be happening.
> 
> -----Original Message-----
> From: Christian Theune [mailto:ct at gocept.com] 
> Sent: Wednesday, March 07, 2007 9:43 AM
> To: Paul Williams
> Cc: zodb-dev at zope.org
> Subject: Re: [ZODB-Dev] zeo client patch in connection.py
> 
> Hi,
> 
> could you send this again as a patch produced using `diff` against your
> original sources OR tell us  which version of Zope/ZODB you used as the
> starting point for this patch.
> 
> Thanks,
> Christian
> 
> Am Mittwoch, den 07.03.2007, 08:52 -0600 schrieb Paul Williams:
> > Hi Everyone, 
> > 
> > We were experience problems with our zeo client setup on redhat rhel4.
> > The client would just quit responding.  No memory or cpu increase was
> > associated with this.  The client would remain hung until it was
> > restarted. 
> > 
> > We looked on the client using Netstat and the status was ESTABLISHED
> > with the zeo server.  On the zeo server the netstat said LISTENING. 
> > 
> > When running the deadlockdebugger, one thread was in asnycore wait.
> > The others were normal actions such as folder listing or folder
> > contents. 
> > 
> > We implemented a couple of lines of code on line 641 of connection.py
> > in the ZEO/zrpc packages 
> > 
> > We added and else clause to call self.close() if delay is over one
> > second.  We found that one second wasn't quite enough and moved it to
> > 5 seconds. 
> > 
> > Now we find out that this drastically improved our performance. The
> > servers are now < 1 second per page load.  Before, they could be 5
> > seconds or more, if they loaded at all.  Also, our servers used to
> > crash several times a day and they now haven't crashed in almost a
> > week. 
> > 
> > I just wanted to put this out there and see if anyone has any comments
> > at all.  I need to get a more permanent solution than this, but it is
> > what we have for now. 
> > 
> > System Configuration 
> > Zope 2.9.5 
> > Plone 2.5.1 
> > Python 2.4.3 
> > Redhat Rhel4 
> > 
> > Communications between our zeo clients and zeo server only route
> > through a switch. 
> > 
> > 
> > Thank you for any help, 
> > Paul Williams
> > 
> > 
> > _______________________________________________
> > For more information about ZODB, see the ZODB Wiki:
> > http://www.zope.org/Wikis/ZODB/
> > 
> > ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> > http://mail.zope.org/mailman/listinfo/zodb-dev
-- 
gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany
www.gocept.com - ct at gocept.com - phone +49 345 122 9889 7 -
fax +49 345 122 9889 1 - zope and plone consulting and development