[ZODB-Dev] Zope or Zeo hanging up sockets

Matt matt@virtualspectator.com
Wed, 28 Nov 2001 16:44:21 +1300


I would be happy to help.


First of all, and don't laugh, the following are the external methods
involved :

def makeHQOned(self,REQUEST,RESPONSE):
    RESPONSE.setHeader('content-type','text/xml')
    if hasattr(self.aq_base,'oned_std'):
        oned_txt = getattr(self.aq_base,'oned_std').content
        new_oned = change_version(oned_txt)
        return new_oned
    else:
        return "<error/>"

def change_version(oned_txt):
    document_node = vs_dom.ReadXMLFromString(oned_txt)
    vs_xpath.modifyElementAttributes(document_node,'//interval/@value','40')
    new_text = vs_dom.retPrettyPrint(document_node)
    return new_text


Basically, all it does is make sure there is an object called oned_std in
the current container object that the method HQOned is called on.  This
object is a simple Product of mine where content is just a python string
that holds some xml.  This method calls change_version which 1) creates a
dom document of the 4Suite kind, 2) uses xpath to find the node I want to
change, and 3) print the result back into a string and return that as the
response of the method.  It was a simple method to dynamically change some
xml in a response based on a specific domain name of the requester ....
which is determined by the initial external request on another external
method below :

def GrabOned(self,REQUEST,RESPONSE):
    """Grab a different oned based on url in name"""
    RESPONSE.setHeader('content-type','text/xml')
    RESPONSE.setHeader('Expires','Fri, 15 Oct 2010 09:00:00 GMT')  # matt
being cache friendly .. query strings change via the client
    if REQUEST['HTTP_HOST'] ==
"volvo2001snap.sailing.live.virtualspectator.com" :
        txt = getattr(self,'oned_volvoHQ')(REQUEST,RESPONSE)
    else :
        txt = getattr(self,'oned_std').content
    return txt


These are all very hacky in nature ... but they serve the purpose for now
while we refactor our thinking.

A call may look like /live/oned.xml?big-client-generated-query-string
where oned.xml is bound to the External Method GrabOned



Now in answer to you other questions :

> It sounds like you've already got a line on where the problem is.

Perhaps I see if I can get it to do the same thing running in the foreground
with -D and put an import pdb and pdb.set_trace() and then sit and wait to
see if anything can get through, even when not busy.  Or am I definately
forcing Zope into single thread mode in that instance?   I use pdb all the
time when debuggin my products and methods and I think I remember seeing
other requests being honoured.  Anyway, not sure how non-linear that would
end up being.

> What does the external method do?

As above

 How long does it take if there is
> no load whatsoever?

about 1-2 seconds at the most when not loaded, and now that I look at
things, up to 20 seconds when very loaded, hmm, that could be a clue, I
should get more emperical evidence for that.

 How often does it get called?

When those particular people are using their client application, about every
40 seconds.

 Can you speed it
> up?
>

Yes I can speed it up, and I can also change the scripts on a dev server so
that I can pretend to be these clients.  BTW : I have a completely identical
setup on a dev server, so I really can hack into things without ruining the
live one, if you have things you want me to try that is.

regards
Matt



----- Original Message -----
From: "Chris McDonough" <chrism@zope.com>
To: "Matt" <matt@virtualspectator.com>; <zodb-dev@zope.org>
Sent: Wednesday, November 28, 2001 3:25 PM
Subject: Re: [ZODB-Dev] Zope or Zeo hanging up sockets


> It sounds like you've got a repeatable situation there.  I'd be very
> interested to try to help you get to the bottom of this.  We've got a
> customer where this symptom tends to happen as well (although less
> predictably).
>
> It sounds like you've already got a line on where the problem is.
> What does the external method do?  How long does it take if there is
> no load whatsoever?  How often does it get called?  Can you speed it
> up?
>
> ----- Original Message -----
> From: "Matt" <matt@virtualspectator.com>
> To: <zodb-dev@zope.org>
> Sent: Tuesday, November 27, 2001 9:06 PM
> Subject: [ZODB-Dev] Zope or Zeo hanging up sockets
>
>
> > I am getting a really strange behavior with requests with Zope on
> ZEO.  Not
> > sure what part of the equation is to blame.  Our site's busy time
> consists
> > of about 20-40 requests per second, mainly because we have some
> heavy caches
> > in front of us.  Sometimes we are finding that the Zope process
> suddenly
> > accelerates to 99.9% CPU and stays there.  A sockstat shows that
> connections
> > are piling up to Zope.  I watched the logs for a while and found
> that when a
> > certain external method was called that takes up to 10 seconds to
> run and
> > return, that all the other requests pile up behind it.  When it
> finally goes
> > through then a burst of some, but not all, of other pending requests
> goes
> > through, I am guessing as many as can fit before the method has been
> called
> > again.  To me this seems as if there is only one thread of python
> processing
> > these requests, though in other instances I know for sure that a
> longer
> > process, on the order of minutes, is being processed by Zope, and I
> can
> > still see requests coming through.  I was wondering where to turn to
> for
> > debugging.  The following is my setup :
> >
> > At present I am running ZEO on a UNIX socket as I only have one Zope
> looking
> > at it right now, but not for long.  I have also switched back to
> Data.fs
> > storage to get the speed increase, but was experiencing the same
> socket
> > lockup while using bdb storage.
> >
> > Zope Version (Zope 2.4.0 (source release, python 2.1, linux2),
> python 2.1.0,
> > freebsd4)
> > Python Version 2.1 (#1, Jul 18 2001, 17:06:28) [GCC 2.95.3 [FreeBSD]
> > 20010315 (release)]
> > System Platform freebsd4
> >
> > the guts of me start scripts are as follows :
> >
> > Zope :
> >
> > exec chroot /usr/local/sandbox /usr/local/bin/python \
> >     $INST_HOME/z2.py \
> >     -m '' -f "${FTPPORT}" -a ${MYIP} -p '' -w "${WWWPORT}" -F '' \
> >     -Z ''
> >
> > Zeo :
> >
> > exec chroot /usr/local/sandbox /usr/local/bin/python \
> >     $INST_HOME/lib/python/ZEO/start.py -U $INST_HOME/var/zeo.soc \
> >     -S main=StorageConfig:data_storage \
> >     -S bdb=StorageConfig:bdb_storage \
> >     ZEO_SERVER_PID=$INST_HOME/var/ZEO_SERVER.pid \
> >     -s "$@"
> >
> > regards
> > Matt
> >
> >
> >
> >
> > _______________________________________________
> > For more information about ZODB, see the ZODB Wiki:
> > http://www.zope.org/Wikis/ZODB/
> >
> > ZODB-Dev mailing list  -  ZODB-Dev@zope.org
> > http://lists.zope.org/mailman/listinfo/zodb-dev
> >