[Zope] Re: Non-responsive objects reprise

[Florent Guillaume]

You probably have a network problem, all the Zope logs show everything 
was completed normally (your points 1 and 3). Your problem may be tied 
to packet size or keepalives. A network trace, for instance using 
Ethereal, will probably help you more than anything.

Florent

Garth B. wrote:
> Hello everyone, this is from an older thread which I'm resurrecting
> with more information.
> 
> Despite Dieter's helpful pointers I'm no closer to solving this
> problem but do have more information about it in case anyone can lend
> a hand.
> 
> To quickly recap:  Periodically when visiting our zope site, certain
> objects appear not to respond.  It's consistently the same objects
> from a Page Template in one folder to an
> image somewhere else.  The site is running on Zope 2.8.1, Python 2.3.5
> and sitting behind a VHM and Apache 2.0.46 using the usual ReWrite
> rules.  This problem suddenly started
> several months ago with the site having been running smoothly for many
> months prior. This is all on Red Hat Enterprise Linux ES release 3
> (Taroon Update 2).  The server is a
> dual processor with 1GB RAM, 300GB of hard disk space, hosted by
> Rackspace.  The site is relatively large and reasonably active.  Its
> content is largely made up of Page
> Templates with a few supporting python scripts and Script (Python)'s. 
> There are also a few ZClass-based objects that offer no real unique
> functionality other than providing an
> interface for the admins to create "News" or "Feature" items. The site
> also utilizes a MySQL database.
> 
> I've noticed the following things about this problem:
> 
> =================
> 1) DeadlockDebugger shows no problems when one of the objects appears
> not to be responding.  Everything appears normal.
> 
> 2) I can ALWAYS successfully get to the non-responsive objects by
> bypassing Apache and directly viewing the Zope server's equivalent
> :8080 address.
> 
> 3) While tailing the trace.log when an object is siezing through
> Apache, I can see the request come to Zope and go right back out with
> no problem.  I think that's what
> this is illustrating:
> 
> B -1348776468 2005-11-13T10:46:37 GET
> /VirtualHostBase/http/www.domain.org:80/portal/html/VirtualHostRoot/resources/contact
> I -1348776468 2005-11-13T10:46:37 0
> A -1348776468 2005-11-13T10:46:38 200 14938
> E -1348776468 2005-11-13T10:46:38
> 
> 4) Turning on debugging output for Apache shows the following proxy
> errors when trying to access an offending object.  I've searched for
> related information about this proxy and
> only found one hit from the ZODB-DEV list from 2004 with no responses.
>  The errors:
> 
> [Sat Nov 12 00:33:33 2005] [error] [client xx.xx.xx.xx] proxy: error
> reading status line from remote server localhost
> [Sat Nov 12 00:33:33 2005] [error] [client xx.xx.xx.xx] proxy: Error
> reading from remote server returned by /contact
> [Sat Nov 12 00:34:02 2005] [error] [client xx.xx.xx.xx] proxy: error
> reading status line from remote server localhost
> [Sat Nov 12 00:34:02 2005] [error] [client xx.xx.xx.xx] proxy: Error
> reading from remote server returned by /resources/index_html
> 
> I removed the client IP.  Keep #2 and #3 in mind in the context of this problem.
> 
> 5) In case there was something in one of the templates that was
> screwing things up, I methodically removed portions of a page (or its
> inherited template).  When the page suddenly started responding
> through Apache I thought I hit paydirt, but then I noticed in one
> instance that all I removed was a block of plain HTML (no METAL/TALES
> statements) and that put me back at square one.  I think #2 and #3
> make this point irrelevant, and certain images will get hung up, too.
> 
> 6) The server is also running Mailman (using the same Python as Zope).
>  It uses a seperate virtual host container in Apache to expose its
> adminstrative interface.  One of my co-workers swears that when he
> experiences the siezing, he soon after gets several emails from one of
> the Mailman lists which is supposed to be a once-a-day broadcast-only
> list.
> I think this is more of a coincidence though, and I haven't gotten a
> big enough sample size of occurrences to rely on this report.
> 
> 7) Restarting Zope *usually* corrects the problem (on Friday,
> restarting it (several times) didn't help)
> 
> 8) Restarting Apache sometimes corrects the problem without needing to
> restart Zope.
> 
> 9) On one occasion killing Mailman suddenly made one of the offending
> objects respond for a little then stop.
> 
> 10) On the rare occasion we have had to physically reboot the server
> (like on Friday).
> 
> 11) After the server was rebooted on Friday, memory usage for Zope
> went from about 3% to 20+% as reported by 'top' over a period of about
> 12 hours.  I don't know whether that is indicative of a leak or just
> general memory consumption. Restarting Zope appears to return that
> memory back to the OS.  This memory usage is what we normally see for
> this site.
> 
> 12) Upgrading from Zope 2.7.6 to 2.8.1 appeared to help for a little
> while, but the problem either came back or never left.
> 
> 13) I briefly enabled mod_disk_cache in Apache for this site in case
> Zope was getting too stressed out.  It appeared to work wonders, but
> some file objects, like PDFs, would
> periodically be reported as corrupted by Acrobat after being
> downloaded.  I assume this was a failure to configure mod_disk_cache
> appropriately, and we've since disabled it (at
> which point Acrobat stopped complaining about corrupted PDFs.  The
> siezing problem looked as though it disappeared while mod_disk_caching
> was enabled.  Indeed, Watching the Apache and Zope logs showed
> requests more often being fulfilled only by Apache than by Zope. 
> Perhaps the proxy problems in #4 is indicative of a loaded Zope that
> needs caching. We are not running ZEO or anything like that.  Perhaps
> we should.
> =================
> 
> Apologies for the long email but I have no idea what's going on... if
> ANYONE has ANY suggestions or ideas on what else I could investigate
> it would be GREATLY appreciated!
> 
> Thank you!
> 
> Garth

-- 
Florent Guillaume, Nuxeo (Paris, France)   Director of R&D
+33 1 40 33 71 59   http://nuxeo.com   fg at nuxeo.com