- Zope-Dev - Zope lists

RE: [Zope-dev] Re: Zope hanging (poss. threads-related)
by Marcus Collins 17 Apr '00

17 Apr '00

> -----Original Message----- > From: Tres Seaver [mailto:tseaver@palladion.com] > Sent: 17 April 2000 01:20 > To: zope-dev(a)zope.org > Cc: Michel Pelletier; Marcus Collins; Tony Rossignol > Subject: Re: [Zope-dev] Re: Zope hanging (poss. threads-related) <snip> > Here are a couple of commonalities we need to look at: > > * IE 5 seems to be involved. >From combining the logs of Apache and DebugLogger, it doesn't seem restricted to IE5. The following are user agents that, it seems, initiated hung threads over the weekend: * Mozilla/4.0 (compatible; MSIE 5.0; AOL 5.0; Windows 98; DigExt) * Mozilla/4.0 (compatible; MSIE 4.01; Windows NT) * Mozilla/4.7 [en] (Win98; I) > * The management interface seems to trigger the problem (no > application-specific code involved, just manipulation of "stock" > Zope objects). Although I've experienced hanging after using the management interface, this doesn't seem to be the sole cause. There was no manage activity between Zope's last restart and our weekend's hangings. > * Framesets may be involved. Definitely. Although other threads, representing non-frameset documents, have also failed to complete, it seems the the initial hanging is *always* brought on by frameset documents. > * Perhaps premature request cancellation (likely to happen often > in the management interface) is involved. Quite possibly, although it would be incredibly hard to determine a correlation from log files... > I would like to see a description of the interaction between the > ZServer-async-select() thread and the Zope threads to which > it dispatches requests, particularly what happens when a Zope thread > tries to write to a connection which has been closed by the client > without a "clean" shutdown (e.g., attempting to write to the socket > with fail with errno == EPIPE). Do you suspect that IE 5 is not closing connections properly, or that ZServer is not handling broken connections properly? > I have a couple of testcases for this problem. > (Un)fortunately, I don't use IE5 (and don't go with girls who do! > :), and therefore they all work without a hitch. <snip> I've been exercising your test script with IE and Netscape, and have several unfinished requests, although I think at least some of these are the result of premature request cancellations. However, I did come across several instances of the following, which possibly warrant attention/explanation: request 143894896 did not end B 143894896 2000-04-17T14:58:48 GET /mctest/breakZope/test_stall?seconds:int=30 I 143894896 2000-04-17T14:58:48 0 A 143894896 2000-04-17T15:01:05 200 0 B 143894896 2000-04-17T15:01:45 GET /mctest/breakZope/test_stall?seconds:int=30 I 143894896 2000-04-17T15:01:45 0 A 143894896 2000-04-17T15:04:23 200 0 It seems strange that *the same thread*, having failed to complete the request, should begin another. This was evident from the latest hanging log data from our live site, too, but in more detail and with a twist: request 145377824 did not end I 145377824 2000-04-15T07:21:53 found_terminator: using StringIO; size=2057 B 145377824 2000-04-15T07:21:53 POST /ContentInput/Work/Generate/ConfirmSubmitContent I 145377824 2000-04-15T07:21:53 found_terminator: using StringIO; size=7300 I 145377824 2000-04-15T07:21:53 0 X 145377824 2000-04-15T07:21:53 send_response: before PCGIResponse X 145377824 2000-04-15T07:21:53 send_response: before HTTPRequest X 145377824 2000-04-15T07:21:53 send_response: before handle These must surely be two requests (see the different sizes), although I am really puzzled at how two different requests to this page (incidentally, not a frameset) could be made inside of a second, by the same person. It's also peculiar that the log begins with "I", not "B". The plot thickens... In the latest spate of hangings, a thread hung, but *some* requests continued to be served, while failed. This behaviour continued until it appeared that all requests had failed, at which point Zope was restarted (meatware process). During this time period, frameset pages were requested, and at least some failed. The above information is expanded on at http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/MarcusColli ns/ Thanks again, -- Marcus

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Tony Rossignol 17 Apr '00

17 Apr '00

More feedback here. I wasn't sure if this should warrant a new title or not so I've left it here, for those following the thread. I have, on occasion, found threads spinning for no apparent reason on our stable machine. One theory here is that the older slower machine we have that does not restart, may actually still be suffering from similar issues as the other servers but for one reason or another it does not trigger the restart process or does not get severe enough to restart. So when I see a spinner I am starting to pay attention to it. Here is some more info: top - 8:08am up 131 days, 20:18, 1 user, load average: 1.32, 1.28, 1.30 95 processes: 93 sleeping, 2 running, 0 zombie, 0 stopped CPU states: 0.1% user, 0.0% system, 0.0% nice, 0.0% idle Mem: 517528K av, 511300K used, 6228K free, 80676K shrd, 10832K buff Swap: 130748K av, 7068K used, 123680K free 153468K cached PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME COMMAND 31771 nobody 17 0 110M 110M 18132 R 0 96.7 21.9 537:41 python 9436 root 7 0 936 936 728 R 0 12.0 0.1 0:00 top 30717 nobody 0 0 207M 207M 18144 S 0 4.3 41.0 7:45 python 30763 nobody 0 0 207M 207M 18144 S 0 3.4 41.0 11:01 python ... strace -p 31771 - close(-1) = -1 EBADF (Bad file descriptor) close(-1) = -1 EBADF (Bad file descriptor) ... I don't know what to make of this but maybe it will help. During this time Zope was still able to serve content at almost a normal pace. -- ------------------------------- tonyr(a)ep.newtimes.com Director of Web Technology New Times, Inc. -------------------------------

1 0

compile warning
by Dr. Ross Lazarus 17 Apr '00

17 Apr '00

I've noticed this flash by before when compiling source distributions rossl on innana~/zope216>python wo_pcgi.py > wo_pcgi.log ./../Components/zlib/zlib.c: In function `PyZlib_compress': ./../Components/zlib/zlib.c:61: warning: assignment from incompatible pointer type rossl on innana~/zope216> fresh zope 2.1.6 source on redhat 6.1/intel. AFAIK it's been there for several releases - I'd never thought to ask... - in the light of the mysterious core dump/thread problem, does this warning matter? -- Dr Ross Lazarus Associate Professor and Sub-Dean for Information Technology Faculty of Medicine, Room 126A, A27, University of Sydney, Camperdown, NSW 2006, Australia Tel: (+61 2) 93514429 Mobile: +61414872482 Fax: (+61 2) 93516646 Email: rossl(a)med.usyd.edu.au

1 0

Re: Zope hanging (poss. threads-related)
by Dr. Ross Lazarus 17 Apr '00

17 Apr '00

Fired up 2.1.6 with default threads. (This particular installation used to crash at least once a day until I set single threaded mode) Assaulted it using this example code for 15 minutes or so with multiple IE5 clients behaving as badly as I could make them - no sign of core dump or a restart.....not sure if this is helpful or not. I didn't try multiple ip addresses for all the clients - all were on my workstation. I think it was worth the effort even though it didn't break anything - one thing we really need is a reproducible way to make this problem appear. As Amos has said, the problem may lie deep in the bowels of the applications running on top of zserver. Given that there was mention that freebsd installations seemed immune, we have the the Linux thread libraries to content with as well as all the zope code running at crash prone sites..anyone got this happening with an old linux install? I only have access to redhat 6.1 I've added a comment to Marcus's wiki page for collecting data on this problem - suggest anyone else with the problem does the same...http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/Dia… Sigh. Back to single thread. Gak. Tres Seaver wrote > I have a couple of testcases for this problem. (Un)fortunately, I don't use IE5 > (and don't go with girls who do! :), and therefore they all work without a > hitch. > > First, an ExternalMethod, implemented so:: -- Dr Ross Lazarus Associate Professor and Sub-Dean for Information Technology Faculty of Medicine, Room 126A, A27, University of Sydney, Camperdown, NSW 2006, Australia Tel: (+61 2) 93514429 Mobile: +61414872482 Fax: (+61 2) 93516646 Email: rossl(a)med.usyd.edu.au

1 0

zope and writes
by mindlace 17 Apr '00

17 Apr '00

Zopistas, There've been various comments about how the ZODB isn't particularly suited towards write-intensive scenarios. What I'm wondering is whether this shouldn't be altered to be rewrite intensive scenarios. For example, if I add an object to the ZODB, I've just added the object. However, if I *change* that object, I now have two objects- as it is now, and as it was previously. Now, is this all inclusive? I'm thinking of using the ZODB to store information like a mailing list archive. The vast majority of the data- the messages themselves- would never change. However, I'd like to have meta-data, like scoring on messages, that would change. If I change a property on a DTML Document, does the ZODB store a copy of all of the document's other properties (which haven't changed?) If so, is there a way to pack on a "per-object" basis, or would I just need to use some kind of non-versioning storage? Thanks, -- ethan mindlace fremen mindlace(a)imeme.net zope -&- imap email -&- mailing list weave your web with the web at http://imeme.net

3 2

Re: [Zope-dev] Re: Zope hanging (poss. threads-related)
by Tres Seaver 16 Apr '00

16 Apr '00

Michel Pelletier <michel(a)digicool.com> wrote: > > Marcus Collins wrote: > > > > > -----Original Message----- > > > From: Tony Rossignol [mailto:tonyr@ep.newtimes.com] > > > Sent: 14 April 2000 19:19 > > > To: Marcus Collins; zope-dev(a)zope.org > > > Subject: Re: [Zope-dev] RE: [Zope] Re: Zope hanging (poss. > > > threads-related) > > > > > Thank you for starting this. I'll try to gather up information > > > I've been trying to collect here and post it in the next few days. > > > > Thanks! Maybe you could also look at extending the DebugLogger output > > > <URL:http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/DebugL…> > > > and posting the results of any hanging there? > > > > > RE: could be zombie related - > > > > > > Where might I find more info on this? Could this zombie issue be > > > present in FCGI as well? > > > > Amos Lattier remarked in > > http://lists.zope.org/pipermail/zope-dev/2000-April/004194.html that: > > > > "The ZServer zombie stuff is to get rid of zombie client > > connections, not zombie publishing threads. These are quite > > different beasts." > > > > I'm not yet grokking the whole picture, so I can't really answer to that. > > Note that there is an outstanding issue in the Collector at > > http://classic.zope.org:8080/Collector/954/view that might be related. As > > you previously noted, there is no zombie_timeout in the FCGI server. > > What the Zombie timeout means is that after a publishing thread gets > done answering a request, the socket may not go away. This many for a a > number of reasons, the client 'hung' and is not 'putting down the phone > after the converstation is over' (so to speak) or network troubles may > prevent the connection from closing properly. This means that there is > a 'zombie' connection laying around. This zombie will probably end up > going away on its own, but if not, ZServer will kill it after a period > of time. > > The only reasorce laying around during the life of a Zombie is an tiny > little unused open socket, the Mack truck of a Zope thread that served > the request for the zombie socket does not 'hang' for that entire period > of time, but goes on after it has completed the request to serve other > requests. > > Amos is correct in that these problems are almost always at the > Application level, and not at the ZServer level. The fact that Pavlos > can prevent hanging by inserting a print statement in the asyncore loop > is suspicious, but we do not have enough information yet to point > fingers anywhere. Here are a couple of commonalities we need to look at: * IE 5 seems to be involved. * The management interface seems to trigger the problem (no application-specific code involved, just manipulation of "stock" Zope objects). * Framesets may be involved. * Perhaps premature request cancellation (likely to happen often in the management interface) is involved. I would like to see a description of the interaction between the ZServer-async-select() thread and the Zope threads to which it dispatches requests, particularly what happens when a Zope thread tries to write to a connection which has been closed by the client without a "clean" shutdown (e.g., attempting to write to the socket with fail with errno == EPIPE). I have a couple of testcases for this problem. (Un)fortunately, I don't use IE5 (and don't go with girls who do! :), and therefore they all work without a hitch. First, an ExternalMethod, implemented so:: from time import sleep def stall( secs = 30 ): """ Delay to allow testing. """ sleep( secs ) return "Slept %d seconds" % secs Called trivially from DTML method called 'test_stall':: <html> <head> <title><dtml-var document_title></title> </head> <body> <h2><dtml-var document_title></h2> <dtml-unless seconds> <dtml-call "REQUEST.set( 'seconds', 60 )"> </dtml-unless> <p><dtml-var "stall( seconds )"></p> </body></html> Exercised repeatedly through a frameset:: <html> <head> <title> Test Stalling </title> </head> <frameset rows="33%, 33%, 34%"> <frameset cols="33%, 33%, 34%"> <frame name="r1c1" src="test_stall?seconds:int=30"> <frame name="r1c2" src="test_stall?seconds:int=30"> <frame name="r1c3" src="test_stall?seconds:int=30"> </frameset> <frameset cols="33%, 33%, 34%"> <frame name="r2c1" src="test_stall?seconds:int=30"> <frame name="r2c2" src="test_stall?seconds:int=30"> <frame name="r2c3" src="test_stall?seconds:int=30"> </frameset> <frameset cols="33%, 33%, 34%"> <frame name="r3c1" src="test_stall?seconds:int=30"> <frame name="r3c2" src="test_stall?seconds:int=30"> <frame name="r3c3" src="test_stall?seconds:int=30"> </frameset> </frameset> </html> I can run this repeatedly, stopping it in mid-run, etc., all without causing any problems that I can diagnose. Anyone care to try it on their setup with IE5 as the client? Tres. -- ========================================================= Tres Seaver tseaver(a)digicool.com tseaver(a)palladion.com

1 0

Welcome message: where?
by Alexandre Miguel de Andrade Souza 16 Apr '00

16 Apr '00

hello people! I'm trying to translate Zope to anoter language, so I have translated some text and dtml files. But I can't find the text source to Welcome page, that is showed before the first acess to manage area. It´s in a binary file? Someone can help me? Alexandre Miguel [ZopeBrasil]

1 0

Print a document !!
by Juan Javier Carrera Obrero 16 Apr '00

16 Apr '00

Hello !! Anybody to know to how to print a document ? I have used the next JavaScript code: <script language="javascript"> window.print(); </script> But, it only is valid for Netscape and it is not valid for Microsoft Internet Explorer. Exists any way to print a document from ZOPE or trought JavaScript ? Thanks. ------- Juan Javier Carrera Obrero e-mail: jcarrera(a)uco.es

4 4

RE: [Zope-dev] Some thoughts on splitter
by Sin Hang Kin 16 Apr '00

16 Apr '00

> This is a good idea, although this might be a bit cumbersome to handle. A > text filled with these entities will be quite hard to read. For that reason, > I thought of using spaces. The display engine could strip out single spaces > and reduce sequences of more spaces by one. NO! &zwnj; is displayed as zero-width, not thing at all! It is one unicode char. And it does not seems to be used in other way, so it will be safe to remove from the file without hurting anyone. It is not easy for the browser or server to strip out the space as you might think. > I think the encoding should be responsibility of the user -- After all, he > knows what he wants to do. Sites that use more than one Han-Encoding could > go with unicode, other sites might prefer to use the local encoding, since > there are much more tools that can be used. Not anymore. As I have already point out, you may have your site filled with only one encoding, but someone may quote your content which might encoded in other encoding methods. Also, the user might not have the browser with the encoding you desire, so they will not able to see your content. Even they can make conversion and see them, the different in encoding prevent them from searching them. Building search engines for many different encoding is certainly not fun. Why limited your client with no good reason? Just convert your content to unicode before publish. That save you and your client a lots of time. Your content may last longer (than you want?). > If Zope starts normalizing the text, some of the users might be surprised by > the results. Normalize for the index, not altering the text. This would be the good practice which improve the search experience. Unicode have been thinking of the normalization very hard, and I think IBM have some sample implementation available at their developerWorks site. > BUt of course, certainly it should be possible to use Unicode! For this, we > will have to wait for Python 1.6, though. It is ver near. > Right, but then the user should declare the language. As I explained, if > this is inherited like the acquisition, they might get away with just one > declaration for a whole site. I can not speak for every people, but I think your surfing experience will tell. I would like to suggest working hard to make things last longer. Bearing other's mistake and prevent frequent re-write. Rgs, Kent Sin

1 0

RE: [Zope-dev] RE: [Zope] Re: Zope hanging (poss. threads-related )
by Marcus Collins 15 Apr '00

15 Apr '00

> -----Original Message----- > From: Tony Rossignol [mailto:tonyr@ep.newtimes.com] > Sent: 14 April 2000 19:19 > To: Marcus Collins; zope-dev(a)zope.org > Subject: Re: [Zope-dev] RE: [Zope] Re: Zope hanging (poss. > threads-related) > Thank you for starting this. I'll try to gather up information > I've been trying to collect here and post it in the next few days. Thanks! Maybe you could also look at extending the DebugLogger output (http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/DebugLogge r) and posting the results of any hanging there? > RE: could be zombie related - > > Where might I find more info on this? Could this zombie issue be > present in FCGI as well? Amos Lattier remarked in http://lists.zope.org/pipermail/zope-dev/2000-April/004194.html that: "The ZServer zombie stuff is to get rid of zombie client connections, not zombie publishing threads. These are quite different beasts." I'm not yet grokking the whole picture, so I can't really answer to that. Note that there is an outstanding issue in the Collector at http://classic.zope.org:8080/Collector/954/view that might be related. As you previously noted, there is no zombie_timeout in the FCGI server. > We have noticed once restarts start they get worse when under a > load. I've been suspecting that the longer pages take to load the > more people are just stopping the page load, and this would/could > create a zombie. You'll sometimes note on the console or your logs something like the following: 2000-04-14T14:00:26 ERROR(200) ZServer uncaptured python exception, closing channel <PCGIChannel at 87567b0> (socket.error:(32, 'Broken pipe') [/usr/local/Zope-2.1.6-src/ZServer/medusa/asynchat.py|initiate_send|211] [/usr/local/Zope-2.1.6-src/ZServer/medusa/asyncore.py|send|237]) This occurs (I surmise) when the client closes the channel before ZServer has sent its response. I presume the fact that it closes the channel would mean no zombie, but I'd like to know more about this. > The problem is I don't know how to identify or research what > is going on under the hood. Could when someone terminates a > connection be an issue here? I mean; if the request in queued > waiting for a thread might act differently than a request that is > being processed by Zope and waiting on a DB query, or even > termination once zope has already started passing results back > through the pipe. Our restarts are so hard to tie down I'm > guessing it's a very subtle issue or a combination of just the > wrong factors. We have quite a number of the above errors occurring, and they seem to correlate to people terminating the connection (or browser timeout), from what I've seen internally. I also feel at a loss here, focussing on a very small part of ZServer and possibly missing the big picture. > Just some more food for thought. Thanks. -- Marcus

2 1