- Zope-Dev - Zope lists

[zope.org] We want YOU - we need a new installation manual!
by Andreas Jung 22 May '04

22 May '04

Hi, the current installation instructions on zope.org http://www.zope.org/Documentation/Misc/InstallingZope.html are totally obsolete. Anyone volunteering to update this document for Zope 2.7? The appreciation of the Zope community will be yours. -aj -- Andreas Jung -------------------------------- www.zopyx.com Software-Entwicklung und Consulting Andreas Jung

2 2

Zope Bug Day - Friday 28th May?
by Chris Withers 22 May '04

22 May '04

Hi People, I offered to schedule and organise the Zope bug days for the last Friday of each month, and the first one of those is looming. Do people have any objections to Friday 28th May as a bug day? If I hear nothing, I'll begin more widely advertising next week... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

4 6

RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 21 May '04

21 May '04

Juan David, Indeed, thanks for the input, I've ruled out new_publish as the cause as well ... Right now, I'm at the stage where I severely suspect it's got something to do with i18n support or translation somehow, but don't know where exactly yet. If I remove the message_catalog, the problem goes away. If I remove the TranslationService instance, the problem goes away. Do you use this? Does anybody know of alternatives to the TranslationService+Localizer combo to enable i18n in ZPT's on Zope 2? If I keep the above two, but delete my custom Localizer instances, the problem persists. I was trying to find an alternative message_catalog to use, but couldn't find one (CMFLocalizer is based on yours). So I'm going to try and narrow the field some more. I *quickly* looked the message_catalog and TranslationService code, and nothing jumped out. I have to also look at the ZPT engine itself, maybe it's i18n support is the cause of the problem. I'll run further tests ... Thanks, J.F. -----Original Message----- From: "J. David Ibáñez" [mailto:jdavid@itaapy.com] Sent: May 21, 2004 8:17 AM To: Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca Cc: dieter(a)handshake.de; zope-dev(a)zope.org Subject: Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors The problem I have with one of my sites seems to have a different cause, as it does not show up when errors occur, but when requests are succesful. So far I haven't been able to reproduce the problem. You can try to modify "new_publish" to catch exceptions: def new_publish(request, module_name, after_list, debug=0): id = get_ident() Publish._requests[id] = request try: x = Publish.old_publish(request, module_name, after_list, debug) except: del Publish._requests[id] raise else: try: del Publish._requests[id] except KeyError: ... Though it is unlikely the leak to be there. Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca wrote: >Indeed, I was hoping the ref in the dictionary there was getting in the way >when old_publish was raising an exception, but I tried using a >weakref.proxy() >instead and that didn't help. > >I managed to test without Localizer (It dawned on me that since I'm looking >at errors, it really doesn't matter if I don't have Localizer installed >since >not having it will generate errors, which is what I want!) ... And the >problem >diappeared ... > >So it's definitely Localizer, but it's not the Globals() request patch ... >Or >at least not the act of patching it itself, that causes the problem. > >*sigh* >J.F. > >-----Original Message----- >From: Dieter Maurer [mailto:dieter@handshake.de] >Sent: May 20, 2004 2:58 PM >To: Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca >Cc: jdavid(a)itaapy.com; tseaver(a)zope.com; zope-dev(a)zope.org; >chrism(a)plope.com; brian(a)zope.com >Subject: RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage >caused by errors > > >Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca wrote at 2004-5-20 09:58 -0400: > > >>... >>def new_publish(request, module_name, after_list, debug=0): >> id = get_ident() >> print "Localizer got thread id: " + str(id) >> Publish._requests[id] = request >> print "Request dict is now: " + str(Publish._requests) >> x = Publish.old_publish(request, module_name, after_list, debug) >> try: >> del Publish._requests[id] >> except KeyError: >>... >> >> > >This code cannot leak requests in large numbers (though it may >leak a few requests for some time). > >Reason: Zope does not normally create new threads. This implies >that thread_ids are reused and thereby old requests flushed from >the dict. > > > -- J. David Ibáñez Founder and CTO of Itaapy <http://www.itaapy.com> 9 rue Darwin, 75018 Paris Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88

3 3

Re: Read-only root database doesn't work ... bug or feature?
by Paul Winkler 21 May '04

21 May '04

On Fri, May 21, 2004 at 03:09:48PM -0400, Tres Seaver wrote: > Paul Winkler wrote: (snip) > >I have not yet found exactly how Formulator triggers a commit, > >but it is definitely something called by its initialize() function. > > It is probably trying to do the Right Thing (TM) by registering with the > help system. The fact that the help system writes to the database is a > wart the size of my foot. Good thought, but practically everything else does that too, and Formulator's the only one that triggers this issue. I've tracked it further to Formulator.FieldRegistry.initializeFieldForm() but I don't grok what that's doing yet, only just started looking at it... -- Paul Winkler http://www.slinkp.com

1 1

RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 21 May '04

21 May '04

Indeed, I was hoping the ref in the dictionary there was getting in the way when old_publish was raising an exception, but I tried using a weakref.proxy() instead and that didn't help. I managed to test without Localizer (It dawned on me that since I'm looking at errors, it really doesn't matter if I don't have Localizer installed since not having it will generate errors, which is what I want!) ... And the problem diappeared ... So it's definitely Localizer, but it's not the Globals() request patch ... Or at least not the act of patching it itself, that causes the problem. *sigh* J.F. -----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: May 20, 2004 2:58 PM To: Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca Cc: jdavid(a)itaapy.com; tseaver(a)zope.com; zope-dev(a)zope.org; chrism(a)plope.com; brian(a)zope.com Subject: RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca wrote at 2004-5-20 09:58 -0400: > ... >def new_publish(request, module_name, after_list, debug=0): > id = get_ident() > print "Localizer got thread id: " + str(id) > Publish._requests[id] = request > print "Request dict is now: " + str(Publish._requests) > x = Publish.old_publish(request, module_name, after_list, debug) > try: > del Publish._requests[id] > except KeyError: > ... This code cannot leak requests in large numbers (though it may leak a few requests for some time). Reason: Zope does not normally create new threads. This implies that thread_ids are reused and thereby old requests flushed from the dict. -- Dieter

2 1

Preliminary findings: Zope 2.7 leakage caused by errors
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 20 May '04

20 May '04

Gents, In following to the discussions regarding the leakage zope.org and my site suffer from, I've done some testing, and here's what I've found. First of all, the leak does seem to occur when errors occur. Unlike what Chris suggested, I still suffer from this leak even when I completely remove standard_error_message. How I tested: I use JMeter, and a 99% identical box, dedicated for testing: - RH 7.3 - Python 2.3.3 compiled with ./configure and no options - Zope 2.7 + a variety of products, both custom and well known (CMF, Localizer, etc ...) I look at my error log on my main site to see the types of errors coming up and note the URL/parameters. Such errors include KeyErrors, a RecursionDepth error, a Method-something-or-other error, and so on. I setup JMeter with multiple concurrent threads and run it. Within minutes I can see refcounts growing abnormally on the test server. I then shut everything down and minimize the caches. The following are left around that shouldn't be: Class May 19, 2004 1:52 pm May 19, 2004 2:26 pm Delta Acquisition.ImplicitAcquirerWrapper 1025 1309 +284 ZServer.HTTPResponse.ZServerHTTPResponse 150 247 +97 ZPublisher.BaseRequest.RequestContainer 146 240 +94 ZPublisher.HTTPRequest.HTTPRequest 172 262 +90 Right now I've focused on finding out why the Requests are still around. Using the gc module, I've found that all leaked requests are being held by a dictionary. If I look at said dictionaries, this is what they might look like: {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/francais/mapsatoz>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/francais/mapsatoz>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/search/political>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/search/political>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/search/political>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/search/political>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/search/political>}, {3075: <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/maps/peopleandsociety/age/youth/HEAD>, 4100: <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/debug>, 5125: <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/debug>, 6150: <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/english/search/political>, 7175: <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/vh_ site/english/maps/archives/5thedition/other/referencemaps>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/debug>}, {'REQUEST': <HTTPRequest, URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh _site/misc_/WFSAdapter/sqlmethod.gif>} NOTE: I had to hack the __repr__ of HTTPRequest not to use self['URL'] since it seems that attribute is unavailable in a closed request. I use PATH_INFO instead. Most of the time I get the dictionary with the 'REQUEST' key. Those dictionaries with int's as keys are ones generated by the ExternalMethod I use to run the code to debug ... So even errors raised while trying to debug this leaks a dictionary that holds on to the request(s). Said dictionary does not seem to be referred to by anything, at least not anything the gc module can find (gc.get_referrers()). One possible situation is that said dictionary is actually part of the HTTPRequest, and we have a cyclic reference problem. I don't think so however, I've looked at the HTTPRequest instances, and they have nothing in them other than strings, lists, basic dict's and an HTTPResponse instance. I'll take another look to be sure. Next step is too look at the Wrappers and see what they're all about and what's holding them there. I'm hoping this information might give a clue to someone who knows Zope error handling better than I as to what might be the problem here ... Also I understand zope.org suffers from this? :) Based on this information also, I have no reason to believe this is caused by product code ... I have to guess it's a Zope bug? In some ways this is motivating me to reduce the number of errors that could occur, but because ALL errors seems to cause the problem, I ultimately have no control over it and under certain circumstances I suffer heavily from this. 404's alone probably cause this (I'm plannign to test that explicitely as well). Thoughts, suggestions, comments welcome ... Jean-François Doyon Internet Service Development and Systems Support / Spécialiste de dèveloppements internet et soutien technique Canada Centre for Remote Sensing / Centre Canadien de télédétection Natural Resources Canada / Ressources naturelles Canada http://atlas.gc.ca Tel./Tél. : (613) 992-4902 Fax: (613) 947-2410

5 6

RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 20 May '04

20 May '04

Juan David, I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ... BTW, does that mean it's alpha software or something? is 1.1.0 stable? If the problem comes from Localizer I suspect this patch might be the cause: def new_publish(request, module_name, after_list, debug=0): id = get_ident() print "Localizer got thread id: " + str(id) Publish._requests[id] = request print "Request dict is now: " + str(Publish._requests) x = Publish.old_publish(request, module_name, after_list, debug) try: del Publish._requests[id] except KeyError: # XXX # Some people has reported that sometimes a KeyError exception is # raised in the previous line, I haven't been able to reproduce it. # This try/except clause seems to work. I'd prefer to understand # what is happening. LOG('Localizer', PROBLEM, "The thread number %s don't has a request object associated." % id) except: print "Localizer got an exception" return x What happens if an exception occurs here: x = Publish.old_publish(request, module_name, after_list, debug) ?? If some clean up code is supposed to occur it might be held back by the fact that there's a reference to the request kept in the global dictionary ... At least that's my theory :) But then I'm mostly uneducated about the intricacies of exception handling at this level, though I've been forced to learn quickly :) I'd be curious to know at least if you can reproduce the leak by generating loads of errors on a Localizer enabled site? One option would also be to simply wrap it in a try:catch and see what happens there ... But I noticed 1.1.0a3 uses a different model for wrapping the request, so that just might do the trick. Thanks for a very cool and useful product BTW. This is combination with the TranslationService is saving me loads of time. Cheers, J.F. -----Original Message----- From: zope-dev-bounces(a)zope.org [mailto:zope-dev-bounces@zope.org]On Behalf Of "J. David Ibáñez" Sent: May 20, 2004 6:47 AM To: Tres Seaver Cc: Jean-Francois.Doyon(a)ccrs.nrcan.gc.ca; brian(a)zope.com; chrism(a)plope.com; zope-dev(a)zope.org Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in the module "ZPublisher.Publish". The keys are integers, they are the values returned by Python's "thread.get_ident". The values are "ZPublisher.HTTPRequest.HTTPRequest" instances, they are not acquisition wrappers. Jean-François, which Localizer version do you use? Just to be sure we look at the same code. Tres Seaver wrote: > Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca wrote: > >> Gents, >> >> In following to the discussions regarding the leakage zope.org and my >> site >> suffer from, I've done some testing, and here's what I've found. >> >> First of all, the leak does seem to occur when errors occur. Unlike >> what >> Chris suggested, I still suffer from this leak even when I completely >> remove standard_error_message. >> >> How I tested: >> >> I use JMeter, and a 99% identical box, dedicated for testing: >> >> - RH 7.3 >> - Python 2.3.3 compiled with ./configure and no options >> - Zope 2.7 + a variety of products, both custom and well known (CMF, >> Localizer, etc ...) >> >> I look at my error log on my main site to see the types of errors >> coming up >> and note the URL/parameters. >> Such errors include KeyErrors, a RecursionDepth error, a >> Method-something-or-other error, and so on. >> >> I setup JMeter with multiple concurrent threads and run it. Within >> minutes >> I can see refcounts growing abnormally on the test server. >> >> I then shut everything down and minimize the caches. The following >> are left >> around that shouldn't be: >> >> Class May 19, 2004 1:52 pm May 19, 2004 2:26 pm Delta >> Acquisition.ImplicitAcquirerWrapper 1025 1309 +284 >> ZServer.HTTPResponse.ZServerHTTPResponse 150 247 +97 >> ZPublisher.BaseRequest.RequestContainer 146 240 +94 >> ZPublisher.HTTPRequest.HTTPRequest 172 262 +90 >> >> Right now I've focused on finding out why the Requests are still around. >> >> Using the gc module, I've found that all leaked requests are being >> held by a >> dictionary. If I look at said dictionaries, this is what they might >> look >> like: >> >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/francais/mapsatoz>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/francais/mapsatoz>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/search/political>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/search/political>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/search/political>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/search/political>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/search/political>}, >> {3075: <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/maps/peopleandsociety/age/youth/HEAD>, >> 4100: <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/debug>, >> 5125: <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/debug>, >> 6150: <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/english/search/political>, >> 7175: <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/vh_ >> >> site/english/maps/archives/5thedition/other/referencemaps>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/debug>}, >> {'REQUEST': <HTTPRequest, >> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh >> >> _site/misc_/WFSAdapter/sqlmethod.gif>} > > > I know that the Localizer product does (or used to do) some kind of > magic to make the REQUEST available to code as a "global". Can you > reproduce this data without that product present? > >> >> NOTE: I had to hack the __repr__ of HTTPRequest not to use >> self['URL'] since >> it seems that attribute is unavailable in a closed request. I use >> PATH_INFO >> instead. >> >> Most of the time I get the dictionary with the 'REQUEST' key. Those >> dictionaries with int's as keys are ones generated by the >> ExternalMethod I >> use to run the code to debug ... So even errors raised while trying >> to debug >> this leaks a dictionary that holds on to the request(s). >> >> Said dictionary does not seem to be referred to by anything, at least >> not >> anything the gc module can find (gc.get_referrers()). One possible >> situation is that said dictionary is actually part of the >> HTTPRequest, and >> we have a cyclic reference problem. I don't think so however, I've >> looked >> at the HTTPRequest instances, and they have nothing in them other than >> strings, lists, basic dict's and an HTTPResponse instance. I'll take >> another look to be sure. > > > Again, I would be looking at the Localizer's module-level globals for > this one. > >> Next step is too look at the Wrappers and see what they're all about and >> what's holding them there. > > > The wrappers are almost certainly there because the request is holding > on to objects, which hold on to others, etc. > >> I'm hoping this information might give a clue to someone who knows Zope >> error handling better than I as to what might be the problem here ... >> Also I >> understand zope.org suffers from this? :) >> >> Based on this information also, I have no reason to believe this is >> caused >> by product code ... I have to guess it's a Zope bug? >> >> In some ways this is motivating me to reduce the number of errors >> that could >> occur, but because ALL errors seems to cause the problem, I >> ultimately have >> no control over it and under certain circumstances I suffer heavily from >> this. 404's alone probably cause this (I'm plannign to test that >> explicitely as well). >> >> Thoughts, suggestions, comments welcome ... > > > I am CC'ing Juan-David, in case he may be able to offer any insights > from the perspective of the Localizer product. > > Tres. -- J. David Ibáñez Founder and CTO of Itaapy <http://www.itaapy.com> 9 rue Darwin, 75018 Paris Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 _______________________________________________ Zope-Dev maillist - Zope-Dev(a)zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )

3 2

RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 20 May '04

20 May '04

Juan David, OK, 1.1.0a3 indeed suffers from the same problem, I'm going to take a closer look the the publishing mechanism to see if I can find anything relevant. Thanks, J.F. -----Original Message----- From: "J. David Ibáñez" [mailto:jdavid@itaapy.com] Sent: May 20, 2004 10:17 AM To: Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca Cc: tseaver(a)zope.com; brian(a)zope.com; chrism(a)plope.com; zope-dev(a)zope.org Subject: Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors The "a" in 1.1.0a3 means "alpha". If the problem is in Localizer 1.0.1 I think it will be in 1.1.0a3 too. Yes, I have a site with a similar behaviour, unfortunately today the network connection is too slooow, so I can't test that site. I'm going to do some local tests, though. Yes, the problem might be the fact that "new_publish" does not catch exceptions to remove the request from the dictionary. Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca wrote: >Juan David, > >I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ... > >BTW, does that mean it's alpha software or something? is 1.1.0 stable? > >If the problem comes from Localizer I suspect this patch might be the cause: > >def new_publish(request, module_name, after_list, debug=0): > id = get_ident() > print "Localizer got thread id: " + str(id) > Publish._requests[id] = request > print "Request dict is now: " + str(Publish._requests) > x = Publish.old_publish(request, module_name, after_list, debug) > try: > del Publish._requests[id] > except KeyError: > # XXX > # Some people has reported that sometimes a KeyError exception is > # raised in the previous line, I haven't been able to reproduce it. > # This try/except clause seems to work. I'd prefer to understand > # what is happening. > LOG('Localizer', PROBLEM, > "The thread number %s don't has a request object associated." % >id) > except: > print "Localizer got an exception" > > return x > >What happens if an exception occurs here: > > x = Publish.old_publish(request, module_name, after_list, debug) > >?? > >If some clean up code is supposed to occur it might be held back by the fact >that >there's a reference to the request kept in the global dictionary ... >At least that's my theory :) >But then I'm mostly uneducated about the intricacies of exception handling >at this >level, though I've been forced to learn quickly :) > >I'd be curious to know at least if you can reproduce the leak by generating >loads >of errors on a Localizer enabled site? > >One option would also be to simply wrap it in a try:catch and see what >happens there >... But I noticed 1.1.0a3 uses a different model for wrapping the request, >so that >just might do the trick. > >Thanks for a very cool and useful product BTW. This is combination with the >TranslationService is saving me loads of time. > >Cheers, >J.F. > >-----Original Message----- >From: zope-dev-bounces(a)zope.org [mailto:zope-dev-bounces@zope.org]On >Behalf Of "J. David Ibáñez" >Sent: May 20, 2004 6:47 AM >To: Tres Seaver >Cc: Jean-Francois.Doyon(a)ccrs.nrcan.gc.ca; brian(a)zope.com; >chrism(a)plope.com; zope-dev(a)zope.org >Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by >errors > > > >Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in >the module >"ZPublisher.Publish". The keys are integers, they are the values >returned by Python's >"thread.get_ident". The values are "ZPublisher.HTTPRequest.HTTPRequest" >instances, they are not acquisition wrappers. > >Jean-François, which Localizer version do you use? Just to be sure we >look at >the same code. > > > > -- J. David Ibáñez Founder and CTO of Itaapy <http://www.itaapy.com> 9 rue Darwin, 75018 Paris Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88

1 0

RE: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by e rrors
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 20 May '04

20 May '04

Dieter, Thanks yet again for all the valuable input. I've started tracking down/running into the whole ExtensionClasses instances/issues. As you suggest, the dict is probably help by BaseRequest.RequestContainer, which I've seen grow in parallel to the HTTPRequest refcounts ... Which seems to make sense. And as you suggest, as I've been doing this, I've been thinking I'm really looking forward to getting ri dof ExtensionClass, makes this kind of work much easier. The requests themselves look clean, but if they contain ExtensionClasses based instances gc wouldn't find them for the same reason. I've gotten far enough to be able to tell there's a variety of them. I suspect the instance the error occured ON is what's leaked (So if it's a CMFDocument, then that said Document ends up leaked). I was just getting to this yesterday, hopefully I'll solve that today. As Tres suggested, I'm going to go back to looking at Localizer ... It's the likliest suspect at this time. It's just hard for to test since pretty much everything I have depends on i18n PT's, and in some cases I have gettext() calls in Python code. Localizer "wraps" the Publish.publish, and I'm thinking that when exceptions occur in there Localizer needs to be aware of it and react accordingly, which it doesn't right now. The hunt goes on ... Cheers, J.F. -----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: May 20, 2004 3:52 AM To: Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca Cc: zope-dev(a)zope.org; tseaver(a)zope.com; chrism(a)plope.com; brian(a)zope.com Subject: Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by errors Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca wrote at 2004-5-19 14:44 -0400: > ... >Right now I've focused on finding out why the Requests are still around. > >Using the gc module, I've found that all leaked requests are being held by a >dictionary. If I look at said dictionaries, this is what they might look >like: > >{'REQUEST': <HTTPRequest, >URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_v h >_site/francais/mapsatoz>}, > {'REQUEST': <HTTPRequest, >URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_v h >_site/francais/mapsatoz>}, These dictionaries look like the "instance dict"s of the "ZPublisher.BaseRequest.RequestContainer"s. When you have any cycle involving an AcquisitionWrapper, then its reference to the "RequestContainer" will keep this alive together with its instance dict. You may analyse the content of your requests. Do they contain acquisition wrapped objects (or other objects with (maybe indirect) references to this dict)? If not, "REQUEST" does not seem to be the root of your cycles but rather a side effect. More important notes: * When you see "AcquisitionWrapper"s in the reference information in "Control_Panel --> Debug Information", then there *must* also be other prominent ExtensionClass'es -- the "aq_base" of these wrappers. These classes may provide a valuable clue towards the cause of the cycles -- of global variables (by the way). * The reference information in "Control_Panel" covers only instance objects but not elementary Python data types. Thus, when cycles are caused by elementary Python objects, we may not directly see them in the list. GC's "DEBUG_LEAK" may provide some information. * The leaks may not be caused by cycles at all (though it is the most likely cause) but could result from a global cache containing references to acquisition wrapped objects. Such a cache should never exist -- global variables and class attributes must never directly or indirectly reference persistent objects -- as this results in nasty persistency bugs. Nevertheless, sometimes software is buggy. I noticed such a cache in Archetypes 1.3a2 (fixed in newer versions). .... >Said dictionary does not seem to be referred to by anything, at least not >anything the gc module can find (gc.get_referrers()). That's probably because "RequestContainer" is an ExtensionClass and ExtensionClasses are not yet (only from Zope 2.8 on) GC aware. > ... >Also I >understand zope.org suffers from this? :) Leaks can have innumerable causes. It is by no means clear that your leak and the one on Zope.org has the same cause. >Based on this information also, I have no reason to believe this is caused >by product code ... I have to guess it's a Zope bug? I do not observe leaks in my Zope installation -- even when I make an error stress test. This does not mean it could not be a Zope bug -- but it implies that such a bug does not surface regularly, at most under special conditions. >In some ways this is motivating me to reduce the number of errors that could >occur, but because ALL errors seems to cause the problem, I ultimately have >no control over it and under certain circumstances I suffer heavily from >this. 404's alone probably cause this (I'm plannign to test that >explicitely as well). It does not here.... Something is special with your installation -- maybe "Localizer" as Tres suggested, maybe another misbehaving product. -- Dieter

1 0

Re: [Zope3-dev] Re: RFC: TALES adapters and TAL/Tales variable namespaces
by Jim Fulton 19 May '04

19 May '04

Casey Duncan wrote: > On Wed, 19 May 2004 16:00:43 -0400 > Jim Fulton <jim(a)zope.com> wrote: > > >>I've posted two proposals: >> >> http://dev.zope.org/Zope3/TALESPathExpressionAdapters > > > I have to say I am not found of "*", I'm not attached to '*'. Feel free to suggest alternatives (other than ':', '/', '|', or '?'. Is that all ;) > but using the parenthesis to "cast" > the variable seems fairly natural and isn't colored by completelt > different meaning in other languages. > > So the example: > tal:content="x/y*foo.bar.baz/z" > > would be: > tal:content="x/(foo.bar.baz)y/z" Yup > Which seems reasonable. The dotted notation seems ok, but it implies > that this notation is recognized in general in path expressions which is > confusing because it isn't. If we didn't use dots then it might look > like: > tal:content="x/(modules/foo/bar/baz)y/z" I'm not sure what you mean here. If the thing in the parens was a path expression, it would be: x/(modules/foo.bar/baz)y/z IOW, modules excepts dotted names. > If the thing in the parenthesis is just another path expression, that > mitigates the need for namespaces IMO. The above could then become: > tal:define="baz modules/foo/bar/baz" > tal:content="x/(baz)y/z" True, but then, the obvious syntax would be: x/baz(y)/z which is the top of a slippery slope. :) It would also make it harder to provide predefined adapter names. We'd like to be able to define some adapters (e.g. 'zope', 'format', etc.) in ZCML and let people just use them in ZPT without having to use defines. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

2 1