- Zope-Dev - Zope lists

Investigating a reference leak...
by Ben Last (ZopeDev) 28 Jan '05

28 Jan '05

Dear Zopistas I sent this to the Zope list and it was suggested that I post it here; hope that's ok. I've been trying to track down what's either a memory or a reference leak in a Zope 2.7.3 (Python 2.3.4) system. The symptoms are that after several days of normal running on the agressively proxy-cached production site or a few hours on the development site under simulated load, the number of OFS.Image.Image references reaches very high levels, such as 17761. In the cache at that time are just 4345 images. There are only 1332 images on the whole site, so assuming (as I understand it) that each of the 5 Zope thread has its own cache, there should be a maximum of (1332 * threads) = 6660 Images. The site never creates new Images, it just loads existing ones. In a bid to see where these references are being created, I installed LeakFinder, but that doesn't help me because Images are persistent objects, and thus their __init__ methods are not called (LeakFinder appears to patch just the __init__ and __del__ methods for tracking). Thus I've been on a quest to find a place to stick a patch that will show me, for a persistent object, from where a reference to it is being created. I've tried __setstate__, and overridden it for OFS.Image.Image to print a traceback to stdout. Then I run the development Zope site with bin/runzope and watch. This shows me Images being created in the expected places, but not in quantities that would explain those huge refcounts. So; to questions: 1) Is there a better place than __setcache__ to identify where references to Images are being created? I don't fully understand the way in which persistent objects are actually created and populated. 2) Our site is very dependent on ExternalMethods (Zope provides the visual layout, business logic is in ExternalMethods, all from one module). In some of those methods, the code loads Images in order to look up information about them - this is because we have some particular standards for things like Image tags which the built-in tag() method can't support, so we have to build our own. Thus there's code like: img = getattr(self.Images,'image.gif') #all images live in /Images, got by acquisition from self buildTag(img.title,img.width,img.height) #use image attributes to build fuller tag del img #avoid leaving references around (this is paranoid!) Is there any reason that this is bad practise? I see __setstate__ for images being called from this code and then again shortly afterwards when the Image is served up to the browser. I'd be very grateful for any help on this! Regards to all Ben Ben Last Technical Director SleepyDog

1 1

RE: [Zope-dev] Re: Log rotation on Windows
by Mark Hammond 25 Jan '05

25 Jan '05

[Duncan] > +1 on fixing the problem of log rotation on windows. > > I don't like a solution that requires you to wait an > arbitrary period for > the snapshot to appear. Yes, I see your concern. Assuming we stick to a rename operation, the external process could simply wait for the expected file to appear (using, say, FindFirstChangeNotification or ReadDirectoryChanges). Even a loop with 1 second sleeps before checking the file should work. > Especially given that windows has a > tendency to > swap processes out to disc just because they haven't done anything > recently. This means that if you run the logrotate at > midnight you might > easily spend the 5 seconds just swapping Zope back into > memory. Given that > you don't have a system 'kill' command so you'll need to run > something zope > specific to generate the equivalent, you might as well make > it two-way to > begin with and signal when the snapshot has been created. Certainly that is doable, and it touches on a subject I (indirectly) brought up here quite some time ago - a general notification mechanism so Zope can let its "parent" know its status (eg, "starting", "running", etc). If we can get a more generic notification scheme off the ground, it may make sense to reuse that. OTOH, it adds a level complexity where a slightly smarter "wait for file to appear" loop in the external process may be all that is needed. > I guess that the most obvious Windows equivalent of the kill > command would > be to use win32event.CreateEvent() to create > a named event. That can then be easily signalled from outside > zope, but a thread inside zope would have > to explicitly wait on the event (unless > RegisterWaitForSingleObject could > be used, but it isn't currently exposed in win32event). Yep - that is exactly what I was thinking. > Adding a thread to handle windows events to Zope could also > be used to improve the communication when run as a service. > Currently the zope process is terminated using > TerminateProcess and it would be nicer if the service > could generate a termination signal to give Zope a chance to shutdown > cleanly and only resort to TerminateProcess if the clean shutdown was > ignored. Collector issues 1527 and 1533 provide startup/shutdown for Windows as it exists on Linux, hopefully making that moot. As mentioned above, it has been identified that Zope really could do with a generic notification mechanism to improve the situation on all platforms. I believe that if we keep the task small (ie, just log rotation), we can come up with a fairly minimal patch which does the job perfectly. I'm concerned that if we try and expand it into any kind of generic mechanism, it simply will not happen. I note that the only comments you made were on the mechanics of the operation - which is good! I expected the scheme itself (ie, "rename to .snapshot if possible") to be contentious. Assuming no other serious objections come up, we will steam ahead! Thanks, Mark.

1 0

Log rotation on Windows
by Mark Hammond 25 Jan '05

25 Jan '05

Hi all, A number of Enfold's customers have requested a reasonable logfile rotation scheme for Zope on Windows. Enfold would like to work on this and contribute it back to Zope. The intention of this mail is to find a consensus on the general solution we should adopt, so we can supply patches with the greatest chance of getting into the core. Since version 2.7, Zope has moved to a logging model similar to that used by most Linux servers. This means Zope can rely on external tools for all rotation, making the Zope implementation very simple - all it need to do it re-open the logfile upon request. Unfortunately, this model relies on the operating system being able to rename a file while it is in use, which is not true for Windows. There are a number of solutions to this problem. In the interests of brevity, I have briefly listed a few alternatives and why they suck, then my preferred solution is covered in more detail. * Move to a Zope 2.6 model, where Zope itself manages all rotation logic. This has the obvious drawback of requiring all supporting code exist in Zope just for Windows, and all the reasons Zope dropped it in the first place. * Use a default log name based on the date and rotation schedule. For example, assuming a rotation schedule of "weekly", the logfile name would include the date of the start of the current week. This is similar to the scheme used by Microsoft IIS. Drawbacks include the need to wake at midnight to check if the name of the logfile has changed, that the scheme leaves lots of old files around, and that logging on Windows is significantly different than for other operating systems. * Do nothing. The drawback of this is that it leaves Zope on Windows a second-class citizen, and with administrators unable to control their logs. * Point users at the win32 "event log" logger. This is not pratical when the number of events being logged is high, so is probably only suitable as a special handler for only ERROR events, for example. My preferred model is the following alternative: * Have the Zope logfile handler make a "snapshot" of the log, and like Zope on Linux, rely on external tools for all logfile rotation semantics. When making a "snapshot", Zope simply closes the existing logfile, renames it to a well-known name (eg, "{logname}.snapshot", and then re-opens the original logfile name (which should no longer exist as it was just renamed). If the rename fails (ie, the .snapshot file already exists), Zope simply ignores the error and continues to append to the existing log-file. This now means that "{logname}.snapshot" can be rotated by an external tool. For demonstration purposes, the following script should offer a simple rotation tool for Windows: % kill --signal=USR2 zope_pid # whatever the Windows equiv. is % sleep 5 # wait for the file to appear % mv -f event.log.1 event.log.2 % mv -f event.log.snapshot event.log.1 You will note that this is quite similar to a script that would work on Linux. The key differences are that on Windows we send the signal *before* the process, and that we make no attempt to touch the file currently being logged to, only the snapshot just made. Existing tools (eg logrotate) should be capable of rotating Zope (unfortunately logrotate doesn't seem to have a reasonable windows port available.) Note that for simplicity, the above assumes that the signal module will work for us on Windows. This is probably not true, but it should be possible to implement a win32 specific simulation. I've avoided this so we can focus on the log semantics. I welcome all comments on these schemes, or any others I have not listed. Thanks, Mark.

2 1

FYI: test.py changes rolled back
by Stefan H. Holek 24 Jan '05

24 Jan '05

I have restored test.py to the version shipping with Zope 2.7.3. The changes made to accommodate Florent's use-case (adjacent symlinks) broke some of mine. In particular, without calling realpath (or abspath) on libdir, trailing slashes caused no tests to be found, as did some combinations of '.' and '..'. [1] I am confident we can find a way to solve this, but wanted to avoid releasing a "challenged" test runner with Zope 2.7.4. Florent, in the meantime I am hoping you can run your tests by using a combination of --libdir and --dir flags. Sorry for the inconvenience. Stefan [1] The real issue in PathInit IMO is the return value of os.getcwd(), which is realpath'd one some systems and abspath'd on others... -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/

2 1

accents in rst
by Thierry Benita 24 Jan '05

24 Jan '05

Hi, In order to get a full support of accents in rst in Zope 2.7.3 I try to add the translation of accents and special chars into entities. What is the good place for this piece of code ? BTW, where do we translate entities to chars in the rendering process ? Regards, Thierry.

2 2

Zope in Segmentation fault
by Fabrizio Reale 21 Jan '05

21 Jan '05

Hello, I've found a situation that send Zope (both 2.7.2 and 2.7.4) in Seg Fault. I have created an object which inherits ParentManagedSchema from ATSchemaManager and the last release of PloneArticle. How can I test what is wrong and give you feedback? Fabry Fabrizio Reale fabrizio.reale(a)redomino.com Redomino S.r.l. C.so Monte Grappa 90/b - 10145 Torino - Italy Tel: +39 011 19502871 - Fax: +39 011 19502871 http://www.redomino.com

2 1

Announce: January Zope Bug Day!
by Chris Withers 20 Jan '05

20 Jan '05

Hi All, Time to announce the 1st bug day of 2005! This time it's on a Thursday to see if that generates more or less interest... What? Zope Bug Day working from http://collector.zope.org/Zope When? Thursday 27th January - from when you wake up until when the day ends! Where? #zope-dev on irc.zope.org Come along and help if you've got a chance! Read this if you want to know how: http://dev.zope.org/CVS/BugDays cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

1 0

ZopeChinaPak
by Reilly, Joseph M 20 Jan '05

20 Jan '05

Hi Pan, Is the ZopeChinaPak safe to install on English systems? The reason I'm asking is that we package base Plone in a commercial product called BrightStor Document Manager(BDM) and we intend to sell BDM in China as well as the rest of the world. It would be easier for the BDM installer to always install the ZopeChinaPak regardless of whether we are installing on an English or Chinese OS. Otherwise we will need to ask the User if they intend to run BDM on a Chinese OS(or detect that it is a Chinese OS) and install the Pak only then. My co-worker Varun has done some initial testing with the Pak on an English system and it seems to run fine. He will look at the code but I wanted to get your opinion. Moreover, thanks for creating the ZopeChinaPak, we can certainly use it. Best Regards, Joe Joseph Reilly Computer Associates Director, Development tel: +1 631 342-2731 cell phone: +1 516 642-6385 joseph.reilly(a)ca.com

1 0

Extending SiteErrorLog
by Christian Theune 19 Jan '05

19 Jan '05

Hi, I'm starting to prepare an extension for the SiteErrorLog that will allow you to store the detailed view of an Exception as a file on the server (directory will be configurable from zope.conf). I'm adding this feature because we have consulting situations where it is easier to transport a file on the server to us, than from the workstation we are using. The "copy to error log" isn't exactly the same. Also it doesn't need to be done for *every* exception, but I like to take a "snapshot" of single pathological exceptions. If you have any comments to this, I'd be happy to include those in my thoughts. I'm starting to prepare that on a branch for Zope 2.8. Cheers, Christian -- gocept gmbh & co. kg - schalaunische str. 6 - 06366 koethen - germany www.gocept.com - ct(a)gocept.com - phone +49 3496 30 99 112 - fax +49 3496 30 99 118 - zope and plone consulting and development

2 4

November Bug Day Roundup & January Bug Day Announcement!
by Chris Withers 19 Jan '05

19 Jan '05

Hi All, It's a long time overdue, but here's the roundup. I suspect these all made it into 2.7.4, if anyone's curious :-) The great news is that lots of bugs were resolved, and it was especially good to see patches contributed by people outside the usual core of developers. Thanks to those people! The next bug day is going to be on Thursday 27th November as an experiment to see if those who complain about the Friday bug days can make it. Hope to see all on #zope-dev next Thursday! cheers, Chris Here's the lowdown: Resolved: #551 - ExtentionClass.h might conflict with other system versions #679 - ZPT: bogus close tag </link> should be removed #721 - ZPT escapes attributes when it shouldn't #808 - Site Access Rule not working with dtml method in 2.6.1 #881 - Error: The ZGadflyDA product must be installed #995 - zope/python seg fault with infinite __str__ recursion #1003 - Overlong HTTP headers handled badly #1219 - XML Export #1371 - Surface cgi.maxlen in zope.conf #1407 - WebDav displayname Title #1450 - ZODBTools Packaging problem #1478 - Unittesting improvements #1489 - Invalid cookie causes other cookies to silently disappear #1510 - Unicode broken for non text/* content-types #1565 - monitor server not started #1593 - OFS/CopySupport _get_id() is buggy #1599 - ZSQL sqltest and unicode Wontfix: #922 - Version feature is broken in Zope #930 - Problems with Zope 2.6 config of sys.path on Windows #1461 - SCRIPT_NAME empty #1546 - Structured text links and IPv6 addresses Rejected: #576 - Importing ZClass fails #764 - ZCTextIndex is CMF incompatible #941 - Emergency user can own objects in Zope 2.7 #971 - Version only saves the last change #1023 - Tutorial import problem #1252 - VIRTUAL_URL #1512 - 2.8 failure with Plone: DocumentTemplate.ustr.py #1547 - import from one instance to another Deferred #1469 - win32 install 2.7.2 XP SP1 ..and here's the "player stats" :-) Resolved Assists Wontfix Rejected Deferred chrisw 6 1 5 1 ajung 6 1 2 1 efge 3 2 tseaver 1 regebro 1 ctheune 1 dpollard 1 sirilyan 1 jajcus 1 shh 1 sabaini 1 -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

2 2