January 2005 - Zope-Dev

Event & Timer Service for Zope 2.8
by Andrew Veitch 26 Jul '05

26 Jul '05

I noticed there was a thread last month on an event service for Zope. Did that reach any conclusion? From what I can see the present products are: - http://www.zope.org/Members/lstaffor/Xron - Zope 2.6 only, not being developed - http://zope.org/Members/naumen/NauScheduler - shipped with one of the Zope CMS's works fine with 2.7 - http://dev.legco.biz/products/ZopeScheduler and http://dev.legco.biz/products/timerservice - works fine with 2.7 too - http://cvs.zope.org/Products/Event/ - event service without a time service, not sure of development status - http://mail.zope.org/pipermail/zope-cmf/2005-January/021747.html - Dieter's project to be based on Python's sched module - http://api.nuxeo.org/CPS3/public/ CPSCore.EventServiceTool.EventServiceTool-class.html - CPS's event service - http://cvs.sourceforge.net/viewcvs.py/*checkout*/mailmanager/ mailmanager/MailCheckThread.py?rev=1.15 - our custom scheduler for MailManager - Zope X3's event service It would be great to some consolidation of all of this into the core. I would also strongly favour a timer service along with the event service. I know that the argument has always been that operating systems provide time services but this is not very convenient for cross platform development and on some Windows platforms it's difficult. A -- Logical Progression Ltd, 20 Forth Street, Edinburgh EH1 3LH, UK Tel: +44 (0)131 550 3733 Web: http://www.logicalprogression.net/

5 14

Patch for attribute permisions problems in Zope 2.7.3
by Santi Camps 21 Feb '05

21 Feb '05

Hi all again, We have been written last week about some attribute permission problems with Zope 2.7.3 beta due to a patch applied by Tres. First of all, Tres, apologies for my too fast written test case and my too late test of Zope 2.7.3. Now, with some more time, I've tested and debugged on Zope 2.7.3 and found exactly what's happen. Supose we have a structure of objects like this: A.__of__(B) "A" inherits from Acquisition.Implicit, has security assertions, but has not __allow_access_to_unprotected_subojects__ We want to access, from a Zope Page Template, an attribute of "B" that is not present in "A" Accessing B.our_attribute attribute works fine. But accessing A.__of__(B).our_attribute fails, and should work. The problem is the call to "validate" done in "guarded_getattr" method of ImplPython.py. The actual call is "if validate(inst, inst, name, v)", but the validate function says: Arguments: accessed -- the object that was being accessed container -- the object the value was found in name -- The name used to access the value value -- The value retrieved though the access. roles -- The roles of the object if already known. Now, "accessed" and "container" are always the same, and in some cases should be different. I attach a patch to solve this case that works for me. I'm not sure if my code is the best way to solve the problem but, as I said, it seems to work fine. Of course, If the patch is accepted, the same change should be done in the C version. Thanks Santi Camps http://www.earcon.com --- ImplPython.py 2004-08-07 19:15:48.000000000 +0200 +++ /usr/local/zope273/lib/python/AccessControl/ImplPython.py 2004-10-29 10:56:11.000000000 +0200 @@ -534,6 +534,12 @@ # exceptions are caught early. try: v = getattr(inst, name) + container = inst + while hasattr(container,'aq_explicit') and \ + not(hasattr(container.aq_explicit, name)) and \ + hasattr(container, 'aq_parent'): + # Find real container when attribute is acquired + container = container.aq_parent except AttributeError: if default is not _marker: return default @@ -551,6 +557,6 @@ return v validate = SecurityManagement.getSecurityManager().validate - if validate(inst, inst, name, v): + if validate(inst, container, name, v): return v raise Unauthorized, name

2 2

Removal of aq_acquire from guarded_getattr
by Stefan H. Holek 07 Feb '05

07 Feb '05

The bug: http://zope.org/Collectors/CMF/259 The fix: http://mail.zope.org/pipermail/zope-checkins/2004-August/028152.html This effectively changes how acquisition works in restricted Python. I understand this may well be the point <wink>. The consequences: Zope sites experiencing seemingly random Unauthorized errors. [1] I have added tests to the AccessControl suite on 2.7 branch that demonstrate the new behavior. Note that all of them pass in Zope 2.7.2. What it _appears_ to mean is that when a container denies access, the object security of the acquiree is checked. Therefore, a potential acquiree (read: _any_ object) must make sure to declareObjectProtected or it may end up not being acquirable. This is not always the case in current Zope/CMF/Plone which would explain the Unauthorized errors we see. Tres, I am happy to discuss this further once you had a look at the tests. I also have tests for the CMF in case you want them. Stefan [1] http://zope.org/Collectors/CMF/318 http://zope.org/Collectors/Zope/1654 http://zope.org/Collectors/Zope/1669 http://plone.org/collector/3682 -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/

2 3

Renamed the Zope package to Zope2 and including Zope 3 packages in Zope 2.8
by Jim Fulton 03 Feb '05

03 Feb '05

I've just checked in a change on the Zope 2 trunk for Zope 2.8 that renames the Zope package to Zope2. Importing the Zope module is now deprecated, but will be supported until Zope 2.11. When using the Zope head or Zope 2.8, you should import Zope2 instead. Originally, I had intended not to include any Zope 3 packages until Zope 2.9, however, Zope 2.8 has been delayed long enough that I think it makes sense to include some parts of Zope 3 sooner. I also want to use some of the Zope 3 persistent code support, which depends on zope.interface to help get ZClasses working again. I haven't decided which parts of Zope 3 should be included in Zope 2.8 and would like to get input. If you have suggestions on what to include or exclude, please respond here or on the z3-file list, where I am also posting this message. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

12 37

ZCatalog and Zope 2.8
by Andreas Jung 01 Feb '05

01 Feb '05

I changed the ZCatalog and indexes codebase for Zope 2.8 in the following way: - ZCatalog and indexes derived from UnIndex used a nasty implementation of __len__ which lead to problems in Zope 2.8 with new-style classes. The corresponding code has been cleaned up. For ZCatalog instances there is an implicit migration built-in for the __len__ attribute. For all indexes I added a manage_convertIndexes() method that re-creates and reindexes all indexes of a given ZCatalog instance. Outstanding issues: - some indexes show the number of indexed objects vs the number of indexes terms within the ZMI which is totally inconsistent. I think the ZMI should show the number of indexed objects. Index specific information e.g. the number of indexed terms should be shown within the indexes default view (if necessary). Objections? - Indexes derived from UnIndex also store informations about objects although they do not index useful information. E..g. if you have one thousand objects and a keyword index indexing only two objects (because these two objects have the required property or method returning indexable values) than you have the RIDs of one thousand objects within the internal data structure of the index....a total waste of space. An optimised version of UnIndex would store only values evaluating to non-zero. Such an optimisation is already used in Dieter's Managablexes. This might change the behaviour of some applications but I am not completely sure about this issue. Thoughts? Andreas

2 4

January Bug Day Roundup
by Chris Withers 28 Jan '05

28 Jan '05

Hi All, Well, it was certainly a quiet one. Both Andreas and myself were busy and things felt very quiet indeed... One 3 issues resolved, but lots of work done by Tres on some nasties. Special mention must go to brentmh/Crouton who was brave enough to try and solve a bug in dtml's sequence stuff: http://collector.zope.org/Zope/456 Brave man! Anyway, looks like not-having-it-on-a-Friday was major flop, even the one person who always asks for it not to be on a Friday wasn't there, so we'll see you all again, last Friday of February! cheers, Chris PS: Here's the lowdown: Resolved: #533 - ZMI "find" doesnt find in File with text #1469 - win32 install 2.7.2 XP SP1 #1648 - bug with medusa FTP Wontfix: #1331 - Most Frequently used product selection #1418 - Hotfix for TAL issue has no version check Rejected: #1422 - AssertionError after installing Zope Hotfix Release, 2004/07/13 #1485 - Re-ocurrence of Hotfix_2002-03-01/security_alert #1555 - more flexible logger config (duplicate) #1578 - Mounting a non-folder object fails #1670 - No log out button in Zope ..and here's the "player stats" :-) Resolved Assists Wontfix Rejected Deferred shh 1 ajung 1 chrisw 1 1 1 5 tseaver 3 Tiran 1 Crouton 1 -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

1 0

Investigating a reference leak...
by Ben Last (ZopeDev) 28 Jan '05

28 Jan '05

Dear Zopistas I sent this to the Zope list and it was suggested that I post it here; hope that's ok. I've been trying to track down what's either a memory or a reference leak in a Zope 2.7.3 (Python 2.3.4) system. The symptoms are that after several days of normal running on the agressively proxy-cached production site or a few hours on the development site under simulated load, the number of OFS.Image.Image references reaches very high levels, such as 17761. In the cache at that time are just 4345 images. There are only 1332 images on the whole site, so assuming (as I understand it) that each of the 5 Zope thread has its own cache, there should be a maximum of (1332 * threads) = 6660 Images. The site never creates new Images, it just loads existing ones. In a bid to see where these references are being created, I installed LeakFinder, but that doesn't help me because Images are persistent objects, and thus their __init__ methods are not called (LeakFinder appears to patch just the __init__ and __del__ methods for tracking). Thus I've been on a quest to find a place to stick a patch that will show me, for a persistent object, from where a reference to it is being created. I've tried __setstate__, and overridden it for OFS.Image.Image to print a traceback to stdout. Then I run the development Zope site with bin/runzope and watch. This shows me Images being created in the expected places, but not in quantities that would explain those huge refcounts. So; to questions: 1) Is there a better place than __setcache__ to identify where references to Images are being created? I don't fully understand the way in which persistent objects are actually created and populated. 2) Our site is very dependent on ExternalMethods (Zope provides the visual layout, business logic is in ExternalMethods, all from one module). In some of those methods, the code loads Images in order to look up information about them - this is because we have some particular standards for things like Image tags which the built-in tag() method can't support, so we have to build our own. Thus there's code like: img = getattr(self.Images,'image.gif') #all images live in /Images, got by acquisition from self buildTag(img.title,img.width,img.height) #use image attributes to build fuller tag del img #avoid leaving references around (this is paranoid!) Is there any reason that this is bad practise? I see __setstate__ for images being called from this code and then again shortly afterwards when the Image is served up to the browser. I'd be very grateful for any help on this! Regards to all Ben Ben Last Technical Director SleepyDog

1 1

RE: [Zope-dev] Re: Log rotation on Windows
by Mark Hammond 25 Jan '05

25 Jan '05

[Duncan] > +1 on fixing the problem of log rotation on windows. > > I don't like a solution that requires you to wait an > arbitrary period for > the snapshot to appear. Yes, I see your concern. Assuming we stick to a rename operation, the external process could simply wait for the expected file to appear (using, say, FindFirstChangeNotification or ReadDirectoryChanges). Even a loop with 1 second sleeps before checking the file should work. > Especially given that windows has a > tendency to > swap processes out to disc just because they haven't done anything > recently. This means that if you run the logrotate at > midnight you might > easily spend the 5 seconds just swapping Zope back into > memory. Given that > you don't have a system 'kill' command so you'll need to run > something zope > specific to generate the equivalent, you might as well make > it two-way to > begin with and signal when the snapshot has been created. Certainly that is doable, and it touches on a subject I (indirectly) brought up here quite some time ago - a general notification mechanism so Zope can let its "parent" know its status (eg, "starting", "running", etc). If we can get a more generic notification scheme off the ground, it may make sense to reuse that. OTOH, it adds a level complexity where a slightly smarter "wait for file to appear" loop in the external process may be all that is needed. > I guess that the most obvious Windows equivalent of the kill > command would > be to use win32event.CreateEvent() to create > a named event. That can then be easily signalled from outside > zope, but a thread inside zope would have > to explicitly wait on the event (unless > RegisterWaitForSingleObject could > be used, but it isn't currently exposed in win32event). Yep - that is exactly what I was thinking. > Adding a thread to handle windows events to Zope could also > be used to improve the communication when run as a service. > Currently the zope process is terminated using > TerminateProcess and it would be nicer if the service > could generate a termination signal to give Zope a chance to shutdown > cleanly and only resort to TerminateProcess if the clean shutdown was > ignored. Collector issues 1527 and 1533 provide startup/shutdown for Windows as it exists on Linux, hopefully making that moot. As mentioned above, it has been identified that Zope really could do with a generic notification mechanism to improve the situation on all platforms. I believe that if we keep the task small (ie, just log rotation), we can come up with a fairly minimal patch which does the job perfectly. I'm concerned that if we try and expand it into any kind of generic mechanism, it simply will not happen. I note that the only comments you made were on the mechanics of the operation - which is good! I expected the scheme itself (ie, "rename to .snapshot if possible") to be contentious. Assuming no other serious objections come up, we will steam ahead! Thanks, Mark.

1 0

Log rotation on Windows
by Mark Hammond 25 Jan '05

25 Jan '05

Hi all, A number of Enfold's customers have requested a reasonable logfile rotation scheme for Zope on Windows. Enfold would like to work on this and contribute it back to Zope. The intention of this mail is to find a consensus on the general solution we should adopt, so we can supply patches with the greatest chance of getting into the core. Since version 2.7, Zope has moved to a logging model similar to that used by most Linux servers. This means Zope can rely on external tools for all rotation, making the Zope implementation very simple - all it need to do it re-open the logfile upon request. Unfortunately, this model relies on the operating system being able to rename a file while it is in use, which is not true for Windows. There are a number of solutions to this problem. In the interests of brevity, I have briefly listed a few alternatives and why they suck, then my preferred solution is covered in more detail. * Move to a Zope 2.6 model, where Zope itself manages all rotation logic. This has the obvious drawback of requiring all supporting code exist in Zope just for Windows, and all the reasons Zope dropped it in the first place. * Use a default log name based on the date and rotation schedule. For example, assuming a rotation schedule of "weekly", the logfile name would include the date of the start of the current week. This is similar to the scheme used by Microsoft IIS. Drawbacks include the need to wake at midnight to check if the name of the logfile has changed, that the scheme leaves lots of old files around, and that logging on Windows is significantly different than for other operating systems. * Do nothing. The drawback of this is that it leaves Zope on Windows a second-class citizen, and with administrators unable to control their logs. * Point users at the win32 "event log" logger. This is not pratical when the number of events being logged is high, so is probably only suitable as a special handler for only ERROR events, for example. My preferred model is the following alternative: * Have the Zope logfile handler make a "snapshot" of the log, and like Zope on Linux, rely on external tools for all logfile rotation semantics. When making a "snapshot", Zope simply closes the existing logfile, renames it to a well-known name (eg, "{logname}.snapshot", and then re-opens the original logfile name (which should no longer exist as it was just renamed). If the rename fails (ie, the .snapshot file already exists), Zope simply ignores the error and continues to append to the existing log-file. This now means that "{logname}.snapshot" can be rotated by an external tool. For demonstration purposes, the following script should offer a simple rotation tool for Windows: % kill --signal=USR2 zope_pid # whatever the Windows equiv. is % sleep 5 # wait for the file to appear % mv -f event.log.1 event.log.2 % mv -f event.log.snapshot event.log.1 You will note that this is quite similar to a script that would work on Linux. The key differences are that on Windows we send the signal *before* the process, and that we make no attempt to touch the file currently being logged to, only the snapshot just made. Existing tools (eg logrotate) should be capable of rotating Zope (unfortunately logrotate doesn't seem to have a reasonable windows port available.) Note that for simplicity, the above assumes that the signal module will work for us on Windows. This is probably not true, but it should be possible to implement a win32 specific simulation. I've avoided this so we can focus on the log semantics. I welcome all comments on these schemes, or any others I have not listed. Thanks, Mark.

2 1

FYI: test.py changes rolled back
by Stefan H. Holek 24 Jan '05

24 Jan '05

I have restored test.py to the version shipping with Zope 2.7.3. The changes made to accommodate Florent's use-case (adjacent symlinks) broke some of mine. In particular, without calling realpath (or abspath) on libdir, trailing slashes caused no tests to be found, as did some combinations of '.' and '..'. [1] I am confident we can find a way to solve this, but wanted to avoid releasing a "challenged" test runner with Zope 2.7.4. Florent, in the meantime I am hoping you can run your tests by using a combination of --libdir and --dir flags. Sorry for the inconvenience. Stefan [1] The real issue in PathInit IMO is the return value of os.getcwd(), which is realpath'd one some systems and abspath'd on others... -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/

2 1