September 2003 - Zope-Dev

More Beginner doubts
by Emiliano Marmonti 18 Sep '03

18 Sep '03

Thanks Andrew, I'll try it. Now I'm testing inside Zope and everything looks well, except for: self.cat.addIndex('id','FieldIndex' ) self.cat.addIndex('name' ,'FieldIndex' ) self.cat.addIndex('observaciones','FieldIndex' ) get_transaction().commit() Zope is telling me that from this version (2.6) I could not addIndex here, must be done outside(where?), in the Application (that I haven't). My problem is that I create the catalog inside the class class Authors: def __init__: .... if self.root.has_key( 'cat' ): self.cat= self.root['cat'] else: self.cat = Catalog() self.cat.aq_parent= self.root self.root['cat']= self.cat ### Here come the index I could not imagine where (and how) could make the indexes. I have seen a lot of documentation that makes references to addIndex... Thanks a lot again Emiliano.

2 1

Zope 2.7 final?
by Andrew Veitch 18 Sep '03

18 Sep '03

Just wondering what the latest on Zope 2.7 is, it's been nearly a month since the last release. We're looking forward to it, there's some changes that will help us a lot but we don't want to deploy on a beta. Andrew -- Logical Progression Ltd, 20 Forth Street, Edinburgh EH1 3LH, UK Tel: +44 (0)131 550 3733 Web: http://www.logicalprogression.net/

2 1

Will restrictedTraverse() load every obj into memory?
by zhimin＠iss.nus.edu.sg 18 Sep '03

18 Sep '03

Hello, I have a function that returns the object when given a path. def path2Object(path): obj = context,restrictedTraverse(path) return obj My concern is whether restrictedTraverse(path) will load into memory every object it traverses through and thus increase memory usage, especially if it is run very frequently and with different path each time. e.g. if my path is '/x/y/z/myobj', will object x, y, and z also get loaded when the object I really want is myobj? What's the state of x,y,z after calling restrictedTraverse(path) - loaded or ghost? If intermediate objects (like x, y, z above) get loaded, is there anyway to ghostify them to reduce memory usage? ...or is there a memory-economic way to get object by path? Many thanks! cheers, Zhi Min

2 1

Beginner doubts
by Emiliano Marmonti 18 Sep '03

18 Sep '03

Hello all I'm newbie with Zope, ZODB and so on. First of all I have choiced Zope as application server and ZODB because I have to implement a bibliographical project that handles a lot of hierarchichal data, with need and will add in few months all the workflow related to send documents to a reviewer, publish, and have to connect with other bibliographic system through XML because of this is that I think could be made very easily with Zope/Plone, isn't it? The first thing that I've tried to implement is to add functionality for search/add authors. For the business logic I have made an external-script that have the abstractions and the persistent-logic. After all the classes code I have defined two functions for invoking inside Zope. The entire idea is to put inside a Persistent class something that adds the functionality to search over the text of the objects properties. I'have found in this list an example that uses a Catalog, I think must be another way to search over a ZODB but have not found any example. For me a catalog is OK, perhaps later will want a TextIndex, but: ------- code ------ import sys sys.path.append('c:\\Archivos de programa\\Zope\\lib\\python\\') import ZODB import Persistence from Persistence import Persistent from ZODB.FileStorage import FileStorage from Products.ZCatalog.Catalog import Catalog class Autor(Persistent): id='' name='' observaciones='' def __init__(self,id,nombre,comments): self.name = nombre self.id = id self.observaciones = comments def setName(self,nombre): self.name = nombre def setId(self,id): self.id = id def setObs(self,comments): self.observaciones = comments def getName(self): return self.name def getid(self): return self.id class Autores: def __init__( self, file='dcAuthor.fs' ): self.file= file self.db = ZODB.DB( FileStorage( file ) ) self.co = self.db.open() self.root= self.co.root() if self.root.has_key( 'cat' ): self.cat= self.root['cat'] else: self.cat = Catalog() ## I don't understand what is doing here self.cat.aq_parent= self.root self.root['cat']= self.cat self.cat.addIndex('id','FieldIndex' ) self.cat.addIndex('name' ,'FieldIndex' ) self.cat.addIndex('observaciones','FieldIndex' ) get_transaction().commit() def setAutor( self, autor ): uid= id( autor ) self.root[uid]= autor self.cat.catalogObject( autor, uid ) get_transaction().commit() def searchAutor( self, **kw ): r= self.cat.searchResults( kw ) paths= self.cat.paths root = self.root k= [] for i in r: id= i.data_record_id_ k.append( root[paths[id]] ) return k def AddAutor(nombre,obs,documento): autores = Autores() autor = Autor(documento,nombre,obs) autores.setAutor(autor) def GetAutor(expresion,lista): autores = Autores() lista=autores.searchAutor(expresion) ------- end of code ------ I've tried this code with python. After waiting, python said: ------ python -------- >>> import dcAuthorDefClass Traceback (most recent call last): File "<stdin>", line 1, in ? File "dcAuthorDefClass.py", line 7, in ? from Products.ZCatalog.Catalog import Catalog File "C:\Archivos de programa\Zope\lib\python\Products\ZCatalog\__init__.py", line 16, in ? import ZCatalog, Catalog, CatalogAwareness, CatalogPathAwareness, ZClasses File "C:\Archivos de programa\Zope\lib\python\ZClasses\__init__.py", line 20, in ? import ZClass File "C:\Archivos de programa\Zope\lib\python\ZClasses\ZClass.py", line 16, in ? import Method, Basic, Property, AccessControl.Role, re File "C:\Archivos de programa\Zope\lib\python\ZClasses\Method.py", line 18, in ? import App.Dialogs, ZClasses, App.Factory, App.Product, App.ProductRegistry File "C:\Archivos de programa\Zope\lib\python\App\Factory.py", line 19, in ? import Products, Product File "C:\Archivos de programa\Zope\lib\python\App\Product.py", line 44, in ? from Factory import Factory ImportError: cannot import name Factory ------ end of python -------- My problem is that if I could not use a Catalog, what will I use? I could put a set of objects into a file and recovering by the id, but could not realize the way to search into the collection of objects and obtaining a filtered set or list. Another basic question. My first approach was to develop somenthing external to Zope to put my abstractions and drive the file storage, is this correct? Is there another way to map for example an abstraction with a folder, another abstraction with a file and so on, and for search using an standard catalog for example? My last question. Supposing that anybody could help me to make the above code work, I have a dtml-problem, how could I invoke an external method with parameters? I mean, I wish to call GetAutor with an expression like (name='a*') from dtml and after that iterate over the list that in some way, has returned from the external method to the dtml. Well, thanks a lot for your help and sorry by the extremely long of this mail Emiliano.

2 1

RE: [Zope-dev] Etag support in page templates
by Bjorn Stabell 17 Sep '03

17 Sep '03

Dieter wrote: > Please read the HTTP 1.1 spec... > > Caching requires either an "ETag" or "Last-Modified" header. > For good reasons... That would explain why it never got "fixed", but that's not how I understand the RFC: http://www.w3.org/Protocols/rfc2616/rfc2616.html http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.21 http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13.2 Indicates that "Expires" is enough to make content cacheable, and that it should be considered "fresh" until its expire time unless clients or servers use cache-control headers to force a revalidation. Last-Modified and Etag are used to validate the content after it expires using If-Modified-Since and If-None-Match respectively. Is there a thread discussing this already somewhere? [ Btw, Zope does return an empty Etag header; does that cause any confusion? ] Bye, -- Bjorn

7 10

Communication between Zserver and ZPublisher
by Alexander Schad 15 Sep '03

15 Sep '03

Hi all, I'd like to understand how the communication between the ZServer and the ZODB works. As far as i know the ZServer (Web server) talks to the ZPublisher (ORB) to get the Requested Object. But do they communicate via tcpip sockets or unix sockets? I have a problem with hanging connections (for more than 10000s) on the debug screen in zopes control panel and when all connections are jammed zope freezes. I need to find out where the connections gets stuck and why there is no timeout (i even tried to hack in the timeoutsocket module without any effect). I think i need to debug this on the os level to find out what these blocked connections are doing. But since i'm no unix system administrator i really don't know where i should start my investigation. thanks, Alexander Schad ---------------------------------------------------------------------------------------- Alexander G. Schad (Project Manager) beehive KG ---------------------------------------------------------------------------------------- mailto: ags(a)beehive-eu.com http://www.beehive-eu.com

4 9

Very severe memory leak
by Leonardo Rochael Almeida 15 Sep '03

15 Sep '03

Hi, For a long time now, one of our clients, running Zope 2.5.1, has been experiencing memory leaks. For a time this has been relieved by restarting Zope every day at 4am. Lately this was not enough, however, as Zope started taking more and more memory, to the point that it frequently required more than one restart during the day. The machine this is running on is a Xeon 500 MHz with 512 Mb of memory. Zope is a very memory intensive application, but 512Mb used to be enough for a medium sized portal. This Zope site makes very heavy use of both ZClasses and ZCatalog, and it appears to me that this "worsening" of perfomance and memory consumption was being caused by the increase in content, which caused an increase in the size of the ZCatalog and in the number of ZClass instances. A total of 6796 ZClass instances representing News Items are cataloged. There are other ZCatalogs and ZClass instances representing calendar events and other stuff (this is a very old Zope portal coded by hand in ZClasses, no CMF, no Plone, not even Pagina1, our ZClass based CMS) Fiddling with the cache parameters in the control panel showed that while we could keep the memory consumption to a point where the daily restart would be enough (say, with 10k objects per thread), we would get constant thrashing of cache objects, specially DateTime objects due to the ZCatalog queries, and the machine performance would be close to intolerable, whereas if the cache parameters where set to allow a fast performance (with 50k objects per thread) the machine would run out of memory in 3 to 4 hours. Needless to say this was with heavy use of RAMCacheManagers, not counting the accelerator proxy in front of it. Without the RAMCaches, the machine would go down in under 5 minutes of work hour load. Even with the caches on, the load would never go down from 2.0 during work hours. Last tuesday we decided no longer to wait for 2.6.2 and migrated the site to 2.6.1. We dealt with the ObjectManager-based-ZClass issue, reformed the ZCatalogs to replace the DateTime FieldIndexes with DateTimeIndexes and then had a working testing environment, which we stress tested lightly without detecting any problems and quickly moved to production. This was late at night The next morning we were surprised to notice the machine very quickly ran out of memory. The memory leak was *far more severe* than before. Zope needed a restart every 15m or so before it would send the machine into heavy swaping. In a very non-intuitive hunch I suggested we shut down all RAMCaches and, amazingly enough, this made the situation a bit more manageable. We're now restarting every 45 minutes. To our relief, disabling the RAMCaches had only a barely noticeable effect on performance. The site kept churning out pages really fast, a testatment to the optimization job done in the 2.6 series. The load on the machine is rarely above 0.8, except when it goes into swap :-) The number of DateTime refcounts in the Control_Panel, although much smaller than in Zope 2.5.1 is very high and, mostly importantly, constantly increasing, as far as I can tell. After 12 minutes of uptime, the top refcounts are: DateTime.DateTime.DateTime: 96168 BTrees._IOBTree.IOBucket: 43085 BTrees._IIBTree.IIBTree: 40400 BTrees._IIBTree.IIBucket: 23696 OFS.DTMLDocument.DTMLDocument: 23190 BTrees.OIBTree.OIBucket: 14582 BTrees._IIBTree.IISet: 12479 BTrees._IIBTree.IITreeSet: 10823 BTrees.OOBTree.OOBucket: 7088 OFS.Image.Image: 6860 OFS.DTMLMethod.DTMLMethod: 5894 DocumentTemplate.DT_Util.Eval: 3250 OFS.Image.File: 2796 BTrees._IOBTree.IOBTree: 2761 ZClasses.Method.MWp: 1592 In time, DateTime refcounts eventually dwarves the second place by an order of magnitude. I think this is related to the fact that DateTime instances are stored as metadata, even though the date indexes have been converted to DateTime indexes. The question is, why aren't those instances being released? What is holding on to them? I tried installing the LeakFinder product but discovered it didn't work before stumbling in a message in the archives that told me exactly that :-) The RefCounts view in the LeakFinder object fails with the following traceback: Traceback (innermost last): [...] Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: REQUEST Module <string>, line 0, in ? Module Products.LeakFinder.LeakFinder, line 240, in manage_getSample Module Products.LeakFinder.LeakFinder, line 163, in getControlledRefcounts Module Products.LeakFinder.LeakFinder, line 188, in resetCache TypeError: function takes at most 2 arguments (3 given) The code in question is: def resetCache(c, gc): cache = c._cache if gc: cache.minimize(3) # The minimum that actually performs gc. # Replace it with an empty cache. c._cache = PickleCache(c, cache.cache_size, cache.cache_age) PickleCache is imported from ZODB.cPickleCache, which is (surprise!) in C and I couldn't figure out how I should fix the call above to work. In an ExternalMethod, I consulted the python module gc and checked that gc *was* enabled and *no* uncollectable cycles where found. In trying to narrow down the causes of the leak, we detected what looks like a minor one. If you insert a very simple dtml method in Zope root and bang on it at a rate of, say, 100 requests per second, Zope will increase it's memory footprint very slowly, say, a couple hundred Ks every 10 minutes (both RSS and virtual size). I figure this has something to do with memory fragmentation or some other low level stuff. But this is definetly not our main concern, which is to find out how is it that Zope is leaking DateTimes so heavily in our site, and if it is really all those DateTimes that are hogging our memory. We need help, and we need it desperately. If anyone wants any other information we'd be happy to provide. -- Leonardo Rochael Almeida <leo(a)hiper.com.br> Hiperlogica informatica e educacao ltda.

13 38

helper script for permission renames ?
by Simon Michael 14 Sep '03

14 Sep '03

Upgrading a product whose permissions have been renamed can be a pain.. you lose your permissions settings and have to recreate them. Has anyone seen a script that would walk the zodb and replicate the old permission settings under the new name ?

2 1

[Summary] Caching prob with AHCM and headers
by Bjorn Stabell 13 Sep '03

13 Sep '03

Just for the record to help others in the same situation... PROBLEM Apache (1.3.28 and 2.0.47) will not cache content with just an Expires header, but it will correctly cache content with a Last-Modified header. This problem has been reported before in 1999 by James Cooper, but the bug request seems to have been forgotten/ left behind. It makes sense for dynamic content to produce Expires headers, not Last-Modified headers, so this is a problem. Looking at the code for Zope, it seems that the standard HTTP accelerator product used to produce a Last-Modified header, but no longer does so, thus making it useless with the current versions of Apache. The problem has been re-reported to the Apache group with patches: http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23129 (for Apache 1.3.28) http://nagoya.apache.org/bugzilla/show_bug.cgi?id=23130 (for Apache 2.0.47) WORK-AROUNDS 1) Produce a Last-Modified header yourself by using 2) Patch Zope's standard HTTP accelerator product so that it produces a Last-Modified header (the line is commented out now) SOLUTIONS 3) Patch Apache (see bug report links above for patches) 4) Wait until the problem is hopefully fixed in Apache Bye, -- Bjorn Stabell bjorn(a)exoweb.net Tel +86 (10) 65918490

2 1

Etag support in page templates
by Bjorn Stabell 13 Sep '03

13 Sep '03

Janko wrote: > I think, this is one problem with the current code. There is > no defined way to decide, when something is last modified. > With CMF-based sites there is a clearly defined property for > this. On the other hand all objects have at least a > bobobase_modification_time, but this one changes with every > ZODB action on the object. > > I'm not sure what to do. Also, in most cases, the content of pages doesn't all come from one object, so Last-Modfied doesn't really make sense since it should refer to the whole page. I think the problem is with Apache's inability to cache content with Expires set, and that Apache should be fixed. ETAG NOTES As for Zope, I'm not sure if setting and empty Etag header is the right thing to do if no Etag was provided...? I've been thinking about how Zope could be extended to support HTTP validators on Etags. How about this (Page Templates = ZPT or DTML): - Page Templates are extended with a customizable Etag function/ property. Could be part of the cachable interface. You should set this to return a different etag if, and only if, the content of the page was different. - Page Templates are extended to support If-Match and other such conditionals by calling the customized Etag function and returning 304 Not Modified if the Etag conditions are met (e.g., the calculated etag is the same as the browser/cache provided etag). - Upon rendering the top most Page Template the Etag is automatically calculated and included in the HTTP Headers. If all objects have an etag (Zope seems to be going in that direction looking at the WebDAV code), then a simple way to calculate the etag for a page would be the hash of all the etags for all of the content of that page added together (string concats). Another way could be to let the etag change every X seconds (to simulate the current expires caching model). Bye, -- Bjorn

2 1