August 2003 - Zope-Dev

Very severe memory leak
by Leonardo Rochael Almeida 15 Sep '03

15 Sep '03

Hi, For a long time now, one of our clients, running Zope 2.5.1, has been experiencing memory leaks. For a time this has been relieved by restarting Zope every day at 4am. Lately this was not enough, however, as Zope started taking more and more memory, to the point that it frequently required more than one restart during the day. The machine this is running on is a Xeon 500 MHz with 512 Mb of memory. Zope is a very memory intensive application, but 512Mb used to be enough for a medium sized portal. This Zope site makes very heavy use of both ZClasses and ZCatalog, and it appears to me that this "worsening" of perfomance and memory consumption was being caused by the increase in content, which caused an increase in the size of the ZCatalog and in the number of ZClass instances. A total of 6796 ZClass instances representing News Items are cataloged. There are other ZCatalogs and ZClass instances representing calendar events and other stuff (this is a very old Zope portal coded by hand in ZClasses, no CMF, no Plone, not even Pagina1, our ZClass based CMS) Fiddling with the cache parameters in the control panel showed that while we could keep the memory consumption to a point where the daily restart would be enough (say, with 10k objects per thread), we would get constant thrashing of cache objects, specially DateTime objects due to the ZCatalog queries, and the machine performance would be close to intolerable, whereas if the cache parameters where set to allow a fast performance (with 50k objects per thread) the machine would run out of memory in 3 to 4 hours. Needless to say this was with heavy use of RAMCacheManagers, not counting the accelerator proxy in front of it. Without the RAMCaches, the machine would go down in under 5 minutes of work hour load. Even with the caches on, the load would never go down from 2.0 during work hours. Last tuesday we decided no longer to wait for 2.6.2 and migrated the site to 2.6.1. We dealt with the ObjectManager-based-ZClass issue, reformed the ZCatalogs to replace the DateTime FieldIndexes with DateTimeIndexes and then had a working testing environment, which we stress tested lightly without detecting any problems and quickly moved to production. This was late at night The next morning we were surprised to notice the machine very quickly ran out of memory. The memory leak was *far more severe* than before. Zope needed a restart every 15m or so before it would send the machine into heavy swaping. In a very non-intuitive hunch I suggested we shut down all RAMCaches and, amazingly enough, this made the situation a bit more manageable. We're now restarting every 45 minutes. To our relief, disabling the RAMCaches had only a barely noticeable effect on performance. The site kept churning out pages really fast, a testatment to the optimization job done in the 2.6 series. The load on the machine is rarely above 0.8, except when it goes into swap :-) The number of DateTime refcounts in the Control_Panel, although much smaller than in Zope 2.5.1 is very high and, mostly importantly, constantly increasing, as far as I can tell. After 12 minutes of uptime, the top refcounts are: DateTime.DateTime.DateTime: 96168 BTrees._IOBTree.IOBucket: 43085 BTrees._IIBTree.IIBTree: 40400 BTrees._IIBTree.IIBucket: 23696 OFS.DTMLDocument.DTMLDocument: 23190 BTrees.OIBTree.OIBucket: 14582 BTrees._IIBTree.IISet: 12479 BTrees._IIBTree.IITreeSet: 10823 BTrees.OOBTree.OOBucket: 7088 OFS.Image.Image: 6860 OFS.DTMLMethod.DTMLMethod: 5894 DocumentTemplate.DT_Util.Eval: 3250 OFS.Image.File: 2796 BTrees._IOBTree.IOBTree: 2761 ZClasses.Method.MWp: 1592 In time, DateTime refcounts eventually dwarves the second place by an order of magnitude. I think this is related to the fact that DateTime instances are stored as metadata, even though the date indexes have been converted to DateTime indexes. The question is, why aren't those instances being released? What is holding on to them? I tried installing the LeakFinder product but discovered it didn't work before stumbling in a message in the archives that told me exactly that :-) The RefCounts view in the LeakFinder object fails with the following traceback: Traceback (innermost last): [...] Module DocumentTemplate.DT_Util, line 201, in eval - __traceback_info__: REQUEST Module <string>, line 0, in ? Module Products.LeakFinder.LeakFinder, line 240, in manage_getSample Module Products.LeakFinder.LeakFinder, line 163, in getControlledRefcounts Module Products.LeakFinder.LeakFinder, line 188, in resetCache TypeError: function takes at most 2 arguments (3 given) The code in question is: def resetCache(c, gc): cache = c._cache if gc: cache.minimize(3) # The minimum that actually performs gc. # Replace it with an empty cache. c._cache = PickleCache(c, cache.cache_size, cache.cache_age) PickleCache is imported from ZODB.cPickleCache, which is (surprise!) in C and I couldn't figure out how I should fix the call above to work. In an ExternalMethod, I consulted the python module gc and checked that gc *was* enabled and *no* uncollectable cycles where found. In trying to narrow down the causes of the leak, we detected what looks like a minor one. If you insert a very simple dtml method in Zope root and bang on it at a rate of, say, 100 requests per second, Zope will increase it's memory footprint very slowly, say, a couple hundred Ks every 10 minutes (both RSS and virtual size). I figure this has something to do with memory fragmentation or some other low level stuff. But this is definetly not our main concern, which is to find out how is it that Zope is leaking DateTimes so heavily in our site, and if it is really all those DateTimes that are hogging our memory. We need help, and we need it desperately. If anyone wants any other information we'd be happy to provide. -- Leonardo Rochael Almeida <leo(a)hiper.com.br> Hiperlogica informatica e educacao ltda.

13 38

bug in manage_pasteObjects ??
by Romain Slootmaekers 05 Sep '03

05 Sep '03

hi, after an object is copied & paste, the following method is called: def manage_afterClone(self, item): """ """ as it happens, both self and item refer to the same object.... which severely limits the use of the method. You probably want a reference to the original object from which this one was cloned. I googled and found a mail with the same remarks posted in 1999: http://mail.python.org/pipermail/zope-dev/1999-November/002303.html from which I copy the relevant part for your convenience: <--------------- c & p -----------------------> In lib/OFS/CopySupport.py In the function manage_pasteObjects the following line appears: ob=ob._getCopy(self) ob.manage_afterClone(ob) Passing ob as an argument doesn't make sense because the function is being called on ob, (so self is already ob) it would make more sense to to something along the lines of: original_ob=ob ob=ob._getCopy(self) ob.manage_afterClone(original_ob) </--------------- c & p -----------------------> but there's no follow- up, no reply, no fix. :( Romain Slootmaekers

3 7

Help!!
by Todd Loomis, (SAIC) 04 Sep '03

04 Sep '03

All: I'm getting this error message every time I try to start Zope: /var/www/zope/lib/python/ts_regex.py:87: DeprecationWarning: the regex module is deprecated; please use the re module import regex, regsub #, Sync /usr/lib/python2.1/regsub.py:15: DeprecationWarning: the regsub module is deprecated; please use re.sub() DeprecationWarning) What gives, it was working fine then I got a call saying the site wasn't responding. So I restarted Zope and got this. Any help will be greatly appreciated. ============================================ Todd

6 9

Re: tal i18n and ZopeStarter test failures
by Chris Withers 29 Aug '03

29 Aug '03

Hi, I see these tests fail on both Windows and Linux. What gives? Who checked in this code without running the tests?! Chris PS: Does anyone know why there's that SearchIndex warning appearing, but on Linux only? zope-tests(a)squishdot.org wrote: > ====================================================================== > Python Version:2.2.3 (#1, Jun 10 2003, 13:52:48) > [GCC 2.95.2 20000220 (Debian GNU/Linux)] > > Modules included: > Zope (HEAD) > > Warning: no test input files found!!! > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/SearchIndex/__init__.py:19: DeprecationWarning: The usage of the SearchIndex package is deprecated since Zope 2.4. > This package is only kept for backwards compatibility for a while > and will go away in a future release. > > Please use instead the re-factored modules in Products/PluginIndexes. > > warnings.warn("The usage of the SearchIndex package is deprecated since \ > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes="value title"). Please use semicolon to separate attributes (i18n:attributes="value; title"). > File None at row, column (1, 0) > Attributes name > , DeprecationWarning) > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:862: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes="value title"). Please use semicolon to separate attributes (i18n:attributes="value; title"). > File None at row, column (14, 0) > Attributes name title > , DeprecationWarning) > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:862: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes="value title"). Please use semicolon to separate attributes (i18n:attributes="value; title"). > File None at row, column (17, 0) > Attributes name title > , DeprecationWarning) > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes="value title"). Please use semicolon to separate attributes (i18n:attributes="value; title"). > File None at row, column (1, 0) > Attributes value > , DeprecationWarning) > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes="value title"). Please use semicolon to separate attributes (i18n:attributes="value; title"). > File None at row, column (1, 0) > Attributes alt > , DeprecationWarning) > /stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/TAL/TALGenerator.py:870: DeprecationWarning: Space separated attributes in i18n:attributes are deprecated (i18n:attributes="value title"). Please use semicolon to separate attributes (i18n:attributes="value; title"). > File None at row, column (5, 3) > Attributes alt > , DeprecationWarning) > ====================================================================== > ERROR: testSetupServers (testStarter.ZopeStarterTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "testStarter.py", line 166, in testSetupServers > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/Zope/Startup/__init__.py", line 186, in setupServers > raise ZConfig.ConfigurationError(socket_err > ConfigurationError: There was a problem starting a server of type "HTTPServer". This may mean that your user does not have permission to bind to the port which the server is trying to use or the port may already be in use by another application. (Address already in use) > > ====================================================================== > ERROR: test_fcgi_factory (test_config.ZServerConfigurationTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 113, in test_fcgi_factory > factory = self.load_factory("""\ > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 48, in load_factory > StringIO.StringIO(text)) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 45, in loadConfigFile > return _get_config_loader(schema, overrides).loadFile(file, url) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 55, in _get_config_loader > loader = ConfigLoader(schema) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 163, in __init__ > if schema.isabstract(): > AttributeError: 'NoneType' object has no attribute 'isabstract' > > ====================================================================== > ERROR: test_ftp_factory (test_config.ZServerConfigurationTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 135, in test_ftp_factory > factory = self.load_factory("""\ > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 48, in load_factory > StringIO.StringIO(text)) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 45, in loadConfigFile > return _get_config_loader(schema, overrides).loadFile(file, url) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 55, in _get_config_loader > loader = ConfigLoader(schema) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 163, in __init__ > if schema.isabstract(): > AttributeError: 'NoneType' object has no attribute 'isabstract' > > ====================================================================== > ERROR: test_http_factory (test_config.ZServerConfigurationTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 66, in test_http_factory > factory = self.load_factory("""\ > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 48, in load_factory > StringIO.StringIO(text)) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 45, in loadConfigFile > return _get_config_loader(schema, overrides).loadFile(file, url) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 55, in _get_config_loader > loader = ConfigLoader(schema) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 163, in __init__ > if schema.isabstract(): > AttributeError: 'NoneType' object has no attribute 'isabstract' > > ====================================================================== > ERROR: test_icp_factory (test_config.ZServerConfigurationTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 161, in test_icp_factory > factory = self.load_factory("""\ > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 48, in load_factory > StringIO.StringIO(text)) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 45, in loadConfigFile > return _get_config_loader(schema, overrides).loadFile(file, url) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 55, in _get_config_loader > loader = ConfigLoader(schema) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 163, in __init__ > if schema.isabstract(): > AttributeError: 'NoneType' object has no attribute 'isabstract' > > ====================================================================== > ERROR: test_monitor_factory (test_config.ZServerConfigurationTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 148, in test_monitor_factory > factory = self.load_factory("""\ > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 48, in load_factory > StringIO.StringIO(text)) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 45, in loadConfigFile > return _get_config_loader(schema, overrides).loadFile(file, url) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 55, in _get_config_loader > loader = ConfigLoader(schema) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 163, in __init__ > if schema.isabstract(): > AttributeError: 'NoneType' object has no attribute 'isabstract' > > ====================================================================== > ERROR: test_pcgi_factory (test_config.ZServerConfigurationTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 104, in test_pcgi_factory > factory = self.load_unix_domain_factory("""\ > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 57, in load_unix_domain_factory > factory = self.load_factory(text % fn) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZServer/tests/test_config.py", line 48, in load_factory > StringIO.StringIO(text)) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 45, in loadConfigFile > return _get_config_loader(schema, overrides).loadFile(file, url) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 55, in _get_config_loader > loader = ConfigLoader(schema) > File "/stuff/chrisw/ZopeTests/sandbox/Zope/lib/python/ZConfig/loader.py", line 163, in __init__ > if schema.isabstract(): > AttributeError: 'NoneType' object has no attribute 'isabstract' > > ====================================================================== > ERROR: test_webdav_source_factory (test_config.ZServerConfigurationTestCase) > - > <truncated> > > _______________________________________________ > Zope-Coders mailing list > Zope-Coders(a)zope.org > http://mail.zope.org/mailman/listinfo/zope-coders > >

2 4

Zope 2.7 running with Stackless 3.0
by Christian Tismer 29 Aug '03

29 Aug '03

Dear friends, just by chance, I got into the position to try out Zope 2.7.0 beta 1/2, and since it needs Python 2.2.3 at least, and Stackless 3.0 was just ported to that, I tried to build Zope with Stackless 3.0. It works very very well! After a few patches to get the includes right, here the installation instructions. Get the current Stakless 3.0 beta. CVSROOT=:pserver:anonymous@centera.de:/home/cvs export CVSROOT cvs co stackless cd stackless/src ./configure make # su if your weren't root make install Then, install the Zope source, and make the same dance as usual. Just a note: Zope doesn't (yet) use any of the Stackless features. It just builds and works with it. But of course, *you* can use the stackless features, of course. I also expect, that Zope will create a branch in some future and implement a new Medusa which doesn't need real threads. cheers and let me know of your experience -- chris -- Christian Tismer :^) <mailto:tismer@tismer.com> Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

4 6

upgrading and getting rid of SearchIndex
by Anthony Baxter 28 Aug '03

28 Aug '03

I've got a couple of ZODBs here that's been gradually upgraded over time from Zope version to Zope version (I think they were initially started on Zope 1.x!) I'm sick of the SearchIndex deprecation warnings, so I've gone and re-created all of the catalog indexes. This fixed it for the indexes, but it's still showing up - any ideas how I can determine which objects are still referencing SearchIndex? Is it worth making a little script that Does The Right Thing when upgrading? It's a pretty trivial script to open a Data.fs and hunt for indexes, then re-create them - is this something that could/should be included with Zope? Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

2 1

Zope 2.7 Timeline
by Andy McKay 27 Aug '03

27 Aug '03

This is a really silly question but I'll try... Anyone got any idea when 2.7 might be ready? I building installers now and trying to judge which will come first a working 2.7 version, or the working code. I'd love to use 2.7 if its working so Im taking that risk... -- Andy McKay http://www.agmweb.ca

4 11

2.6.2b..., Find is busted
by Anthony Baxter 27 Aug '03

27 Aug '03

Go to the Find tab in the ZMI, select "All types", containing "<foo>" Error type: TypeError Error value: expected a character buffer object Traceback (innermost last): Module ZPublisher.Publish, line 49, in publish Module ZPublisher.mapply, line 32, in mapply Module ZPublisher.Publish, line 38, in call_object Module Shared.DC.Scripts.Bindings, line 250, in __call__ Module Shared.DC.Scripts.Bindings, line 270, in _bindAndExec Module App.special_dtml, line 110, in _exec Module DocumentTemplate.DT_With, line 58, in render Module DocumentTemplate.DT_Util, line 175, in eval __traceback_info__: obj_metatypes Module <string>, line 0, in ? Module OFS.FindSupport, line 51, in ZopeFind TypeError: expected a character buffer object

3 2

Why not ZODB 3.1.3 with Zope 2.6.2?
by Paul Winkler 27 Aug '03

27 Aug '03

I downloaded the "hidden" 2.6.2b6 release (from http://zope.org/Products/Zope/2.6.2b6/Zope/view) and had a look in lib/python/ZODB/__init__.py, which gives its version as ZODB 3.1.2. Is there any reason it's not ZODB 3.1.3? Maybe just an oversight? I could easily check any ZODB-3.1.3 changes to the Zope-2.6 branch, unless somebody knows a reason not to do so. Also, why doesn't ZODB/Tools come with zope? There's some really handy stuff in there, some of which gets mentioned on the zope list from time to time; it would be nice if newbies could get hold of it easily. On a related note, NEWS.txt in ZODB 3.1.3 mentions that there is a new Tools/README.txt ... but it's not actually in the tarball :-) -- Paul Winkler http://www.slinkp.com

2 2

zope.org down....
by Romain Slootmaekers 26 Aug '03

26 Aug '03

Sorry for the offtopic/crosspost but, www.zope.org is down, and with a rather ugly error message: <ugly_message> ERROR The requested URL could not be retrieved -------------------------------------------------------------------------------- While trying to retrieve the URL: http://12.155.117.33:8082/VirtualHostBase/http/zope.org:80/zopeorg/VirtualH… The following error was encountered: Connection Failed The system returned: (111) Connection refused The remote host or network may be down. Please try the request again. Your cache administrator is sa(a)zope.com. </ugly_message> hope someone can do something ;) Romain Slootmaekers

3 2