December 2004 - Zope

RE: [Zope] Zope 2.7.3 Memory Leaks
by Andy Yates 05 Dec '04

05 Dec '04

> -----Original Message----- > From: Chris McDonough [mailto:chrism@plope.com] > Sent: Thursday, December 02, 2004 5:17 PM > To: Andy Yates > Cc: zope(a)zope.org > Subject: Re: [Zope] Zope 2.7.3 Memory Leaks > > On Thu, 2004-12-02 at 17:24, Andy Yates wrote: > > Now it seems to leak memory (400+ Mb) to the point I have to restart > > Zope every 3 days. > > > > I've read that memory leaks are almost always caused by the programmer, > > so I set up some very simple tests that did not involve existing code. > > Great idea. Note that I'm not trying to find a problem with Zope. My plan was to start with something very simple and then add more and more of the things we do on our production site and see when it starts leaking. However, even the simplest tests seem to leak memory. > > Next I created an 8Mb page template. This caused the memory usage to > > climb much faster. After the test the memory never goes back down. > > Do these templates access the sessioning machinery? > No. It is just a page template file with 8Mb of random text. > > I installed LeakFinder and started running more tests. > > > > I used ab to get a python script that stored a 1k string in the session > > object. The transient object container timeout was set to 1 minute. > > This also caused Zope to consume memory. LeakFinder said the > > Products.Transience.TransientObject.TransientObject ref count grew with > > each request and never went down. Everything else seemed to level off. > > The debug output of Transience.py showed that the "buckets" seemed to be > > getting deleted as expected, but the memory usage never goes down. 1 > > minute after the test stopped the transient object container showed that > > there were no more items in the container. It seems like when Zope > > deletes expired sessions but contents of the sessions are not deleted. > > This is actually probably normal. > > TransientObject buckets hang around for some period of time before they > are "garbage collected" (deleted from their container). A particular > TransientObject isn't garbage collected immediately when it expires, but > gets gc'ed much later along with other TransientObjects that were > created around the same time as a side effect of otherwise exercising > the sessioning machinery. Thanks for the explaination of how the transient gc works. That is exactly what I found when I looked in the source code and I can see all of this happening when I turn on the debug output. Sessions are created and placed in buckets. The active sessions move to the current bucket and expired session and old empty buckets get removed. Then there is a 1 in 6 chance they will be gc'ed on each request. I see all this happening but when the buckets are gc'ed the memory usage according to top does not go down. I ran the test again that puts a 1k string in the session objects. I used ab to run the python script 3000 times with a concurrency of 10. Python's memory went from 22Mb to 54Mb. Transient objects went from 0 to about 900 and stayed steady during the test. After the test the trans obs dropped back down to 0 as they expired. I continued to make occasional Zope requests to plain pages to make sure gc was getting tickled. After 8+ hours of this the memory was never released. Python was still using 54Mb. If I continue to create session objects the 54Mb is not reused by python it just continues to grow. > > Note that Python will usually not release memory back to the OS once it > has been allocated for its own use, so a large concrete amount of memory > consumed by Python may not even be an indicator of a leak! See > http://mail.python.org/pipermail/python-dev/2004-October/049480.html for > more info. The only solid basis for determining leaks is refcounts, and > it's tricky to figure out whether growing refcounts are leaks or if > they're just side effects of the ZODB cache holding on to old objects. I tested that too and python 2.3.4 on RedHat 9 does not seem to behave this way. I wrote a script that creates 100Mb of random strings and places them in a list. Then it waits a few seconds and deletes the list. When I look at the process with top I see memory (RSS) go up by about 100Mb and then a few seconds later it goes back down to just above the level when the script started. Then I put this in a loop. I can watch the memory usage just cycle up and down. This is what I would expect. Thanks! Andy

4 3

Connecting to a MaxDB Database via ODBC (porting old C++ application to Zope)
by Hermann Himmelbauer 05 Dec '04

05 Dec '04

Hi, I am developing a Zope web application which relies on an existing Database (MaxDB, formerly SAPDB) Client written in C++. This client accessed my database via ODBC (on UNIX). My web application should on one hand access the database directly via a Zope Database Adapter and on the other hand use functions from this C++ Client which also accesses the database. (Some functions of this C++ Client are compiled as a library which can then be called from Zope). What I now need is a way to access my Database over one Database connection. What I am thinking of is to use some Zope Database Adapter that uses ODBC, and give the database handle to my C++ functions. My problem is: How would I get the ODBC Database handle from the Zope Database Adapter object? Is this easy to do? Will I run into problems by doing this? Moreover I don't know if there's a ODBC Database adapter available which is compatible to Zope and to MaxDB. There is a MaxDB Zope DA available but this does not rely on ODBC. Then there is this commercial mxODBC available but I don't know if it supports MaxDB. Can you give my some hints? Best Regards, Hermann -- x1(a)aon.at GPG key ID: 299893C7 (on keyservers) FP: 0124 2584 8809 EF2A DBF9 4902 64B4 D16B 2998 93C7

2 1

GPL question
by Alan Snyder 05 Dec '04

05 Dec '04

Hi everyone. I have a sort-of off-topic question about zope, plone and the GPL. If I write an application that sits on top of plone by using the CMF and the plone components and such, how must my portions be licensed? I've not written anything yet but just to be sure, if i write plone stuff that calls plone and cmf and zope functions and objects, must that also be GPL'd. I was under the assumption that calling GPL software didn't required making your stuff GPL but if you were to write code inside a module that is already gpl'd - like expanding it, then you'd obviously have to gpl it because you're modifying code - not writing new code. Also, I know the zope license is ZPL and more commercially friendly than the gpl, but plone is gpl so obviously there is a boundary there that is being enforced correctly. TIA

2 1

Fwd: [Zope] Zope <base> tag
by Alan Snyder 05 Dec '04

05 Dec '04

Just got my "Definitive Guide to Zope" book and can't wait to start. On Sun, 05 Dec 2004 10:50:39 +0100, Johan Carlsson <johanc(a)easypublisher.com> wrote: > Alan Snyder wrote: > > aaahhhh... i see. I'll start messing with the VHM thingy. > > > > I'm still new to Zope and am overwhelmed frankly - but in a holy *$&!@ > > this is awesome kinda way! I'm getting a zope and a plone book today > > so hopefully i'll be more experienced in the not too distant future. > > Well. IMHO Zope is the Dope :-) > > I've been using it since 1999 and really have a hard time > when ever I'm forced to work with other less comprehensive > web development environment. > > I hope the books are great. Good luck :-) > > Cheers, > Johan > > -- > > > Johan Carlsson Tel: + 46 8 31 24 94 > Colliberty Mob: + 46 70 558 25 24 > Torsgatan 72 Email: johanc(a)easypublisher.com > SE-113 37 STOCKHOLM >

1 0

number of objects in ZooDB
by Giuseppe Bonelli 05 Dec '04

05 Dec '04

Hi list, I use ZooDB to store a fairly large amount (?) of objects and I am wondering if it is best from a performance point of view to have a lot of small object or fewer bigger. About 75% of my objects are also cataloged in the Zcatalog. I have complete flexibility on chosing the object number/size ratio, as they are actualli xml fragments. My Zope instance is cached by a front-end apache and the objects are mainly read-only (apart from the initial import in the ZooDB). Is 500.000 objects to be considered an "high" number? Thanks in advance for any insight. __peppo

2 1

Zope <base> tag
by Alan Snyder 03 Dec '04

03 Dec '04

It seems that every page that zope generates has this <base href=""> tag in the html header. Where is this set and how / should I change it. I'm trying to hook up apache as a proxy server to zope with url address re-writing but it keeps failing, and I think part of the issue is that this <base> tag over-rides what apache does and forces my browser to look at relative links from that :8080 zserver port instead of going through apache. Thanks in advance

3 2

Re: [Zope] Can Zope Do....?
by Alan Snyder 03 Dec '04

03 Dec '04

I'm in love with zope from what i've seen of this Elvis Sightings interactive demo. It's so friggin' easy to build functionalities into pages. Holy smokes!! Before I go further with this, I have some questions... 1) I am creating all of these pages through the zope manager interface which is great, but is it possible, and if so how, to do what I'm doing here and update a client remotely through something like a CVS or some pre-packaged zope update? Is it a matter of just copying files and such from a folder (like the parent elvis sightings folder), or do i need to grab stuff from different zope config areas and work it. Bottom line - is there any synchronization or update mechanisms available that let me develop and fix sites locally and effect those changes on a remote server, i.e. at a client? 2) I saw in one of the chapters that each page can reference a template (like standard_template.something)... This looks really cool for giving a similar layout to pages in a "folder". Can sub-folder templates "over-ride" parent folder templates. In other words, if i copy the "standard_template" file to a sub folder and slightly modify it will that version be picked up by those folder's pages, or will the parent version be used? 3) What are the items in that drop down list of things in the upper right corner of every folder called? It has stuff like "DTML Document, Mai Host..." Are those "products"? Where can I find more of those. They're wonderful. Also - how hard are they to install at a client? What mechanisms are available for producing "install scripts" for clients who want to use the stuff I build? Thanks again. Lovin' Zope so far! On Thu, 2 Dec 2004 12:04:51 -0500, Alan Snyder <alan8373(a)gmail.com> wrote: > I'm in love with zope from what i've seen of this Elvis Sightings > interactive demo. It's so friggin' easy to build functionalities into > pages. Holy smokes!! > > Before I go further with this, I have some questions... > > 1) I am creating all of these pages through the zope manager interface > which is great, but is it possible, and if so how, to do what I'm > doing here and update a client remotely through something like a CVS > or some pre-packaged zope update? Is it a matter of just copying files > and such from a folder (like the parent elvis sightings folder), or do > i need to grab stuff from different zope config areas and work it. > Bottom line - is there any synchronization or update mechanisms > available that let me develop and fix sites locally and effect those > changes on a remote server, i.e. at a client? > 2) I saw in one of the chapters that each page can reference a > template (like standard_template.something)... This looks really cool > for giving a similar layout to pages in a "folder". Can sub-folder > templates "over-ride" parent folder templates. In other words, if i > copy the "standard_template" file to a sub folder and slightly modify > it will that version be picked up by those folder's pages, or will the > parent version be used? > 3) What are the items in that drop down list of things in the upper > right corner of every folder called? It has stuff like "DTML Document, > Mai Host..." Are those "products"? Where can I find more of those. > They're wonderful. Also - how hard are they to install at a client? > What mechanisms are available for producing "install scripts" for > clients who want to use the stuff I build? > > Thanks again. > > Lovin' Zope so far! > > > > > On Thu, 02 Dec 2004 17:40:28 +0100, bruno modulix <bruno(a)modulix.org> wrote: > > Alan Snyder wrote: > > > Ok - I think I'll play with Zope 2 - I have a book on it and can flip > > > through that. I'll play with the base Zope packages first and then hop > > > onto Plone. I love the layout and look and feel of Plone and I'm > > > curious about the visuals of naked zope. > > > > 'naked' Zope doesn't have much visuals - unless you're talking about the > > ZMI !-) > > > > > There's no better way to find > > > out than to play so I'm hopefully able to dedicate today to that task. > > > > Have fun, and don't despair : beginning with Zope is known to be > > somewhat difficult, but the reward is worth the time invested. > > > > Bruno > > -- > > > > > > Bruno Desthuilliers - Analyste-programmeur > > bruno(a)modulix.org > > www.modulix.com > > > > >

6 8

Zope 2.7.3 Memory Leaks
by Andy Yates 03 Dec '04

03 Dec '04

We recently upgraded from: Zope Version (Zope 2.6.1 (source release, python 2.1, linux2), python 2.2.2, linux2) Python Version 2.2.2 (#1, Feb 24 2003, 19:13:11) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] To: Zope Version (Zope 2.7.3-0, python 2.3.4, linux2) Python Version 2.3.4 (#1, Sep 14 2004, 10:22:31) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] Now it seems to leak memory (400+ Mb) to the point I have to restart Zope every 3 days. I've read that memory leaks are almost always caused by the programmer, so I set up some very simple tests that did not involve existing code. First I used ab to get a simple page template. After running 1000s of request the memory had climbed slowly and never went down. Next I created an 8Mb page template. This caused the memory usage to climb much faster. After the test the memory never goes back down. I installed LeakFinder and started running more tests. I used ab to get a python script that stored a 1k string in the session object. The transient object container timeout was set to 1 minute. This also caused Zope to consume memory. LeakFinder said the Products.Transience.TransientObject.TransientObject ref count grew with each request and never went down. Everything else seemed to level off. The debug output of Transience.py showed that the "buckets" seemed to be getting deleted as expected, but the memory usage never goes down. 1 minute after the test stopped the transient object container showed that there were no more items in the container. It seems like when Zope deletes expired sessions but contents of the sessions are not deleted. Is anyone else seeing this behavior? Is there a workaround or fix? Thanks! Andy Yates

5 5

Zope deamon doesn't start any more
by Roland Giesler 03 Dec '04

03 Dec '04

I have installed zope 2.7.0 using Mandrake urpmi (rpm installed) and proceeded to play with zope. All worked fine for about 4 hours. Then I downloaded the plone tarball, extracted it into the /var/lib/zope/Products/ folder, and used the zope control panel to restart zope. It didn't start. It looks like it starts, but does nothing. [root@inet lib]# /etc/rc.d/init.d/zope start Starting zope: Traceback (most recent call last): File "/usr/lib/zope/lib/python/Zope/Startup/zopectl.py", line 207, in ? main() File "/usr/lib/zope/lib/python/Zope/Startup/zopectl.py", line 190, in main c.onecmd(" ".join(options.args)) File "/usr/lib/python2.3/cmd.py", line 210, in onecmd return func(arg) File "/usr/lib/zope/lib/python/Zope/Startup/zopectl.py", line 134, in do_start ZDCmd.do_start(self, arg) File "/usr/lib/zope/./build/build-base/python-2.3/build-lib/zdaemon/zdctl.py", line 214, in do_start File "/usr/lib/python2.3/os.py", line 543, in spawnvp return _spawnvef(mode, file, args, None, execvp) File "/usr/lib/python2.3/os.py", line 504, in _spawnvef wpid, sts = waitpid(pid, 0) OSError: [Errno 10] No child processes [root@inet zope]# /etc/rc.d/init.d/zope start Starting zope: daemon process started, pid=7855 So it appears to be running. But I cannot connect to it via a browser. (Error: Connection refused) My /var/log/zope/event.log doesn't have any more entries whether I try to start zope or not: 2004-12-01T12:18:23 INFO(0) Zope Set effective user to "zope" ------ 2004-12-01T12:18:37 INFO(0) Zope Ready to handle requests ------ 2004-12-01T21:54:47 INFO(0) ApplicationManager Restart requested by "admin" ------ 2004-12-01T21:54:47 INFO(0) ZServer closing HTTP to new connections ------ 2004-12-01T21:54:47 INFO(0) ZServer closing FTP to new connections ------ 2004-12-01T21:54:47 INFO(0) Zope Shutting down with exit code 1 What could be wrong? Roland

6 11

Formulator or html form code, recommendations?!?
by Guidry, Wayne 03 Dec '04

03 Dec '04

A newbie to Zope here: 1.) Is it better to write forms using Formulator or use standard html code to generate forms for an application created in Zope? Just curious for input from others... any issues with either method? 2.) Also, is it possible to set values of form objects programmatically (such as an IntegerField value) using DTML? or would you have to use javascript/python to set the values? thanks Wayne

1 0