April 2000 - Zope - Zope lists

Re: [Zope] Zope Eating Memory for Breakfast
by Hung Jung Lu 13 Apr '00

13 Apr '00

>From: Monty Taylor <mtaylor(a)goldridge.net> Thanks for the feedback. It's very helpful. > > (4) ZODB: I don't have anything that dynamically changes Zope's > > ZODB, that is, I don't upload files into ZODB (Do you?), I >... > My site changes a bit more than that, but it's still fairly >static. My web-content manager may post a new org-chart or internal news >story, but we don't even have that many people connecting to read yet. > > > (5) Size of ZODB: my Data.fs is about 9 MB. > > That would be 525Meg. My god, that's large. But I think that's >mainly pictures. Aha! There is the first clue. The memory increase has to do with Zope ZODB operations. Why does it have to do with ZODB operations? (1) There are Zope sites out there running for a long time and people aren't complaining about memory leak problems. These include some heavily-visited sites. (2) Everytime you restart Zope the memory usage starts out OK. (3) The size of the problem is proportional to the size of Data.fs (4) I played around a bit with my site, and when I am viewing and accessing my site as an external user, the memory usage did not go up. (5) But when I use Zope management interface to do some operations, like clicking on the Find tab (and then the Find button in that page), the memory usage jumped. I managed to raise the memory usage by 5 MB just by clicking the Find tab/button all over places and search for files containing a diverse text pattern. Could you try with "Finding" some items in your website by using Zope's "Find" feature, and let us know whether it also raises the memory usage? So at the moment the memory leak problem does not seem to be coming from usage of products (like database adapter), but from some of the management function calls in Zope. ----------------------------------------------------------- In your case, I would recommend you using external storage for images/files. And use a product like LocalFS (see http://www.zope.org/Members/sabaini/externalfiles-howto ) to handle large files. You can also write your own external methods to save/delete/update/upload/download the external files. That should bring your memory usage drastically down to megabytes instead of gigabytes. So, in a sense, your problem can be solved (since you have a few GB of RAM memory to play around... ha ha... I am not that lucky.) It's a harder question how we are going to pinpoint the exact source of the memory leak. regards, Hung Jung ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com

3 2

Re: MIME types and zope
by anser 13 Apr '00

13 Apr '00

Sathya <linuxcraft(a)redspice.com> wrote: > I am trying to handle a vendor specific mime type within zope > basically it is the adobe forms plugin whihc encodes > data in application/vnd.fdf type. > It is basically an ascii stream set in a particular format > question is how do I access this data coming from the browser > the request variable does not contain information about this. There is a brand new HowTo about this at Zope.org: http://www.zope.org/Members/anser/howto_dtml_mimetype (although www.zope.org seems to be unreachable this morning...) The simplest way is simply to put Content-type: application/vnd.fdf followed by a blank line, at the very top of your DTML Method that's supposed to return one of these forms.

2 2

RE: [Zope] Issue Tracker
by Ken Manheimer 13 Apr '00

13 Apr '00

The issue tracker source is currently unavailable, only because we haven't had the time to shepherd its release. We were expecting to get to that in a little while (when some time is free from other commitments), but we're considering making it available unsupported in the meanwhile. I'll try to follow up soon with a more solid answer. (Sorry to be vague about the timing, but our it's either that or "not soon", because the other commitments are overriding at the moment.) Ken Manheimer klm(a)digicool.com > -----Original Message----- > From: Paolo Marini [mailto:paolom@prisma-eng.it] > Sent: Wednesday, April 12, 2000 5:52 PM > To: zope(a)zope.org > Subject: [Zope] Issue Tracker > > I would like to knwo where is the code for the PTK Issue Tracker

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Marcus Collins 13 Apr '00

13 Apr '00

I'm convinced there's some deep, dark, timing problem, and that it's thread-related... This morning, I had a thread hang, again on Zope 2.1.3 running with four threads. I was watching for unfinished requests at the time, and caught this twenty minutes after it occurred. I tried viewing a frameset page on the site. No problem. Then I tried launching /manage, and suddenly *all* the threads hung. This is the first time all the threads have hung. It's also the first time I've managed to catch an unfinished request and try viewing framesets within half an hour, which leads me to believe that, previously, ZServer was doing a good job of cleaning up the hung zombie threads (recall the zombie_timeout of 30 minutes). I'm actually wondering if reducing that zombie_timeout (and maintenance_interval in medusa/http_server.py) would go anywhere towards alleviating this problem as a temporary measure. Would there be any reasons not to try this? I've added quite a lot of DebugLogger stuff to ZServer/PCGIServer.py, and modified the log analyser accordingly, in the hopes of nailing this sucker: def send_response(self): # create an output pipe by passing request to ZPublisher, # and requesting a callback of self.log with the module # name and PATH_INFO as an argument. self.done=1 # MC 2000-04-13 additional logging DebugLogger.log('X', id(self), 'send_response: before PCGIResponse') response=PCGIResponse(stdout=PCGIPipe(self), stderr=StringIO()) # MC 2000-04-13 additional logging DebugLogger.log('X', id(self), 'send_response: before HTTPRequest') request=HTTPRequest(self.data, self.env, response) # MC 2000-04-13 additional logging DebugLogger.log('X', id(self), 'send_response: before handle') handle(self.server.module, request, response) Everything before the call to handle() is 100%. Sometimes, however, we don't get from handle() to the next stage. This is on Zope 2.1.6, which I've been running with up to 100 threads, although I unfortunately can't excercise that many! In fact, it has proven quite a mission to get Zope to hang, maybe because of the increased latency in serving requests due to the additional logging. I've added this logging to the Zope 2.1.3 serving the live site, and will report my findings as soon as something untoward occurs. Maybe others who are experiencing hanging would also be able to do some extra logging and report the results [now, there, I see Wiki would be really useful!]. In the meantime, any suggestions as to where to go next will be keenly acted on! Thanks all! -- Marcus

1 0

Raw data/large objects
by Ken Kinder 13 Apr '00

13 Apr '00

Normally, database API has special functions for handling binary columns. Suppose I have a long raw datatype in an Oracle database, which is used to store an image, a word file, or some other binary file (it doesn't really matter). In a normal DB API from C, you wouldn't try to insert the value of the column in a query, but use some special method of handling large objects. With ZSQL Database adapters, the question is: how do I handle raw data? I want the user to be able to use a CGI Upload facility to upload a binary file of a mysterious type that could contain anything. and I want it to go into a long raw data column in Oracle. So how do I do this in ZSQL/DTML? -- Ken Kinder 303.381.7631

1 0

RE: [Zope] Zope Eating Memory for Breakfast
by Hung Jung Lu 13 Apr '00

13 Apr '00

--- In zope(a)egroups.com, Chris McDonough <chrism@d...> wrote: > Both of you seem to be using Sybase DAv2, for which there was a known >memory leak. The latest release got rid of it. Are you running the >latest releases, respectively? That was my fear at first. But I am using the latest version (SybaseDA-2.0b2), so that's not the problem. As I have pointed out in another message, the memory leak is from some of the Zope's management methods. One of the symptoms is that when you do a "Find", the memory usage does not go back down to where it was before. (It does goes up and then down, just not to the original point.) Now the thing to do is to pinpoint the Zope method that is causing the memory leak. (I don't want to mislead people into thinking that "Find" is the thing causing memory leak... it's not. It's a symptom, not a cause. The memory also goes up in other Zope management operations, but I still need to pinpoint which operations.) regards, Hung Jung ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com

4 3

RE: [Zope] Minimal Zope Install - True Story
by Marcus Collins 13 Apr '00

13 Apr '00

Until its harddrive died, my little 486 server at home w. 16MB RAM was running Apache+Zope quite happily. Granted, I wouldn't even have *thought* of opening it to the world, but it was fun to play on. Maybe its drive died because of too much paging? ;-) -- Marcus > -----Original Message----- > From: Hung Jung Lu [mailto:hungjunglu@hotmail.com] > Sent: 13 April 2000 19:18 > To: Jatwood(a)bwanazulia.com; zope(a)zope.org > Subject: [Zope] Minimal Zope Install - True Story > > > >Mem: 13608K av, 12824K used, 784K free, > > > >WHAT? Yes, only 12MB of memory... The machine has lost all > >of its memory and is just running on the system board memory. Oh > >yeah, and it is a 166 MHz Pentium. > > Sorry, but I don't quite understand what your message meant. > (I am serious and not making fun.) > > You only mentioned 12MB memory. Is that all the RAM memory > on the machine? > > Do you know that a person just complained about Zope because > it was sucking up 2GB of memory? No, there is no typo. It's > 2GB, not 2MB. > > I mean, to run a website, at very least be prepared to have > 64MB, it's about US$70 nowadays. I hope I have not misunderstood, > but I can't possibly imagine Zope running on a computer that > has only 12MB! :) > > regard, > > Hung Jung > > ______________________________________________________ > Get Your Private, Free Email at http://www.hotmail.com > > > _______________________________________________ > Zope maillist - Zope(a)zope.org > http://lists.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope-dev ) >

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Marcus Collins 13 Apr '00

13 Apr '00

> -----Original Message----- > From: Tres Seaver [mailto:tseaver@palladion.com] > Sent: 13 April 2000 04:11 > To: zope-dev(a)zope.org > Cc: Monty Taylor; Michel Pelletier; Marcus Collins; Chris McDonough; > Amos Latteier; Jim Fulton; Brian Lloyd > Subject: Re:[Zope] Re: Zope hanging (poss. threads-related) Firstly, it's great to see the involvement of all these brains! Thanks to all of you! Secondly, sorry for the cross-posting... please don't hang me ;-) > I would *really* like for us to capture the information > flowing through the list in some less ephemeral form, so I have set up > a Wiki for discussing this and other availability/stability/ > reliability issues at: > http://www.zope.org/Members/tseaver/Projects/HighlyAvailableZope/ Tres, I also agree that this should not get lost amongst all the regular traffic. This list is busy enough as it is. However, perhaps other folk (Jerome Alet, others?) are just now beginning to say "me too", and so the rest of the list does need a chance to be involved. Not everyone is a member at zope.org either, and perhaps not everyone understands how Wiki works. It can also prove difficult, unless someone is moderating the posts, to not get tangled, whereas a threading mail client... this is basically just the same argument as was levelled about a week ago at the suggestion of using Wiki for bug tracking. Wiki is a fine thing, but I noticed now, when www.zope.org was back up, that despite having logged in, as soon as I when to the above URL, I was "Guest" again, and unable to annotate the discussion... (no "edit" link, etc.) Maybe this is a symptom of operator failure? For now, however, I'll continue CC'ing the zope list, unless you specifically wish it to be shunted to the zope-dev list? Maybe the other zope users have an opinion on this as well? Thanks for taking the initiative in this regard, thanks to all at DC for the ongoing support, and thanks to the list for their reports! Let's all continue together towards the solution! -- Marcus

1 0

Minimal Zope Install - True Story
by J. Atwood 13 Apr '00

13 Apr '00

I have been moving over some of my sites to Zope (fun fun fun) and so I installed Zope 2.1.6 on my RH Linux 5.2 box. After doing so I noticed it was pretty slow. Wait, no, VERY SLOW. I took at look at TOP and here is what I found. Running. - Apache (15 virtual hosts) - Qmail (15 domains) - JServe (15 domains) - PostGreSQL Server - Zope - Every other little Linux thing... (FTP, etc) All of that seems fine... until I take a look at the memory.. Mem: 13608K av, 12824K used, 784K free, WHAT? Yes, only 12MB of memory... The machine has lost all of its memory and is just running on the system board memory. Oh yeah, and it is a 166 MHz Pentium. Ha... lets see Microsoft run NT 4, Exchange 5.5, IIS4 w/ SQL 7.0 an that! Needless to say it shall be fixed this weekend but still it is a testament to Linux, Apache and Zope can do. J

1 0

RE: [Zope] Re: Zope hanging (poss. threads-related)
by Marcus Collins 13 Apr '00

13 Apr '00

> -----Original Message----- > From: Tony Rossignol [mailto:tonyr@ep.newtimes.com] > Sent: 12 April 2000 19:51 > To: Marcus Collins > Cc: zope(a)zope.org > Subject: Re: [Zope] Re: Zope hanging (poss. threads-related) > >Marcus Collins wrote: > >> Our live server, running Zope 2.1.3 with 4 threads, has just had a >> thread hang. There has been no management activity on this server >> since last it was restarted about four hours ago. At the time of the >> hanging, a frameset was being requested (cf. my original post). > The /manage screens also have the potential for the client to send > multiple request. As a twist to this story, it would be interesting to know if those folk who experienced frequent Zope restarts when accessing management screens experienced the same problems using just manage_workspace or ftp (thus bypassing multiple simultaneous requests from the client). I guess what I'm driving at is that this is some kind of timing issue. Thanks for your valuable comments, Tony. -- Marcus

2 1