October 2002 - Zope - Zope lists

VirtualHostMonster and "unpack list of wrong size"
by Jo Meder 24 Oct '02

24 Oct '02

Hi all. After installing Zope 2.6final (which was rather a rather smooth transition from 2.5.1, congrats to all involved!) we found that our current setup exposed a very annoying behaviour. We have Apache doing the virtual host dance in front of Squid in accelerator mode to keep the system load low in front of Zope. Apache is intrumented with the usual Rewrite-Rule a la RewriteRule /(.*) http://localhost:1234/VirtualHostBase/http/%{HTTP_HOST}:80/cms_root/some_na… [L,P] Now there seem to be cases (we tracked most of them to usage of wget as a user agent) where VirtualHostMonster tries to decode "www.hostname.tld:80:80". The error occurs when VHM tries to execute host, port = host.split(':') Granted the original request is invalid and should at least generate a warning I'd suggest replacing the cited line by hostparts = host.split(':') host = hostparts[0] port = hostparts[1] This is not a bugfix since the hostname/port combination is obviously invalid, but it does add reasonable behaviour along the lines of "be liberal in what you accept and be conservative in what you send", which was one of the first lines I memorized when I started learning about protocols. Jo. -- Internetmanufaktur Jo Meder ---------------------- Berlin, Germany http://www.meder.de/ ------------------- fon: ++49-30-417 17 63 33 Kollwitzstr. 75 ------------------------ fax: ++49-30-417 17 63 45 10435 Berlin --------------------------- mob: ++49-170- 2 98 89 97 Public GnuPG-Key ---------- http://www.meder.de/keys/jo-pubkey.txt

3 4

RE: [Zope] Problems with restarts, memory usage, DB connections, FastCGI ?? Help!
by Jean-Francois.Doyon＠CCRS.NRCan.gc.ca 24 Oct '02

24 Oct '02

Charlie, Thanks for the insights, this should proove useful. About the momory, to be honest, I'm not convinced that's it. Yesterday for instance, Zope showed that behavior TWICE in one day, the first time it recovered itself, the second, I restarted. But, throughtout all this, there was PLENTY of swap available (My swap space is 2GB). I've NEVER seen the swap usage go beyond ~150MB actually. >From what you're saying, I think this issue of one thread per request might be the problem. I'm going to start by tackling this. It's the easiest and cheapest way for now. The issue of long requests (People adding data/content) mixed with the heavy load of the server (i.e. requests per second), and it is forseeable that this would happen. The issue of the large images is allways in my mind, but since that only happens occasionally, and in batches (It was going on this week), I'm willing to risk it for the increased convenience. While on the topic, I'm just curious as to why the processes/threads are structured as such: |-python2.1,11334) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | `-python2.1,11337) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | `-python2.1,11353) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | |-python2.1,11354) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | |-python2.1,11355) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | |-python2.1,11356) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D | `-python2.1,11357) /usr/local/Zope-2.5.0/z2.py -F /tmp/fcgi/zope.soc -w - -p - -f - -D One talks to one talks to one talks to four ... Also, if I'm seeing it in PS, is it really a thread ? Or a process ? ps -m isn't implemented on unix, therefore can't show threads, so what am I seeing ??? Thanks for the help! J.F. -----Original Message----- From: Charlie Reiman [mailto:creiman@kefta.com] Sent: Thursday, October 24, 2002 12:30 PM To: Jean-Francois.Doyon(a)CCRS.NRCan.gc.ca; zope(a)zope.org Subject: RE: [Zope] Problems with restarts, memory usage, DB connections, FastCGI ?? Help! > So far I've been unable to track down WHY. Uhh.... > One thing that also happened just once was that the entire server ended up > running out of memory! Sounds like you tracked it down just fine. You are clearly exhasting your physical memory, the system is swapping like crazy, then it may eventually slog through the piggie bits to resume normal operation. You need more more memory and more threads. There are real limits as to what any system can do reasonably. It sounds like just bumping up the number of DB connections won't do anything for you. Arguably, the right thing for you to do is _decrease_ the number of available connections until you get more hardware (assuming the DB connections are used exclusively to upload files). Your system clearly can't maintain the load you are asking it to handle. It would be better to deny upload service to some folks than let the entire system collapse. Look into buying more memory and going with ZEO to split up the load. Read the man page for ps (assuming you're on unix) and top so you can track the amount of memory getting sucked down. Consider an emergency stop-gap solution where only one user may upload a file at a time. Increasing the number of theads (but not connections) may help. Some background here on why this is so painful: Zope starts with N theads (7, I think) for handling requests. A request is handled by a single thread and that thread is locked up until the request terminates. Python has no facility to preemptively stop thread and Zope never times out requests. As a result, it only takes a few long requests to completely lock up a zope server. If you consider that you could have 4 users uploading large files over slow links, you can see how you are in deep trouble even before considering the memory issue. So start by upping the number of threads (-t 15 maybe?) in your start script. On top of that, you are uploading large files which as you already noted zope doesn't handle really well. This topic comes up a lot here. Zope stores objects in memory. These file objects are going to be huge and rapidly consume all your physical memory. So in addition to starving your zope's thread count, you are now swapping like crazy. If I were you, I'd look into doing the file uploads out-of-band (ie, around zope) or buying lots and lots of RAM.

3 2

RE: [Zope] Finding a phone number in a python script
by Schmidt, Allen J. 24 Oct '02

24 Oct '02

I did wind up doing an ex meth. Only about 5 lines and it works well. What really are the risks if it will only be maybe two or three guys who will actually be using RE in py scripts? If there were many developers then I might see the issue. But if you know who is doing stuff on your server, where is the risk? Thanks for the info. -Allen -----Original Message----- From: zope-admin(a)zope.org [mailto:zope-admin@zope.org]On Behalf Of Dennis Allison Sent: Thursday, October 24, 2002 12:28 PM To: Schmidt, Allen J.; list_subscriber(a)neurobs.com Cc: zope(a)zope.org Subject: Re: [Zope] Finding a phone number in a python script Faced with similar problems, I enabled regular expression search in Python Scripts (see the README in the product for instructions). IMHO a risks/reward analysis for the re module easily justifies its inclusion. Andreas Tille wrote: Use an external method. Schmidt, Allen J. wrote: >I have a block of text that contains a phone number in the format of >888/123-4567 and I need to identify that sequence when it occurs and somehow >extract it, and return it to Zope without the slash or the dash. > >Without regex in Python scripts, is there any way to do this? > _______________________________________________ Zope maillist - Zope(a)zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )

1 0

Re: [Zope] Finding a phone number in a python script
by Dennis Allison 24 Oct '02

24 Oct '02

Faced with similar problems, I enabled regular expression search in Python Scripts (see the README in the product for instructions). IMHO a risks/reward analysis for the re module easily justifies its inclusion. Andreas Tille wrote: Use an external method. Schmidt, Allen J. wrote: >I have a block of text that contains a phone number in the format of >888/123-4567 and I need to identify that sequence when it occurs and somehow >extract it, and return it to Zope without the slash or the dash. > >Without regex in Python scripts, is there any way to do this? >

1 0

Zope 2.6 external PRODUCTS_PATH don't work
by Projekt01, Roger Ineichen 24 Oct '02

24 Oct '02

Hello With Zope 2.6 I use a external PRODUCTS_PATH folder. example: set PRODUCTS_PATH=c:\projects\zopeProducts;c:\webapps\zope260\lib\python\Pro ducts; This example works in Zope 2.5.1 whats changed in 2.6 The products in this folder don't find import classes in the real python/lib/Products folder. Mit freundlichen Grüssen Roger Ineichen ___________________________ Projekt01 GmbH www.projekt01.ch Langackerstrasse 8 6330 Cham phone +41 (0)41 781 01 78 mobile +41 (0)79 340 52 32 fax +41 (0)41 781 00 78 email r.ineichen(a)projekt01.ch ___________________________ END OF MESSAGE

3 2

import pyexpat not known in zope 2.6?
by Projekt01, Roger Ineichen 24 Oct '02

24 Oct '02

Hello together I whould like import pyexpat a xml parser in a class. But pyexpat isn't known in zope. I use WinXP, Zope 2.6. Mit freundlichen Grüssen Roger Ineichen ___________________________ Projekt01 GmbH www.projekt01.ch Langackerstrasse 8 6330 Cham phone +41 (0)41 781 01 78 mobile +41 (0)79 340 52 32 fax +41 (0)41 781 00 78 email r.ineichen(a)projekt01.ch ___________________________ END OF MESSAGE

1 0

Re: [Zope] Idea of Caching DTML Methods in Zope
by Elena Schulz 24 Oct '02

24 Oct '02

Just an idea of a newbie, if there is no other caching-possibility maybe creating and using a special folder-object as a subobject of the standard folder-object with a property of the wanted info, i.e. PARENTS[3:], which changes whenever the folder is created and copied elsewhere in the hierachy is a solution. Maybe it can be achieved with the products "AbacadabraObject" and "PropertyFolder" ... To my knowledge it is possible to add a script (i.e. localroot.py) as a zope-property, at least for the property-types "multiple selection" and "selection". HTH Elena

1 0

STX problem
by kosh＠aesaeion.com 24 Oct '02

24 Oct '02

How can I get structured text to leave everything in <dtml-* > alone? Otherwise I get problems of structured text rendering stuff in the dtml and it changes '' to <code></code> which then upsets dtml which can't then render what is in it. The other option is to remove the code that does <code></code> but I would prefer a better solution then that. Thanks Designing and building web applications http://webme-eng.com

1 0

RE: [Zope] <noscript> and dtml
by Passin, Tom 24 Oct '02

24 Oct '02

[ AM ] > solution: > put the following in the <head></head> section of the page. I > didnt know > you could put a <noscript> in there. but it works beutifully. > > <noscript> > <meta HTTP-EQUIV="REFRESH" CONTENT="0; URL=<dtml-var > "errors.no_javascript.absolute_url()">"> > </noscript> > Do not do this, because it is not valid html. A "noscript" element is not allowed in the head element. Browsers may tolerate it, but you should not knowingly create invalid markup. Why not put the redirect in the body element's onLoad event? If javascript is not enabled, nothing will happen. If it is enabled, the javascript-aware page will get loaded. <body onLoad='window.location.replace("javascript-aware-page.html")'> Cheers, Tom P

2 1

Authentication Dialog
by Dylan Reinhardt 23 Oct '02

23 Oct '02

Hi all, Anyone know how/if it is possible to suggest what shows up in the client's login/authentication dialog? I realize this is mostly handled by the browser. Even so, both Opera and IE have a field in the authentication dialog that typically has "Zope" for an entry... so this must get populated by something the server provides. Is there a good way to override the default value for this? TIA, Dylan

2 2