June 2002 - Zope - Zope lists

Serious write-performance issue with multiple threads.
by Arnar Lundesgaard 18 Jun '02

18 Jun '02

Hi, For some time now we have had trouble with write performance on two ZOPE sites we have in operation. The first one has been in production since early April, while the second is being opened to the public in a few days. They are both based on our internally developed TopicMap engine (built on top of CMF). (To be OpenSourced when we have some available time to finish it up.) The problem has been scarily easy to reproduce, and manifests itself whenever multiple persons are working (writing articles etc). As long as only one person is working on the site, it has the expected performance. Not very fast, a few seconds per write operation, but acceptable. However as soon as more people log in and start publishing, we often experience hangs which may last for 10-15 minutes on write operations that usually takes only a few seconds to perform. In this situation Zope doesn't respond to any requests. Fortunately visitors to the first site are served by the cache (squid on the same machine) and reads from ZOPE have ranged from very fast to mostly quickly. Although we do not have any other ZOPE sites with the same write-load as these sites, the TopicMap engine has naturally been our main suspect. (Load on the first site is about the same as 'zope.org' but with much higher peaks.) It seemed obvious that something was/is with the design of our TopicMap engine that triggers this. We have been reviewing our own code and made many optimizations that we expected to yield significant speedups. While they did make the site faster on average, write operations still often result in long hangs when multiple users were working on either of the sites. We followed the procedure in http://www.zope.org/Members/4am/debugspinningzope. The hang is indeed a spinning process (thread) as it always uses 99.9% CPU when Zope stops responding; attaching to the main process in a hang situation and looking at the responsible thread invariably shows it to be in chunk_free() in libc somewhere downstream from pickleCache. Zope will usually "unhang" itself after about 10-15 minutes of spinning. Mr. Kromers comments on ZOPE and SMP in the 'system requirements' thread a couple of weeks ago gave us a few clues, which we followed. We tried binding the ZOPE instance to one CPU using the affinity patch for Linux 2.4, but that did not help either. We then tried disabling one CPU, suspecting SMP trouble, but still no go. So slowly ZOPE became the suspect. Tried different number of threads, 20, 10, 4, 2... then we we tried running with only one thread, and the "hang" problems vanished! What's more; finally we were getting the performance we expected. It works very well now, even under relatively high load and a lot of write activities. The hardware and software of the two sites: Site 1: Site 2: A dual Xeon 1.2 GHZ A dual Xeon 1.0 GHZ 2 GB ram 512 MB ram RAID 1+5 RAID 1+5 CVS ca. ZOPE 2.5.0 ZOPE 2.5.1 CVS ca. CMF 1.2 CMF 1.2 Python 2.1.2 (PThreads) Python 2.1.3 Linux 2.4.14 (XFS) Linux 2.4.18 (ext3) Python is compiled with thread support and large file support. for both sites The database for site 1 is approx 450 MB newly packed - ca. 220 000 objects; site 2 isn't live yet and is much smaller (~100MB packed). We're experiencing the same problems on both sites. The second site will be going live in a few days. Due to its design and requirements we are not able to cache as much content. This forces us to look for a different solution; the one-thread option won't cut it; too many requests will have to go outside the cache... Currently we are running this on a single thread, but we expect that to "kill us" once it is opened to the general public :( A possible workaround is of course to run two different ZOPE instances with a ZEO backend. One with multiple threads for reads and visits, the other with a single thread for writing and publishing. This is perhaps the ideal solution, but we are loath to make untested changes to the production environment just before going live. Secondly this feels like a bug in ZOPE or Python, and if it is we would like to track it down. What we are looking for is information on threading in Python/ZOPE/ZODB, other peoples experiences, workarounds, etc. Regards, Arnar Lundesgaard ---------------------------------- phone: (+47) 982 38 036 mailto:arnar.lundesgaard(a)creuna.no Creuna as Bryggegata 3 NO-0250 Oslo phone office: (+47) 23 23 88 00 fax: (+47) 23 23 88 50 http://www.creuna.no/

3 2

External Editor on Win32 -- downloading files with .html extension
by Grégoire Weber 18 Jun '02

18 Jun '02

Hi Casey, cc: hi All (on the list), I like your ExternalEditor product! It's really cool! I'am working under Win2k with Zope 2.5.1 and ExternalEditor 0.3. I have some problems editing object externaly. I could not solve them by reading the documenttaion or the mailing list. Netscape 4.79 ------------- With NS 4.79 it works as expected as long as zope objects do not have well known extensions like 'html'. I have to set the 'html' extension on objects because I'm working also with dreamweaver (via ftp/WebDAV). What happens if I click on the pencil icon of an object **with** .html as extension: NS 4.79 opens a alert box asking if the file has to be opened with mozilla.exe or saved to disk. (mozilla is my default browser for html-files) Seems to be a browser problem. The http headers of a file with or without a html extension are the same (I looked at them with tcpwatch). NS 4.79 Config: Application (correct IMO) Extension: ZOPE MIME type: application/x-zope-edit Handled by Application: d:\prog\Python22x\pythonw.exe D:\prog\ZopeEdit\zopeedit.py "%1" "%s" Mozilla ------- With Mozilla (1.0rc1) It doesn't even work to edit files without extensions. Instead I get the follwoing message box (it seems to be a mozilla box): Opening login_form.html <temp path>/login_form.html could not be opened, because an unknown error occured. Sorry about that. Try saving to disk first and then opening the file. The file containing the authentification information can be found in the temp directory (wasn't cleared). Mozille 1.0rc1 Config: Helper Applications (correct IMO) Extension: zope MIME type: application/x-zope-edit Handled by Application: d:\prog\Python22x\pythonw.exe D:\prog\ZopeEdit\zopeedit.py "%1" If you send me some instructions I woudl assist in finding the problems. Greetings, Greg ------------------------------------------------------------------------------- ExternalEditor ini file: # Zope External Editor helper application configuration [general] # General configuration options # Uncomment and specify an editor value to override the editor # specified in the environment editor = d:\prog\pfe32\pfe32.exe /m # editor = d:\winnt\system32\notepad.exe # works! # Automatic save interval, in seconds. Set to zero for # no auto save (save to Zope only on exit). save_interval = 1 # Temporary file cleanup. Set to false for debugging or # to waste disk space. Note: setting this to false is a # security risk to the zope server cleanup_files = 1 # Use WebDAV locking to prevent concurrent editing by # different users. Disable for single user use or for # better performance use_locks = 1 # Specific settings by content-type or meta-type. Specific # settings override general options above. Content-type settings # override meta-type settings for the same option. [meta-type:DTML Document] extension=.dtml [meta-type:DTML Method] extension=.dtml [meta-type:Script (Python)] extension=.py [meta-type:Page Template] extension=.pt [meta-type:Z SQL Method] extension=.sql [content-type:text/*] extension=.txt [content-type:text/html] editor = d:\prog\pfe32\pfe32.exe /m # this was only a try - didn't help [content-type:text/html] extension=.html [content-type:text/xml] extension=.xml [content-type:image/*] editor = D:\prog\PaintShopPro6\Psp.exe [content-type:image/gif] extension=.gif [content-type:image/jpeg] extension=.jpg [content-type:image/png] extension=.png _____________________________________ Grégoire Weber mailto:gregoire.weber@switzerland.org

3 4

Re: [Zope] Dreamweaver MX rewrites ZPT XML?
by hpinson＠indepthl.com 18 Jun '02

18 Jun '02

> I am not familiar with Windows Controller, but > it's probably passing arguments to Z2.py the same way start.bat > would. Could be. I think the Zope Windows Controller NT service is some compiled form of the start.bat. I wish I had never installed that Windows Controller, and it has no uninstaller (frown). It kills the default Zope service in favor of its own. It's a real black box. Have to spend some time with its documentation I guess. I hope there is some! >> Array David, you are right about the array-- Zope won't even start on W2K if you specify the WebDAV source port in array brackets, like in the example. >> Dreamweaver MX This is really an improvement over previous versions-- I hope I can get it to deal with ZPT. Otherwise the WebDAV problems of previous versions seem to have been fixed. It still will not see extensionless objects-- at least as far as I have determined. Harlow Pinson Indepth Learning Email: hpinson(a)indepthl.com Web: http://www.indepthl.com Voice: 505-994-2135 FAX: 208-475-7678

1 0

Zope on Solaris
by Dennis Heltzel 18 Jun '02

18 Jun '02

Update! It's up and running fine on Solaris 8. Converting it to a zip file did the trick. If you are interested in creating a zip file for downloading, here are the Linux commands I used: tar xvzf Zope-2.5.1-solaris-2.6-sparc.tgz zip -ry Zope-2.5.1-solaris-2.6-sparc.zip Zope-2.5.1-solaris-2.6-sparc The resulting file, Zope-2.5.1-solaris-2.6-sparc.zip, was slightly larger (8480279 bytes) than the .tgz file, but so much easier to install on a stock Solaris system. I understand that Solaris 9 will include a lot of GNU tools, and all I can say is "it's about time". Thanks for your help with this. Dennis

1 0

Zope for Solaris
by Dennis Heltzel 18 Jun '02

18 Jun '02

I tried several of the Zope Solaris binaries from your site and they all give me a directory checksum error during the "tar xvf" part of the install (they gunzip just fine). If this is a general problem, you might want to at least remove the files from your site. BTW, the Linux binaries work great, and I am quite impressed with Zope so far. If I convince my company to deploy it as part of our intranet portal, I will probability need to get it running on Solaris 8. Dennis

5 4

ANNOUNCE: Pound - reverse proxy and load balancer - v0.2
by Robert Segall 18 Jun '02

18 Jun '02

This is to announce the release of Pound v0.3. Changes from previous version: - added HTTP/1.1 handling, resulting in significant speed-ups. - added HTTP/1.1 to HTTP/1.0 proxying: if the client is 1.1 compliant, Pound will accept multiple requests per connection, while dealing with the need to re-open connections to a back-end HTTP/1.0 server. - fixed a few minor bugs. Many thanks to Ken Lalonde ken(a)torus.ca for his very helpful suggestions. What is Pound (from the Web page) - a reverse-proxy: it passes requests from client browsers to one or more back-end servers. - a load balancer: it will distribute the requests from the client browsers among several back-end servers, while keeping session information. - an SSL wrapper: Pound will decrypt HTTPS requests from client browsers and pass them as plain HTTP to the back-end browsers. - an HTTP/HTTPS sanitizer: Pound will verify requests for correctness and accept only well-formed ones. - an HTTP/1.1 to 1.0 proxy. For further information, download, etc. please see http://www.apsis.ch/pound The software is at version 0.3. Further testing (especially under heavy loads), improvements and suggestions are welcome. Pound is distributed under the GPL. Enjoy and let me know how it works for you. -- Robert Segall Apsis GmbH Postfach, Uetikon am See, CH-8707 Tel: +41-1-920 4904

1 0

Help with Selection Lists and ZSQL methods
by Schmidt, Allen J. 18 Jun '02

18 Jun '02

I'm stumped... And the problem is how to ask the question so it makes sense... I have a page that has 6 little forms (year, make, model, bodytype, etc.) that each contain a single Select/Option list from a ZSQL method. Each has been using an onSubmit() to process the form and returns a page of resultset data. Each other select list only contains valid selections from the resultset. Each form contains a hidden variable to pass previous selections on to each subsequent query. Works well for single selections at each pass. Now I want to do multiple selections for each form. If I select 2 bodytypes (2 DR SEDAN and PICKUP TRUCK) I get the resultset fine. The URL looks like this: /blahblah?bt=2+DR+SEDAN&bt=PICKUP+TRUCK Then if I want to select from this resultset only for the year of 2000, the URL transforms itself and the percent values for the '[' and ']' and the single quotes of the list are in the URL and are passed to the ZSQL method and no results. I have tried the various methods of transforming the variable (html_quote, url_qoute) and nothing seems to work. The URL is still malformed. The above is rather confusing but not sure of the best way to describe the problem but its pretty close. Can anyone shed some light? Or point me in the right direction? THANKS!!!! -Allen

1 0

External Editor on WinXP/Zope 2.5.1b1
by Juergen R. Plasser / Hexagon 18 Jun '02

18 Jun '02

Hi Casey, I have finally managed to add the x-zope-edit mime type to my WinXP/IE6 system (I have done it manually by editing the registry and I don't know if there is a more simple way to do that in WinXP), but I get the following error when I download the document test.txt by clicking the pencil icon: C:\ExternalEditor>c:\Python22\python zopeedit.py c:\ExternalEditor\test.txt FATAL ERROR: unsupported operand types for +: 'NoneType' and 'str' Traceback (most recent call last): File "zopeedit.py", line 685, in ? ExternalEditor(input_file).launch() File "zopeedit.py", line 286, in launch command = self.getEditorCommand() File "zopeedit.py", line 250, in getEditorCommand classname+'\\Shell\\Open\\Command') TypeError: unsupported operand types for +: 'NoneType' and 'str' What is wrong? Should I upgrade to Zope 2.5.1 [I don't think so]? The beta was here and I thought it would do the job for testing your External Editor product. Regards, Juergen

2 2

Re: [Zope] win32 mysql database adapter
by Day Irmiter 18 Jun '02

18 Jun '02

Here is what finally worked for me (under Win98). To use Zope with a MySQL database under Windows, do the following. 1. download the following packages. the MySQL for Windows installer, version 3.23.39 (or newer) at http://www.mysql.com/downloads/. Andy Dustman's Zope database adapter for MySQL, version 2.0.8 at http://www.zope.org/Members/adustman/Products/ZMySQLDA the python extensions package for MySQL, version 0.9.0 (or newer) at http://prdownloads.sourceforge.net/mysql-python/ Isaac Salsberg's BuildMySQLDA package at http://www.zope.org/Members/isalsberg/MySQLDA/BuildMySQLDA.zip 2. Install MySQL. 3. Decompress the ZMySQLDA package in Zope's root directory. 4. Decompress the mysql-python package to a temp directory. From this temp directory, move the following files to the <Zope_root>\bin directory. _mysql_exceptions.py CompatMysqldb.py The whole directory MySQLdb 5. Decompress the BuildMySQLDA package and move _mysql.dll to the <Zope_root>\bin directory. Restart Zope and you should be able to add a new "ZMySQL DA Connection".

1 0

FW: [Zope] another DTML ques
by Chris Beaven 18 Jun '02

18 Jun '02

Here's his code -----Original Message----- From: Aseem Mohanty [mailto:aseem@neurobehavioralsystems.com] Sent: Tuesday, 18 June 2002 2:36 a.m. To: Chris Beaven Subject: Re: [Zope] another DTML ques I return the list via a python function: import string s = str(s_arg) s = s.replace('[','') s = s.replace(']','') s = s.replace('\'','') s = s.replace('"','') ret = s.split(',') return ret s_arg needs to look like "['test','test_str','yes']" the str() is primarily to get around an attribute error any leads as to why it might be failing... I did check it out on the interpreter and it returns a list. -AM Chris Beaven wrote: >I just set up the same situation and it worked for me... (ie I get YES back) >My guess is that you aren't returning a list. > >My get_list() looked like: >return ['test','test_str','yes'] > >-----Original Message----- >From: Aseem Mohanty [mailto:aseem@neurobehavioralsystems.com] >Sent: Tuesday, 18 June 2002 2:19 a.m. >To: zope(a)zope.org >Subject: [Zope] another DTML ques > > > I have a PyScript that returns a list of strings > >Somehow I am unable to do the following: > ><dtml-let ll="get_list()"> > <dtml-if "'test_str' in ll"> > YES > <dtml-else> > NO > </dtml-if> ></dtml-let> > >I know that 'test_str' is in the list. The if test fails always. >What am I doing wrong?? > >TIA >AM > -- ================================================================== Aseem Mohanty Neurobehavioral Systems Inc, 828 San Pablo Ave, Albany, CA 94706 (R) 510 7696011 (M) 510 3014871 (O) 510 5279231 ================================================================== "I saw `cout' being shifted "Hello world" times to the left and stopped right there!!" -- Steve Gonedes ==================================================================

3 2