September 2005 - Zope

RE: [Zope] Zope scalabilty and problems
by Kennamore, Matthew G [NTK] 01 Sep '05

01 Sep '05

Here is some of the information I can provide, some I'll have to go check. Database: In the past hour we had 1204 Object Stores and 159433 Loads and 465 connections. -----Original Message----- From: Andrew Sawyers [mailto:andrew@zope.com] Sent: Thursday, September 01, 2005 1:30 PM To: Kennamore, Matthew G [NTK]; 'Chris McDonough' Cc: zope(a)zope.org Subject: RE: [Zope] Zope scalabilty and problems Sorry, I missed the beginning of this thread, I've got some questions that I didn't see answered or asked: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1,0000 writes a day is not outrageous....so it's possible that really what you need to spread the load across a load balanced pool of zeo app servers. How many users are authenticated at any given time actually using the system for writing? Are you running a persistent or non-persistent cache? If so, what size it is? Check the control panel, under database management and choose your database, then look at the activity tab. Do you have a high volume of object loads? What is the volume of object stores? There is also an in-memory zeo object cache - click the cache parameters tab and see the number of objects in your in-memory cache. If your zeo client(s) have lots of ram (they definitely should), you can crank this up as well. What is the 'horespower' of your app server(s)? Andrew -- Zope Managed Hosting Software Engineer Zope Corporation (540) 361-1700 > -----Original Message----- > From: zope-bounces(a)zope.org [mailto:zope-bounces@zope.org] On Behalf Of > Kennamore, Matthew G [NTK] > Sent: Thursday, September 01, 2005 1:19 PM > To: Chris McDonough > Cc: zope(a)zope.org > Subject: RE: [Zope] Zope scalabilty and problems > > Thanks everyone for their suggestions, bought a squid book today and I > know what I'll be doing all weekend <grin> > > -----Original Message----- > From: Chris McDonough [mailto:chrism@plope.com] > Sent: Thursday, September 01, 2005 1:13 PM > To: Kennamore, Matthew G [NTK] > Cc: zope(a)zope.org > Subject: RE: [Zope] Zope scalabilty and problems > > What you've got now is a pretty reasonable setup. Moving to 2.7.7 will > likely not get you any "free" performance increases and it doesn't > appear that there are any "critical" ZODB bugfixes relating to your > problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off > Data.fs" (as you mentioned in your original email) may be quite costly, > requiring a lot of application recoding. It also might not help solve > the scaling problem you're having. It's a "high risk, potentially low > reward" solution. You need something that is low-risk and potentially > high reward, I'd suspect. > > Others have said this, but it bears repeating. The biggest > "administrative" win you can get for the cheapest cost is to add an HTTP > cache (like Squid or even Apache's mod_proxy) in front of your ZEO > clients and to cause your Zope application to set the proper response > headers which allow the cache server to cache (and thus serve) those > pages on subsequent requests. > > This can be tricky when most of your traffic is authenticated because > there is usually very little win and a lot of danger to serving up > cached pages when a page in the cache represents a view that a > privileged user sees when visiting the site. You typically don't want > other people to see the same thing he does. As a result, typically > people will say "pages that require authentication can't be cached" > which isn't entirely true (you can cache them but it's often useless). > > However, even serving up fully static content that doesn't require any > authentication like CSS, images, and so on can be a win. You might > start there. > > HTH, > > - C > > > On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote: > > Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) > > Pyhton is 2.3.4 > > Apache 1.3 > > > > We have 943 users as of this minute with a bout 1000 objects being > created a day (Lots of creates) > > > > -----Original Message----- > > From: Chris McDonough [mailto:chrism@plope.com] > > Sent: Thursday, September 01, 2005 12:51 PM > > To: Kennamore, Matthew G [NTK] > > Cc: zope(a)zope.org > > Subject: Re: [Zope] Zope scalabilty and problems > > > > Your "tranaaction blocked" error messages seem to imply that either or > > both of the following is true: > > > > - some transactions are taking a "long time" > > (any more than, say, 200 milliseconds is a long time) > > > > - you have a very high transaction volume. > > > > That said, these messages are informational rather than signifying an > > error condition. Some transaction blockage is expected since the ZEO > > server can only deal with one transaction at a time. > > > > What version of Zope are you using? > > > > Is this an "intranet" application? Are there lots of users creating > > content? Can you venture a guess as to how many users are "creating" > > content vs. "viewing" content at any given time? Are the users that are > > creating content logged in? The users that are viewing content? > > > > On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote: > > > Here is our situation, we originally launched zope with a small user > > > base in mind and like most IT implementations it got away from us > > > quickly and we have over 1000 users now. We are experiencing huge > > > slowdowns and lots of complaints of performance. > > > > > > Let me outline our system: > > > > > > I have a load balancer that round robins incoming request to 3 zeo > > > servers > > > > > > I have 1 zope server on the back en that severs the data, this is a > > > large single processor machine with 4 gigs of ram > > > > > > Each zeo server is 2 gigs of ram, single processor 3.2 gighz > > > > > > The throughput doesn't appear to be the problem as ram/processors are. > > > > > > Short term I plan to change all zeo's to 4 gigs ram and add 3 > > > additional servers. > > > > > > > > > The Technologies we are using are Zope, Zeo, plone and so forth. Why > > > are we having these large issues? I see sites with hundreds of > > > thousands of users and they don't seem to have these issues? We plan > > > to recode our site to get away from the Data.fs as soon as we can but > > > that is a ways off yet. What am I doing wrong can anyone help? > > > > > > _______________________________________________ > > > Zope maillist - Zope(a)zope.org > > > http://mail.zope.org/mailman/listinfo/zope > > > ** No cross posts or HTML encoding! ** > > > (Related lists - > > > http://mail.zope.org/mailman/listinfo/zope-announce > > > http://mail.zope.org/mailman/listinfo/zope-dev ) > > > > > > > > _______________________________________________ > Zope maillist - Zope(a)zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev )

1 0

Zope scalabilty and problems
by Kennamore, Matthew G [NTK] 01 Sep '05

01 Sep '05

Here is our situation, we originally launched zope with a small user base in mind and like most IT implementations it got away from us quickly and we have over 1000 users now. We are experiencing huge slowdowns and lots of complaints of performance. Let me outline our system: I have a load balancer that round robins incoming request to 3 zeo servers I have 1 zope server on the back en that severs the data, this is a large single processor machine with 4 gigs of ram Each zeo server is 2 gigs of ram, single processor 3.2 gighz The throughput doesn't appear to be the problem as ram/processors are. Short term I plan to change all zeo's to 4 gigs ram and add 3 additional servers. The Technologies we are using are Zope, Zeo, plone and so forth. Why are we having these large issues? I see sites with hundreds of thousands of users and they don't seem to have these issues? We plan to recode our site to get away from the Data.fs as soon as we can but that is a ways off yet. What am I doing wrong can anyone help?

8 7

RE: [Zope] Zope scalabilty and problems
by Kennamore, Matthew G [NTK] 01 Sep '05

01 Sep '05

I am seeing a large amounts of these 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.12:50602 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.23:49161 Blocked transaction restarted. 2005-08-31T09:05:21 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T09:05:22 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:18:26 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:18:28 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:08 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:59:17 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:31 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:13 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T11:00:15 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:16 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 2. -----Original Message----- From: Paul Winkler [mailto:pw_lists@slinkp.com] Sent: Wednesday, August 31, 2005 1:33 PM To: Kennamore, Matthew G [NTK] Cc: zope(a)zope.org Subject: Re: [Zope] Zope scalabilty and problems > Here is our situation, we originally launched zope with a small user > base in mind and like most IT implementations it got away from us > quickly and we have over 1000 users now. We are experiencing huge > slowdowns and lots of complaints of performance. > > Let me outline our system: > > I have a load balancer that round robins incoming request to 3 zeo > servers Just to get the terminology correct: These are Zope servers. They are ZEO *clients*. > I have 1 zope server on the back en that severs the data, this is a > large single processor machine with 4 gigs of ram I think you mean ZEO server, not Zope server. > Each zeo server is 2 gigs of ram, single processor 3.2 gighz That is plenty of hardware for most sites... > Short term I plan to change all zeo's to 4 gigs ram and add 3 additional > servers. That might help if your users' typical usage patterns are unavoidably dynamic: e.g. lots of unique catalog queries, lots of personalization, and/or lots of writes. But it's more typical to have quite a lot of reads of rarely-changing data. In which case, the usual low-cost answer to performance issues is: cache, cache, and more cache. There are several levels of caching, which vary in difficulty of setup and potential performance impact. See below... > The Technologies we are using are Zope, Zeo, plone and so forth. Why > are we having these large issues? I see sites with hundreds of > thousands of users and they don't seem to have these issues? We plan to > recode our site to get away from the Data.fs as soon as we can but that > is a ways off yet. What am I doing wrong can anyone help? Plone is very highly configurable, and every one of those runtime options adds a performance cost. You certainly can run high-performance plone sites on your hardware, you just need to tune it properly. The general steps are: 1) Test your performance. See http://plope.com/Members/chrism/loadtesting-methodology/view?searchterm=per… 2) Profile to see what's slow. 3) Increase caching and/or optimize slow code identified by profiling. Caching is generally a bigger win than optimizing, for less effort. (I have been doing full-time zope work for about 4 years now and I have yet to ever consciously optimize any code ... I try to have efficiency in mind while writing it but that's it... so far there's always been a simple solution via caching. But I don't work on mega-heavy-load sites.) 4) Repeat until satisfied. The best docs I've found on this are: "Scaling Zope", Chris McDonough's presentation from 2003: http://plope.com/misc/szweb Read that one from start to finish at least once. There is also a good chapter on maintaining & optimizing plone in Andy McKay's "Plone book". Another good one is http://web.archive.org/web/20030723200058/http://matthardy.us/mywiki/HighPe… but the original has gone missing so the charts aren't there. -PW

1 0

RE: Zope Digest, Vol 16, Issue 2
by Kennamore, Matthew G [NTK] 01 Sep '05

01 Sep '05

I am seeing a large amounts of these 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.12:50602 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T08:40:18 INFO(0) ZSS:528/10.110.110.23:49161 Blocked transaction restarted. 2005-08-31T09:05:21 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T09:05:22 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:18:26 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:18:28 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:08 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T10:59:17 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T10:59:31 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:13 INFO(0) ZSS:528/10.110.110.12:50602 Blocked transaction restarted. 2005-08-31T11:00:15 INFO(0) ZSS:528/10.110.110.23:49161 Transaction blocked waiting for storage. Clients waiting: 1. 2005-08-31T11:00:16 INFO(0) ZSS:528/10.110.110.22:49161 Transaction blocked waiting for storage. Clients waiting: 2. -----Original Message----- From: zope-bounces(a)zope.org [mailto:zope-bounces@zope.org] On Behalf Of zope-request(a)zope.org Sent: Thursday, September 01, 2005 12:00 PM To: zope(a)zope.org Subject: Zope Digest, Vol 16, Issue 2 Send Zope mailing list submissions to zope(a)zope.org To subscribe or unsubscribe via the World Wide Web, visit http://mail.zope.org/mailman/listinfo/zope or, via email, send a message with subject or body 'help' to zope-request(a)zope.org You can reach the person managing the list at zope-owner(a)zope.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Zope digest..." Today's Topics: 1. Re: Zope scalabilty and problems (Sascha Ottolski) ---------------------------------------------------------------------- Message: 1 Date: Thu, 1 Sep 2005 16:45:11 +0200 From: Sascha Ottolski <sascha.ottolski(a)gallileus.de> Subject: Re: [Zope] Zope scalabilty and problems To: zope(a)zope.org Message-ID: <200509011645.12601.sascha.ottolski(a)gallileus.de> Content-Type: text/plain; charset="windows-1252" Am Mittwoch, 31. August 2005 18:50 schrieb J Cameron Cooper: > There are lots of ways to improve performance just administratively. You > can improve various zope.conf settings, and set up caching (either RAM > or HTTP Accel.) in Plone. You can also put a caching proxy server (like > Squid) in front of your Zopes. not to forget that playing with the zeo and/or zope database caches might help; if you see lots of conflict errors in you logs, may be an upgrade to 2.8 with MVCC does a benefit. and, of course, look carefully if any other tasks on all of your servers compete with your cpu and io resources. Cheers, Sascha P.S. just in case, debug mode or profiling and such are better to be disabled on your live system...:-) ------------------------------ _______________________________________________ Zope maillist - Zope(a)zope.org End of Zope Digest, Vol 16, Issue 2 ***********************************

1 0

Visibility and Copy&Paste support
by Peter Bengtsson 01 Sep '05

01 Sep '05

I've understood that in product development I have to register all my classes with registerClass() that I want to, as instanciated objects, copy/cut & paste. Some of these objects I don't want to encourage people to create via the ZMI and I don't want to clutter up the "Add new object drop down" with stuff that can't be added. If I register the class, but set 'visibility=None' I still can't copy/cut & paste the instanciated objects. Why? What's the point of the 'visibility=None' parameter? -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com

3 3

Sending Fax through MailHost
by Chris Larsen 01 Sep '05

01 Sep '05

Hi, I'm trying to put a fax number into the "To:" field of the mailhost header and am not having much luck. Obviously the standard for this is to address the email to [FAX:18479131145] but when I do [FAX:1<dtml-var faxnum>] it just tells me that there are no recipients designated. Any ideas? Thanks, chris

2 1

RE: [Zope] Excel spreadsheet attachment in dtml-sendmail
by Chris Larsen 01 Sep '05

01 Sep '05

Never mind.. the dtml-boundary looks like this now.. I guess that zopelabs.com how-to needs to be updated.. oh well. <dtml-boundary disposition=attachment type=application/vend.ms-excel encode=base64 filename=ics.xls><dtml-var expr="ics.data"> _____ From: Chris Larsen Sent: Wednesday, August 31, 2005 3:10 PM To: zope(a)zope.org Subject: [Zope] Excel spreadsheet attachment in dtml-sendmail Hi again, Now on to my next problem.. I have tried the below code to send off a spreadsheet as an attachment and am getting a dat file and a 232B file that can't be read. Here's the code from zopelabs.com appropriated to my own info: <dtml-sendmail mailhost=MailHost> To: chris(a)hssc.com From: chris(a)hssc.com Subject: some subject <dtml-mime type=application/msexcel encode=base64> Content-Disposition: inline; filename="ics.xls" <dtml-boundary name=ics.xls type=application/msexcel disposition=attachment encode=base64> </dtml-mime> </dtml-sendmail> I thought they type was supposed to be application/vend.msexcel I've tried both ways tho but to no avail. Thanks in advance, Chris Larsen

2 1