RE: [Zope] Zope scalabilty and problems
Thanks everyone for their suggestions, bought a squid book today and I know what I'll be doing all weekend <grin> -----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: Thursday, September 01, 2005 1:13 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: RE: [Zope] Zope scalabilty and problems What you've got now is a pretty reasonable setup. Moving to 2.7.7 will likely not get you any "free" performance increases and it doesn't appear that there are any "critical" ZODB bugfixes relating to your problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off Data.fs" (as you mentioned in your original email) may be quite costly, requiring a lot of application recoding. It also might not help solve the scaling problem you're having. It's a "high risk, potentially low reward" solution. You need something that is low-risk and potentially high reward, I'd suspect. Others have said this, but it bears repeating. The biggest "administrative" win you can get for the cheapest cost is to add an HTTP cache (like Squid or even Apache's mod_proxy) in front of your ZEO clients and to cause your Zope application to set the proper response headers which allow the cache server to cache (and thus serve) those pages on subsequent requests. This can be tricky when most of your traffic is authenticated because there is usually very little win and a lot of danger to serving up cached pages when a page in the cache represents a view that a privileged user sees when visiting the site. You typically don't want other people to see the same thing he does. As a result, typically people will say "pages that require authentication can't be cached" which isn't entirely true (you can cache them but it's often useless). However, even serving up fully static content that doesn't require any authentication like CSS, images, and so on can be a win. You might start there. HTH, - C On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote:
Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) Pyhton is 2.3.4 Apache 1.3
We have 943 users as of this minute with a bout 1000 objects being created a day (Lots of creates)
-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: Thursday, September 01, 2005 12:51 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: Re: [Zope] Zope scalabilty and problems
Your "tranaaction blocked" error messages seem to imply that either or both of the following is true:
- some transactions are taking a "long time" (any more than, say, 200 milliseconds is a long time)
- you have a very high transaction volume.
That said, these messages are informational rather than signifying an error condition. Some transaction blockage is expected since the ZEO server can only deal with one transaction at a time.
What version of Zope are you using?
Is this an "intranet" application? Are there lots of users creating content? Can you venture a guess as to how many users are "creating" content vs. "viewing" content at any given time? Are the users that are creating content logged in? The users that are viewing content?
On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote:
Here is our situation, we originally launched zope with a small user base in mind and like most IT implementations it got away from us quickly and we have over 1000 users now. We are experiencing huge slowdowns and lots of complaints of performance.
Let me outline our system:
I have a load balancer that round robins incoming request to 3 zeo servers
I have 1 zope server on the back en that severs the data, this is a large single processor machine with 4 gigs of ram
Each zeo server is 2 gigs of ram, single processor 3.2 gighz
The throughput doesn't appear to be the problem as ram/processors are.
Short term I plan to change all zeo's to 4 gigs ram and add 3 additional servers.
The Technologies we are using are Zope, Zeo, plone and so forth. Why are we having these large issues? I see sites with hundreds of thousands of users and they don't seem to have these issues? We plan to recode our site to get away from the Data.fs as soon as we can but that is a ways off yet. What am I doing wrong can anyone help?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Sorry, I missed the beginning of this thread, I've got some questions that I didn't see answered or asked: Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1,0000 writes a day is not outrageous....so it's possible that really what you need to spread the load across a load balanced pool of zeo app servers. How many users are authenticated at any given time actually using the system for writing? Are you running a persistent or non-persistent cache? If so, what size it is? Check the control panel, under database management and choose your database, then look at the activity tab. Do you have a high volume of object loads? What is the volume of object stores? There is also an in-memory zeo object cache - click the cache parameters tab and see the number of objects in your in-memory cache. If your zeo client(s) have lots of ram (they definitely should), you can crank this up as well. What is the 'horespower' of your app server(s)? Andrew -- Zope Managed Hosting Software Engineer Zope Corporation (540) 361-1700
-----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Kennamore, Matthew G [NTK] Sent: Thursday, September 01, 2005 1:19 PM To: Chris McDonough Cc: zope@zope.org Subject: RE: [Zope] Zope scalabilty and problems
Thanks everyone for their suggestions, bought a squid book today and I know what I'll be doing all weekend <grin>
-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: Thursday, September 01, 2005 1:13 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: RE: [Zope] Zope scalabilty and problems
What you've got now is a pretty reasonable setup. Moving to 2.7.7 will likely not get you any "free" performance increases and it doesn't appear that there are any "critical" ZODB bugfixes relating to your problems since 2.7.3. FWIW, also, I'd venture a guess that moving "off Data.fs" (as you mentioned in your original email) may be quite costly, requiring a lot of application recoding. It also might not help solve the scaling problem you're having. It's a "high risk, potentially low reward" solution. You need something that is low-risk and potentially high reward, I'd suspect.
Others have said this, but it bears repeating. The biggest "administrative" win you can get for the cheapest cost is to add an HTTP cache (like Squid or even Apache's mod_proxy) in front of your ZEO clients and to cause your Zope application to set the proper response headers which allow the cache server to cache (and thus serve) those pages on subsequent requests.
This can be tricky when most of your traffic is authenticated because there is usually very little win and a lot of danger to serving up cached pages when a page in the cache represents a view that a privileged user sees when visiting the site. You typically don't want other people to see the same thing he does. As a result, typically people will say "pages that require authentication can't be cached" which isn't entirely true (you can cache them but it's often useless).
However, even serving up fully static content that doesn't require any authentication like CSS, images, and so on can be a win. You might start there.
HTH,
- C
On Thu, 2005-09-01 at 11:54 -0500, Kennamore, Matthew G [NTK] wrote:
Zope version 2.7.3 (planning to goto 2.7.7 soon with ZODB 3.2.9) Pyhton is 2.3.4 Apache 1.3
We have 943 users as of this minute with a bout 1000 objects being created a day (Lots of creates)
-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: Thursday, September 01, 2005 12:51 PM To: Kennamore, Matthew G [NTK] Cc: zope@zope.org Subject: Re: [Zope] Zope scalabilty and problems
Your "tranaaction blocked" error messages seem to imply that either or both of the following is true:
- some transactions are taking a "long time" (any more than, say, 200 milliseconds is a long time)
- you have a very high transaction volume.
That said, these messages are informational rather than signifying an error condition. Some transaction blockage is expected since the ZEO server can only deal with one transaction at a time.
What version of Zope are you using?
Is this an "intranet" application? Are there lots of users creating content? Can you venture a guess as to how many users are "creating" content vs. "viewing" content at any given time? Are the users that are creating content logged in? The users that are viewing content?
On Wed, 2005-08-31 at 11:03 -0500, Kennamore, Matthew G [NTK] wrote:
Here is our situation, we originally launched zope with a small user base in mind and like most IT implementations it got away from us quickly and we have over 1000 users now. We are experiencing huge slowdowns and lots of complaints of performance.
Let me outline our system:
I have a load balancer that round robins incoming request to 3 zeo servers
I have 1 zope server on the back en that severs the data, this is a large single processor machine with 4 gigs of ram
Each zeo server is 2 gigs of ram, single processor 3.2 gighz
The throughput doesn't appear to be the problem as ram/processors are.
Short term I plan to change all zeo's to 4 gigs ram and add 3 additional servers.
The Technologies we are using are Zope, Zeo, plone and so forth. Why are we having these large issues? I see sites with hundreds of thousands of users and they don't seem to have these issues? We plan to recode our site to get away from the Data.fs as soon as we can but that is a ways off yet. What am I doing wrong can anyone help?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Andrew Sawyers wrote:
Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1,0000 writes a day is not outrageous....
Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Andrew Sawyers wrote:
Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1,0000 writes a day is not outrageous....
Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc...
cheers,
Chris
This sounds like the classic abstraction vs performance debate. In the late 80s some people criticised C++ and OOP versus C for the same reasons. Might it not be better, rather than telling people not to use Plone + add-ons for this reason, to just push for across the board performance improvements, which means removing bottlenecks in Plone, Archetypes, specific Products, as well as the underlying Zope/CMF ? Also some of the products you are talking about may be somewhat immature and you know what they say, don't do premature optimisation, do logic first, performance tune later. What you describe as "appalling" might perhaps be able to be performance tuned? BTW CPSSkins is another performance hit. Would you care to list the worst culprits? ;-) Regards Nick
Hi I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? I'm using CMFMember also-why is it 'apalling'? On 9/2/05, N.Davis <nd51@le.ac.uk> wrote:
Chris Withers wrote:
Andrew Sawyers wrote:
Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1,0000 writes a day is not outrageous....
Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc...
cheers,
Chris
This sounds like the classic abstraction vs performance debate. In the late 80s some people criticised C++ and OOP versus C for the same reasons.
Might it not be better, rather than telling people not to use Plone + add-ons for this reason, to just push for across the board performance improvements, which means removing bottlenecks in Plone, Archetypes, specific Products, as well as the underlying Zope/CMF ? Also some of the products you are talking about may be somewhat immature and you know what they say, don't do premature optimisation, do logic first, performance tune later.
What you describe as "appalling" might perhaps be able to be performance tuned?
BTW CPSSkins is another performance hit. Would you care to list the worst culprits? ;-)
Regards Nick
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
michael nt milne wrote at 2005-9-2 09:13 -0400:
I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise?
No, it is not. It is just that the CMF is unable to fix a year long bug which affects Windows only when Zope is run in debug mode: then each access to a "filesystem directory view" triggers a hierarchical file system scan. The result is a slowdown by several orders of magnitude. The problem disappears when you either: * do not run on Windows * turn off "debug mode" * fix the silly bug Of course, a Plone site is not as fast as a static one. But, I hope, you expect this... -- Dieter
Hi Dieter, Dieter Maurer wrote:
I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise?
No, it is not.
...I agree, if it is correctly set up and optimised, which isn't the case if someone is here complaining about performance, which sadly, new users blown away by Plone's functionality often are...
It is just that the CMF is unable to fix a year long bug which affects Windows only when Zope is run in debug mode:
then each access to a "filesystem directory view" triggers a hierarchical file system scan.
The result is a slowdown by several orders of magnitude.
FWIW, my experiences with Plone are predominantly on Linux, and that's what my comments are based on...
The problem disappears when you either:
* turn off "debug mode"
Yes, this is a biggie. Thankfully, I believe newer versions of Zope ship with this turned off by default? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Dieter Maurer wrote:
I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise?
No, it is not.
It is just that the CMF is unable to fix a year long bug
It's not "CMF"'s fault, it's only the responsibility of Windows developers running CMF. I for one couldn't care less.
which affects Windows only when Zope is run in debug mode:
then each access to a "filesystem directory view" triggers a hierarchical file system scan.
The result is a slowdown by several orders of magnitude.
I believe Tres has now fixed this. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Florent Guillaume wrote:
Dieter Maurer wrote:
I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise?
No, it is not.
It is just that the CMF is unable to fix a year long bug
It's not "CMF"'s fault, it's only the responsibility of Windows developers running CMF. I for one couldn't care less.
which affects Windows only when Zope is run in debug mode:
then each access to a "filesystem directory view" triggers a hierarchical file system scan.
The result is a slowdown by several orders of magnitude.
It got *much* worse when CMF shifted to use subversion, which has many more bookkeeping files in the .svn directories. Plone developers who work on Windows sent up an anguished cry just after we switched over, while they were trying to prepare their 2.1 release candidate.
I believe Tres has now fixed this.
I applied a tourniquet, which was the major reason for pushing out the 1.5.3 release. Mark Hammond volunteered (at the Plone Symposium in New Orleans) to look at the feasibility of a better solution which would use the native Win32 file monitoring APIs: http://www.zope.org/Collectors/CMF/367 As Chris noted, running with 'debug-mode off' also avoids this issue, if you can't upgrade CMF for some reason (and you *should* be running this way if performance matters, anyway). Tres - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDHHG6+gerLs4ltQ4RAl49AJ9+R/aUhA/2UvEWEsRZKIqMnf7fRwCglVT9 4iuXchoDl+Bx6rLMrvd91JI= =Gx7s -----END PGP SIGNATURE-----
Florent Guillaume wrote at 2005-9-5 16:49 +0200:
...
It is just that the CMF is unable to fix a year long bug
It's not "CMF"'s fault, it's only the responsibility of Windows developers running CMF. I for one couldn't care less.
Really? That the CMF makes an hierarchical filesystem scan for each access to a filesystem directory view should be the responsibility of the Windows developers running CMF? There are no bugs: if you meat one, it proves that you use the wrong system...
...
then each access to a "filesystem directory view" triggers a hierarchical file system scan.
The result is a slowdown by several orders of magnitude.
I believe Tres has now fixed this.
It was not yet fixed when I made an svn checkout of the CMF 1.5 branch about 2 weeks ago. -- Dieter
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dieter Maurer wrote:
Florent Guillaume wrote at 2005-9-5 16:49 +0200:
...
It is just that the CMF is unable to fix a year long bug
It's not "CMF"'s fault, it's only the responsibility of Windows developers running CMF. I for one couldn't care less.
Really?
That the CMF makes an hierarchical filesystem scan for each access to a filesystem directory view should be the responsibility of the Windows developers running CMF?
There are no bugs: if you meat one, it proves that you use the wrong system...
...
then each access to a "filesystem directory view" triggers a hierarchical file system scan.
The result is a slowdown by several orders of magnitude.
I believe Tres has now fixed this.
It was not yet fixed when I made an svn checkout of the CMF 1.5 branch about 2 weeks ago.
The patch I merged for the CMF 1.5.2 release (back in late July), did not completely remove the 'os.pathwalk': it merely made the exclusion of the .svn directory work cleanly. The better fix would be to use the Win32 APIs for file monitoring; Mark Hammond volunteered to look into that. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDHXCv+gerLs4ltQ4RAgShAKCuvxwdqHpYGUQMLIxTUT1YGXbbqwCguFIH KANLJ4XcxzwPMnhcYI6raE8= =94l4 -----END PGP SIGNATURE-----
Chris has opinions on performance based on his experiences. Others have different viewpoints. Here is an more empirical treatment of what you might expect: http://www.ifpeople.net/resources/downloads/PlonePerformance.pdf --Paul michael nt milne wrote:
Hi I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise? I'm using CMFMember also-why is it 'apalling'?
On 9/2/05, N.Davis <nd51@le.ac.uk> wrote:
Chris Withers wrote:
Andrew Sawyers wrote:
Did Matt indicate if he was running multiple zeo app servers? It might help to be spreading the load. 1,0000 writes a day is not outrageous....
Yes, but he's using Plone, which implies not only all of the CMF reindexing overhead, but also all the AT and Plone layers on top. That could get even worse if he's gone to town with a workflow tool or is using any of the truly appalling Plone addons, such as CMFMember, CMFForum, etc...
cheers,
Chris
This sounds like the classic abstraction vs performance debate. In the late 80s some people criticised C++ and OOP versus C for the same reasons.
Might it not be better, rather than telling people not to use Plone + add-ons for this reason, to just push for across the board performance improvements, which means removing bottlenecks in Plone, Archetypes, specific Products, as well as the underlying Zope/CMF ? Also some of the products you are talking about may be somewhat immature and you know what they say, don't do premature optimisation, do logic first, performance tune later.
What you describe as "appalling" might perhaps be able to be performance tuned?
BTW CPSSkins is another performance hit. Would you care to list the worst culprits? ;-)
Regards Nick
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
michael nt milne wrote:
I am setting up sites using Plone and am concerned to read some of the comments on it. Is it really that bad performance wise?
"it depends", unless you have a lot of experience with it, Zope and CMF, you might well find yourself with a slow site on your hands and not many apparent options ;-)
I'm using CMFMember also-why is it 'apalling'?
It's an incredibly heavyweight and inefficient solution to a problem that requires an extremely fast and efficient solution: user object are used a LOT in most Zope apps, CMFMember makes them slow, in a numbner of ways. Generating writes which result in AT-based member objects being reindexed and often resulting in conflicterrors that dramatically slow down a site is not a good thing... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
participants (9)
-
Andrew Sawyers -
Chris Withers -
Dieter Maurer -
Florent Guillaume -
Kennamore, Matthew G [NTK] -
michael nt milne -
N.Davis -
Paul Everitt -
Tres Seaver