Zope vs Plone: performance issues!
Hi all, I'm responsible for a big corporate intranet and corporate portal. They were using Zope/Plone before I got here and the problems are too many. Let's talk about performance issues... Plone is VERY SLOW responding requests. All content is stored in ZODB objects and zcatalog is used to search. But even if you request the intranet home only, it takes about 10 seconds to load. The hardware is very good and lots of ram is used in the servers. They also have zeo + 2 zope instances running in the same machine. I've never seen this in my life... only 3 machines running 2 zope instances + zeo instance, separately. Do you think Zope(logic)+MySQL(content)+Apache(HTTP server) is better solution than Zope+Plone ??? TIA -- Hugo Ramos - ramosh@gmail.com Senior Zope consultant
Hugo Ramos wrote:
Hi all,
I'm responsible for a big corporate intranet and corporate portal. They were using Zope/Plone before I got here and the problems are too many.
Let's talk about performance issues... Plone is VERY SLOW responding requests. All content is stored in ZODB objects and zcatalog is used to search. But even if you request the intranet home only, it takes about 10 seconds to load. The hardware is very good and lots of ram is used in the servers.
They also have zeo + 2 zope instances running in the same machine. I've never seen this in my life... only 3 machines running 2 zope instances + zeo instance, separately.
Do you think Zope(logic)+MySQL(content)+Apache(HTTP server) is better solution than Zope+Plone ???
I think the problem has more to do with Plone than with Zope or the ZODB. We have a couple 'barebones' zope applications here that serve public website. Hosting is on a shared server (behind apache) with no zeo, no cache, no tuning, no fancy configuration, and performances are pretty good - even with search. For the record, we've tried Plone on the same server, and even with almost no content, it *is* *very* slow. -- bruno desthuilliers développeur bruno@modulix.org http://www.modulix.com
bruno desthuilliers wrote:
I think the problem has more to do with Plone than with Zope or the ZODB. We have a couple 'barebones' zope applications here that serve public website. Hosting is on a shared server (behind apache) with no zeo, no cache, no tuning, no fancy configuration, and performances are pretty good - even with search. For the record, we've tried Plone on the same server, and even with almost no content, it *is* *very* slow.
Me? I couldn't _possibly_ comment ;-) Chris . o O ( me? say Plohn is slow? no... never....) -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Me? I couldn't _possibly_ comment ;-)
Mr. Chris "FU" Withers, you might very will think so. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science
On 1/24/06, Hugo Ramos <ramosh@gmail.com> wrote:
Hi all,
I'm responsible for a big corporate intranet and corporate portal. They were using Zope/Plone before I got here and the problems are too many.
Let's talk about performance issues... Plone is VERY SLOW responding requests. All content is stored in ZODB objects and zcatalog is used to search. But even if you request the intranet home only, it takes about 10 seconds to load. The hardware is very good and lots of ram is used in the servers.
They also have zeo + 2 zope instances running in the same machine. I've never seen this in my life... only 3 machines running 2 zope instances + zeo instance, separately.
Do you think Zope(logic)+MySQL(content)+Apache(HTTP server) is better solution than Zope+Plone ???
Yes! But not good enough. Most performance issues with Plone can be solved by caching. Don't underestimate it. I know some companies that use Plone for content management but push out the content in a very non-Plone way and cache that heavily and you get a great result. But, you're not going to get away from the 10 sec catalog search problem. I doubt a ZCatalog search takes this long. Once in there it should be possible to do a search on millons of bytes of text under 1 sec. I have a ZCatalog instance here with roughly 55,000 cataloged objectindices which corresponds to about 1.3Gb of text data. When the IssueTrackerProduct does a search on this, the function _searchCatalog() does 6 different searches on that catalog each!! and here's a copy of some debug output Search took 0.149714946747 Search took 0.271862030029 Search took 0.0381798744202 Search took 0.0339410305023 Search took 0.0346419811249 Search took 0.0137779712677 Search took 0.0435471534729 It's a 1Gb RAMed, Intel Mobile 1.6Mhz thinkpad laptop!! Conclusion, ZCatalog is very fast and so is a correctly "cache-headed" website with Squid.
TIA
-- Hugo Ramos - ramosh@gmail.com Senior Zope consultant _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On 1/24/06, Peter Bengtsson <peter@fry-it.com> wrote:
On 1/24/06, Hugo Ramos <ramosh@gmail.com> wrote:
Hi all,
I'm responsible for a big corporate intranet and corporate portal. They were using Zope/Plone before I got here and the problems are too many.
Let's talk about performance issues... Plone is VERY SLOW responding
<snip>
Yes! But not good enough. Most performance issues with Plone can be solved by caching. Don't underestimate it.
Yes Plone is slow but with caching you get very good performance. I have boosted performance on plone sites from default 1-3req/s to 100req/s and then it was the bandwith that was the bottleneck. The sites where serving both anonymous and authenticated content. I recommend to check out the CacheFu product and documentation in the collective. /Sasha Lovely Systems - www.lovelysystems.com
Well... They were already using Zope's cache and Squid to proxy some static HTML I guess the problem is Plone itself... Skins etc... On 1/24/06, Sasha Vincic <sasha.vincic@gmail.com> wrote:
On 1/24/06, Peter Bengtsson <peter@fry-it.com> wrote:
On 1/24/06, Hugo Ramos <ramosh@gmail.com> wrote:
Hi all,
I'm responsible for a big corporate intranet and corporate portal. They were using Zope/Plone before I got here and the problems are too many.
Let's talk about performance issues... Plone is VERY SLOW responding
<snip>
Yes! But not good enough. Most performance issues with Plone can be solved by caching. Don't underestimate it.
Yes Plone is slow but with caching you get very good performance. I have boosted performance on plone sites from default 1-3req/s to 100req/s and then it was the bandwith that was the bottleneck. The sites where serving both anonymous and authenticated content. I recommend to check out the CacheFu product and documentation in the collective.
/Sasha Lovely Systems - www.lovelysystems.com
-- Hugo Ramos - ramosh@gmail.com http://otugga.blogspot.com/
On 1/24/06, Hugo Ramos <ramosh@gmail.com> wrote:
Well... They were already using Zope's cache and Squid to proxy some static HTML I guess the problem is Plone itself... Skins etc...
As I said and Limi, check out CacheFu. With it you can cache most of your site not just "some static HTML" and you cache it with Squid so Zope will have more time for the none cachable stuff. Doing profile on your site is something you should always do when optimizing. Follow Dieters recommendation and then CacheFu.
Hugo Ramos wrote:
Well... They were already using Zope's cache and Squid to proxy some static HTML I guess the problem is Plone itself... Skins etc...
eh, sorry for probably stating the obvious. you menationed in an earlier mail that you have good hardware. If this hardware is multi-cpu then I suggest that you use some tool (i.e. taskset on linux) to bind each zope python process to a single cpu. Threaded python applications does not work well in multi-cpu settings where there is a chance of a threaded python process being spread out over more than one CPU. I causes a, sometimes signinficant, performance loss. Just by binding the zope python processes to a single CPU (not necessarily the same, just make sure that any single python process is not spread over more than one CPU) you will notice a preformance increase. And like all the others said, Caching is you friend. Start by checking the ZODB cache numbers in the zope-conf file (too start simple). IIRC, the default is 5000 objects. Increase that to, say 50.000 or so (you mention that you have a busy site). Good luck. /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley
On Wed, 25 Jan 2006 00:11:04 -0800, Dario Lopez-Kästen <dario@ita.chalmers.se> wrote:
Start by checking the ZODB cache numbers in the zope-conf file (too start simple). IIRC, the default is 5000 objects. Increase that to, say 50.000 or so (you mention that you have a busy site).
Sorry, but this is not good advice. 5000 objects in the cache is normally plenty, and you should look into caching the front-end before you even consider this. 50K objects in the cache will eat *a lot* of memory, and most likely cause swapping. Cache properly with Apache or Squid in front first, then investigate the more complex settings. -- _____________________________________________________________________ Alexander Limi · Chief Architect · Plone Solutions · Norway Consulting · Training · Development · http://www.plonesolutions.com _____________________________________________________________________ Plone Co-Founder · http://plone.org · Connecting Content Plone Foundation · http://plone.org/foundation · Protecting Plone
On 25 Jan 2006, at 10:42, Alexander Limi wrote:
On Wed, 25 Jan 2006 00:11:04 -0800, Dario Lopez-Kästen <dario@ita.chalmers.se> wrote:
Start by checking the ZODB cache numbers in the zope-conf file (too start simple). IIRC, the default is 5000 objects. Increase that to, say 50.000 or so (you mention that you have a busy site).
Sorry, but this is not good advice. 5000 objects in the cache is normally plenty, and you should look into caching the front-end before you even consider this. 50K objects in the cache will eat *a lot* of memory, and most likely cause swapping.
Cache properly with Apache or Squid in front first, then investigate the more complex settings.
Yes, just saying "change the setting to X" is definitely not good advice. But when it comes to twiddling knobs I would work from the bottom up instead of the other way around, meaning optimize the settings you have in Zope first. Even if you know you will use a cache you will want to make sure the backend is tuned properly. jens
On Wed, 25 Jan 2006 00:11:04 -0800, Dario Lopez-Kästen <dario@ita.chalmers.se> wrote:
Start by checking the ZODB cache numbers in the zope-conf file (too start simple). IIRC, the default is 5000 objects. Increase that to, say 50.000 or so (you mention that you have a busy site).
Sorry, but this is not good advice. 5000 objects in the cache is normally plenty, and you should look into caching the front-end before you even consider this. 50K objects in the cache will eat *a lot* of memory, and most likely cause swapping.
Not necessarily bad advice... if you have enough memory to hold the complete zcatalog you will notice significant search time improvements and if your site does a lot of zcatalog searches this can be a significant improvement. We are currently running a zcatalog with over 1 million objects and the catalog and indexes are in excess of 6Gb in size; the more memory the better the search performance. Front end caching is only good if you are serving up 'static' pages. Pages that are dynamically created, for each user/request, are not a good use case for front end caching (the exception is if there are portions of the pages, such a images, etc, which are common across your dynamically created pages, which will then make use of front end caching). Jonathan
How did you go about caching authenticated content? Any simple personalization element like the user name in the corner disqualifies the whole page from effective participation in downstream proxying/caching. The only thing you can do with such page is to cache it for one particular user hoping that s/he may need the same page again (which is not likely). I have an "extreme" situation when 90% of visitors are authenticated. Moreover, they participate in dozens of groups that define their access to individual content elements. Users are being granted and revoked participation in groups very actively, it is a normal situation that a few such changes occur within one session. And to finish the picture - publication and access times are important for records purposes so the current server time should be displayed on all pages. I use RAM caches for parts/macros of pages that may be identical for all users or some of the groups. They help, but the performance is still far from being perfect. Any success stories/best practices for such cases? Thanks. zope 2.7.6, plone 2.0.5 (going to upgrade soon), freebsd 4.10 Sasha Vincic wrote:
Yes Plone is slow but with caching you get very good performance. I have boosted performance on plone sites from default 1-3req/s to 100req/s and then it was the bandwith that was the bottleneck. The sites where serving both anonymous and authenticated content. I recommend to check out the CacheFu product and documentation in the collective.
/Sasha Lovely Systems - www.lovelysystems.com
I'm no expert in this realm, but: Find Chris M's presentation on scaling zope, somewhere on plope.com. It doesn't go into great detail on this topic, but basically, IIRC the strategy is to push personalization (and final page composition) downstream. This generally means either: - Squid and ESI - client-side (javascript) Then the parts of the page that are *not* personalized can still be cached very effectively by squid, and you greatly reduce the load on zope. The JS approach may allow for browser-side caching of (some of) the personalization data, if it can be re-used across pages. -PW On Tue, Jan 24, 2006 at 11:27:39PM +0100, Michael Vartanyan wrote:
How did you go about caching authenticated content? Any simple personalization element like the user name in the corner disqualifies the whole page from effective participation in downstream proxying/caching. The only thing you can do with such page is to cache it for one particular user hoping that s/he may need the same page again (which is not likely).
I have an "extreme" situation when 90% of visitors are authenticated. Moreover, they participate in dozens of groups that define their access to individual content elements. Users are being granted and revoked participation in groups very actively, it is a normal situation that a few such changes occur within one session. And to finish the picture - publication and access times are important for records purposes so the current server time should be displayed on all pages.
I use RAM caches for parts/macros of pages that may be identical for all users or some of the groups. They help, but the performance is still far from being perfect. Any success stories/best practices for such cases?
Thanks.
zope 2.7.6, plone 2.0.5 (going to upgrade soon), freebsd 4.10
Sasha Vincic wrote:
Yes Plone is slow but with caching you get very good performance. I have boosted performance on plone sites from default 1-3req/s to 100req/s and then it was the bandwith that was the bottleneck. The sites where serving both anonymous and authenticated content. I recommend to check out the CacheFu product and documentation in the collective.
/Sasha Lovely Systems - www.lovelysystems.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Paul Winkler http://www.slinkp.com
Paul Winkler wrote:
(and final page composition) downstream. This generally means either:
- Squid and ESI
Has anyone actually implemented this? I see it thrown out as a suggestion quite often, but when I tried the how-to on longsleep.org yesterday I found that the latest Squid 3.0 pre-release won't even compile with ESI enabled :-( cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 1/25/06, Chris Withers <chris@simplistix.co.uk> wrote:
Has anyone actually implemented this? I see it thrown out as a suggestion quite often, but when I tried the how-to on longsleep.org yesterday I found that the latest Squid 3.0 pre-release won't even compile with ESI enabled :-(
Squid 3.0 is a bit of an orphan. Zope Corp at one point paid for ESI support in Squid 3.0 for the Boston.com project, but it was never used for Boston in the end and development on it stalled. With Squid 2.x meeting current demands and little funding and incentive for completing ESI development this has fallen by the wayside. I certianly haven't touched Squid 3.0/ESI since the Boston.com days, and as far as I know only one or two others in the Zope community have looked at it since. -- Martijn Pieters
On 1/24/06, Michael Vartanyan <pycry@doli.biz> wrote:
How did you go about caching authenticated content? Any simple personalization element like the user name in the corner disqualifies the whole page from effective participation in downstream proxying/caching. The only thing you can do with such page is to cache it for one particular user hoping that s/he may need the same page again (which is not likely). Well I did cache the page for each user, because most of the content on a site isn't that dynamic that has to be reloaded every second. Should content on a page change you invalid them for everyone. Even if the cache is for short period it's worth it.
How: I used CacheFu and the description there.
And to finish the picture - publication and access times are important for records purposes so the current server time should be displayed on all pages. Well this is a problem, here you have something that invalidates the page every second or at least minut, but still caching one minute is worth alot on an active site. There are probably ways to do this with JS so it's on the client side instead but then you can't be 100% that it's the same time as on server but it shouldn't diff that much if you initiate the JS with right time.
If your dynamic parts are not to complicated I would consider Squid3 and ESI. /Sasha
On 1/24/06, Hugo Ramos <ramosh@gmail.com> wrote:
Let's talk about performance issues... Plone is VERY SLOW responding requests. All content is stored in ZODB objects and zcatalog is used to search. But even if you request the intranet home only, it takes about 10 seconds to load. The hardware is very good and lots of ram is used in the servers.
They also have zeo + 2 zope instances running in the same machine. I've never seen this in my life... only 3 machines running 2 zope instances + zeo instance, separately.
Do you think Zope(logic)+MySQL(content)+Apache(HTTP server) is better solution than Zope+Plone ???
No. That problem is Plone/CMF, and most of the problem is in the "skins". It's a combination of how portal_skins work, and the fact that Plones skins are not very optimized. CPS, who also uses portal_skins, are faster than an empty Plone site, for example, but still not as fast as a pure Zope-site can be. Zope 3 and Five both promises to solve that by it's "View" paradigm, but neither Plone nor CMF uses that fully yet, so if you use a lot of standard products that probably will not help you. I'm not a speed guru, but load balancing with ZEO would probably be the easiest way of improving the performance in this case. That's my guess. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
standard products that probably will not help you. I'm not a speed guru, but load balancing with ZEO would probably be the easiest way of improving the performance in this case. That's my guess.
If he's got a dual cpu'd machine and is running two zeo instances, he's fine unless he's got a ram problem. To be running 2 zeo instances on the same machine, they have to be doing some type of load balancing or faking load balancing so that ea. zeo client can respond to requests. One thing that might help is CRANK up your on disk zeo cache as high as you can afford - it's just disk space. If this hasn't been done, the zeo clients might be spending a lot of time 'requesting' objects from the zeo storage server when you can be caching them on the zeo clients. Some substantial effort would likely be required to optimize the Plone installation. If you choose this route, look at the skins - there's lots, lots of layers, and lots of searches going on you could eliminate or rewrite to be more efficient. If you go down the Zope/MySQL path, this would require some substantial effort also. I would recommend, if you're going to be putting substantial effort into the application, look at a CMF/Five implementation, relying heavily on using the View paradigm Lennart references. Another option, not knowing your technical level, is to implement in Zope 3. I personally am liking Zope 3 alot. IIRC, there is a Zope 3 CMS project out there. Andrew Sawyers
On 1/24/06, Andrew Sawyers <andrew@sawdog.com> wrote:
If he's got a dual cpu'd machine and is running two zeo instances, he's fine unless he's got a ram problem. To be running 2 zeo instances on the same machine, they have to be doing some type of load balancing or faking load balancing so that ea. zeo client can respond to requests.
I totally forgot that he was running ZEO already. That actually makes it even easier. since I was thinking of buying more machines. ;) Processing power is cheap today. Buying another machine (or two) would not cost much, and probably half the processing time (unless there is something actually wrong). As always, profiling is probably a good idea *before* trying to fix performnce problems. ;) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
Check out CacheFu and it's accompanying tutorial. http://plone.org/products/cachefu http://plone.org/products/cachefu/documentation/how-to/crash-course CacheFu is being used on plone.org at the moment. Neither Zope nor Plone are proper delivery systems, and if you want performance, you should read up on caching. -- Alexander Limi On Tue, 24 Jan 2006 06:08:44 -0800, Hugo Ramos <ramosh@gmail.com> wrote:
Hi all,
I'm responsible for a big corporate intranet and corporate portal. They were using Zope/Plone before I got here and the problems are too many.
Let's talk about performance issues... Plone is VERY SLOW responding requests. All content is stored in ZODB objects and zcatalog is used to search. But even if you request the intranet home only, it takes about 10 seconds to load. The hardware is very good and lots of ram is used in the servers.
They also have zeo + 2 zope instances running in the same machine. I've never seen this in my life... only 3 machines running 2 zope instances + zeo instance, separately.
Do you think Zope(logic)+MySQL(content)+Apache(HTTP server) is better solution than Zope+Plone ???
TIA
-- Hugo Ramos - ramosh@gmail.com Senior Zope consultant _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- _____________________________________________________________________ Alexander Limi · Chief Architect · Plone Solutions · Norway Consulting · Training · Development · http://www.plonesolutions.com _____________________________________________________________________ Plone Co-Founder · http://plone.org · Connecting Content Plone Foundation · http://plone.org/foundation · Protecting Plone
On 24 Jan 2006, at 20:36, Alexander Limi wrote:
Check out CacheFu and it's accompanying tutorial.
http://plone.org/products/cachefu http://plone.org/products/cachefu/documentation/how-to/crash-course
CacheFu is being used on plone.org at the moment.
Neither Zope nor Plone are proper delivery systems, and if you want performance, you should read up on caching.
Zope by itself can be a very capable delivery system. Not sure what you're talking about. It's all about the application you put on top of Zope. jens
Hugo Ramos wrote at 2006-1-24 11:08 -0300:
... Let's talk about performance issues... Plone is VERY SLOW responding requests. All content is stored in ZODB objects and zcatalog is used to search. But even if you request the intranet home only, it takes about 10 seconds to load. The hardware is very good and lots of ram is used in the servers.
Use a profiler to find out where the time is spent (e.g. "ZopeProfiler"). Maybe, your site it badly configured (running in debug mode, ZODB cache too small, missing "skin" cache, ...) -- Dieter
participants (16)
-
Alexander Limi -
Andrew Sawyers -
bruno desthuilliers -
Chris Withers -
Dario Lopez-Kästen -
Dieter Maurer -
Hugo Ramos -
Jens Vagelpohl -
Jonathan -
Lennart Regebro -
Martijn Pieters -
Max M -
Michael Vartanyan -
Paul Winkler -
Peter Bengtsson -
Sasha Vincic