Hi I need to quote for a fairly big Zope site and I am trying to work out what sort of spec machine I will need for hosting. The brief is for a minimum of 50 concurrent users, rising to a possible 2000. They will be doing a lot of searching on the site, and there will be quite a lot of e-mails sent out by Zope. The site will also use an SQL database - either mySQL or postgreSQL. My current thinking is: - Linux: RedHat? - Zope with ZEO in case it expands beyond one box - Apache - Squid? - 512 MB RAM - 18 GB SCSI with RAID - 1.8 GHZ processor Am I along the right lines? Will Zope coexist nicely with the database on the same box or should they be separate from day one? Cheers Tom
for real scalability i would suggest the following: - plan on having separate boxes from day 1: - ZEO server - 1-* appservers - 1-2 cache servers - database server sending lots of mail out of zope is not a good idea if you don't have a solution that uncouples the mailing process from zope itself. your users will be waiting forever if you use the standard built-in mail handling. caching is very important but it is not possible if your content is super-personalized. jens On Tuesday, May 28, 2002, at 10:31 , Tom Nixon wrote:
Hi
I need to quote for a fairly big Zope site and I am trying to work out what sort of spec machine I will need for hosting.
The brief is for a minimum of 50 concurrent users, rising to a possible 2000. They will be doing a lot of searching on the site, and there will be quite a lot of e-mails sent out by Zope. The site will also use an SQL database - either mySQL or postgreSQL.
My current thinking is:
- Linux: RedHat? - Zope with ZEO in case it expands beyond one box - Apache - Squid? - 512 MB RAM - 18 GB SCSI with RAID - 1.8 GHZ processor
Am I along the right lines?
Will Zope coexist nicely with the database on the same box or should they be separate from day one?
Cheers
Tom
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
In addition think about using a loadbalancer between the cache servers and the ZEO servers and maybe between the incoming IP and the cache servers. -aj ----- Original Message ----- From: "Jens Vagelpohl" <jens@zope.com> To: "Tom Nixon" <tom.nixon@aim23.com> Cc: <zope@zope.org> Sent: Tuesday, May 28, 2002 10:52 Subject: Re: [Zope] system requirements
for real scalability i would suggest the following:
- plan on having separate boxes from day 1: - ZEO server - 1-* appservers - 1-2 cache servers - database server
sending lots of mail out of zope is not a good idea if you don't have a solution that uncouples the mailing process from zope itself. your users will be waiting forever if you use the standard built-in mail handling.
caching is very important but it is not possible if your content is super-personalized.
jens
On Tuesday, May 28, 2002, at 10:31 , Tom Nixon wrote:
Hi
I need to quote for a fairly big Zope site and I am trying to work out what sort of spec machine I will need for hosting.
The brief is for a minimum of 50 concurrent users, rising to a possible 2000. They will be doing a lot of searching on the site, and there will be quite a lot of e-mails sent out by Zope. The site will also use an SQL database - either mySQL or postgreSQL.
My current thinking is:
- Linux: RedHat? - Zope with ZEO in case it expands beyond one box - Apache - Squid? - 512 MB RAM - 18 GB SCSI with RAID - 1.8 GHZ processor
Am I along the right lines?
Will Zope coexist nicely with the database on the same box or should they be separate from day one?
Cheers
Tom
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Tom Nixon wrote:
Hi
I need to quote for a fairly big Zope site and I am trying to work out what sort of spec machine I will need for hosting.
The brief is for a minimum of 50 concurrent users, rising to a possible 2000. They will be doing a lot of searching on the site, and there will be quite a lot of e-mails sent out by Zope. The site will also use an SQL database - either mySQL or postgreSQL.
My current thinking is:
- Linux: RedHat? - Zope with ZEO in case it expands beyond one box - Apache - Squid? - 512 MB RAM - 18 GB SCSI with RAID - 1.8 GHZ processor
Am I along the right lines?
Will Zope coexist nicely with the database on the same box or should they be separate from day one?
Cheers
Tom
That sounds fairly reasonable for me. Remember that Zope is primarily goverened by the pystone benchmark; my experience on DDR Athlons is very positive. On a fast uniprocessor, you should be able to slice Apache, Squid, and Zope together with a database. I've no doubt you can outgrow it in the future; but the key notion is to spend as few cycles rendering Zope content as possible. Agressive caching helps you do that, so running local Squid and or Apache will help. Ultimately it comes down to getting some good metrics on how *your* application will perform in Zope. It is easy to write a zope app that doesn't appear to perform poorly until it is under load. It's frustrating to not be able to give you an absolute answer; but that's hard to do. I like casting Zope performance in terms of "pystones used" which isn't really accurate, but provides a reference for scaling. If an Athlon XP 2000+ has 25,700 pystones, and can render 50 application pages / sec, a 1 Ghz P III with 12,000 pystones is only going to render about 23 pages / sec. My old SparcStation 10 with a whopping 43Mhz and about 800 pystones is going to come in at about 1.3 pages/sec. That same SS10 can probably pump 400+ pages/sec with apache, though. Spending a few more CPU cycles to move data into apache or squid can have a big payout if that means not having to rerender the page. -- Matt Kromer Zope Corporation http://www.zope.com/
I'm about to put together a production machine as well and so have been thinking about the issues and cost/performance trade-offs. Comments on hidden gotchas would be particularly welcome. Have there been problems with dual processor Linuz systems and ZServer? Software: Zope 2.5.X and CMF 1.3, ZServer + Squid No Apache -- this will be a dedicated Zope system. RH 7.2 with lots of cruft removed. MySQL. Hardware: Dual AMD Athlon processors (2+ GHz) Tyan motherboard with 760 chipset? 1GB DDR memory with ECC. (2GB if budget allows) 10/100 Ethernet CDROM (for loading...) Floppy (for bootloading/rescue) 40 GB+ disk (probably IDE) SCSI Display controller (for configuration) 2U rackmount box with big fans On Tue, 28 May 2002, Matthew T. Kromer wrote:
I need to quote for a fairly big Zope site and I am trying to work out what sort of spec machine I will need for hosting.
The brief is for a minimum of 50 concurrent users, rising to a possible 2000. They will be doing a lot of searching on the site, and there will be quite a lot of e-mails sent out by Zope. The site will also use an SQL database - either mySQL or postgreSQL.
My current thinking is:
- Linux: RedHat? - Zope with ZEO in case it expands beyond one box - Apache - Squid? - 512 MB RAM - 18 GB SCSI with RAID - 1.8 GHZ processor
Am I along the right lines?
Dennis Allison wrote:
I'm about to put together a production machine as well and so have been thinking about the issues and cost/performance trade-offs. Comments on hidden gotchas would be particularly welcome. Have there been problems with dual processor Linuz systems and ZServer?
Software: Zope 2.5.X and CMF 1.3, ZServer + Squid No Apache -- this will be a dedicated Zope system. RH 7.2 with lots of cruft removed. MySQL.
Hardware: Dual AMD Athlon processors (2+ GHz) Tyan motherboard with 760 chipset? 1GB DDR memory with ECC. (2GB if budget allows) 10/100 Ethernet CDROM (for loading...) Floppy (for bootloading/rescue) 40 GB+ disk (probably IDE) SCSI Display controller (for configuration) 2U rackmount box with big fans
On Tue, 28 May 2002, Matthew T. Kromer wrote:
I do *not* recommend running Zope on multiprocessor machines without an ability to restrict Zope to execution on a single CPU. The reason for this is that the Python Global Interpreter Lock is shared inside a Zope process. However, threads in Python are backed by underlying OS threads. Thus, Zope will create multiple threads, and each thread is likely to be assigned to a different CPU by the OS scheduler. However, all CPUs but one which are dispatching any given Zope process will have to then wait and attempt to acquire the GIL; this process introduces significant latency into Python and thus into Zope. Linux has no native mechanism for processor binding. In fact, there is a CPU dispatch mask for processes, but there is no facility to set the mask that I know of. Solaris can use a system command like 'pbind' to bind a process to a particular CPU. If you *do* go with a multiprocessor, be sure and tune the os.setcheckinterval() call made by z2.py. I think it should be roughly your pystone number divided by 50. This will set the coarse scheduler granularity of Python so that it doesn't attempt to move the lock as often. This will allow Zope to do as much work as it can without injecting latencies shuffling the GIL off between threads on other CPUs. In fact, setting os.setcheckinterval() to pystones/50 is probably a good idea regardless of what platform you are on. The current default is 120, which is better than the Python default of 10 -- but still means there is a lot of unnecessary lock shuffling on fast CPUs (the number 120 was reasonable 2 years ago, but isn't now -- an Athlon XP 2000+ should probably have a number of about 500). The basic idea is to allow some particular time quanta to expire before moving the lock -- but the counter is expressed in python bytecode ops. As CPUs get faster, the quanta is shrinking, rather than remaining constant -- or in other words, overhead is growing linearly with CPU speed improvements unless you make this tuning change. -- Matt Kromer Zope Corporation http://www.zope.com/
On Tue, 28 May 2002, Matthew T. Kromer wrote:
Dennis Allison wrote:
I'm about to put together a production machine as well and so have been thinking about the issues and cost/performance trade-offs. Comments on hidden gotchas would be particularly welcome. Have there been problems with dual processor Linuz systems and ZServer?
Software: Zope 2.5.X and CMF 1.3, ZServer + Squid No Apache -- this will be a dedicated Zope system. RH 7.2 with lots of cruft removed. MySQL.
Hardware: Dual AMD Athlon processors (2+ GHz) Tyan motherboard with 760 chipset? 1GB DDR memory with ECC. (2GB if budget allows) 10/100 Ethernet CDROM (for loading...) Floppy (for bootloading/rescue) 40 GB+ disk (probably IDE) SCSI Display controller (for configuration) 2U rackmount box with big fans
On Tue, 28 May 2002, Matthew T. Kromer wrote:
I do *not* recommend running Zope on multiprocessor machines without an ability to restrict Zope to execution on a single CPU.
The reason for this is that the Python Global Interpreter Lock is shared inside a Zope process. [...]
Thanks Matt -- I was worried about the threads and locks issue in an MP environment. Linix has always been a bit behind the curve about that. What's the situation with FreeBSD? Does it provide the sort of processor lock-in that prevents excessive blocking? -dra
participants (5)
-
Andreas Jung -
Dennis Allison -
Jens Vagelpohl -
Matthew T. Kromer -
Tom Nixon