Memory consumtion of a zope instance
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi List, I'm doing shared zope hosting, where everbody gets his own instance (using the INSTANCE_HOME environment variable). but when I looked at the memory consumtion I noticed that every instance grabed between 11 and 18 Megabytes of RAM. Since our server is not too blessed with RAM I wondered, whether it was possible to reduce the RAM comsumtion of the instances. I thought at doing a 'late' fork: start a supervisor process at first which loads python and initializes the default products. An Instance would fork this process and initialize the products special to it. In this scheme python's and most of zope's RAM pages are copy on write on most UNIX systems (-> they don't eat your mem). As a site effect, instance startup would be greatly accelerated. Is something like this available, or planned? Maybe I can code this on my own, but at first I've got to gather a lot more experience with zope. Thanks AND - -- "I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this." -- Emo Phillips -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE9f7oThlekfiEFJv4RAhMFAKC5ZUTcVFLXnKfHaz8ahYnn9ulvlQCfXu1R WH2H5dNYyrAFRgdc9XF7rbg= =SFpK -----END PGP SIGNATURE-----
so you really think that coming up with a setup like you propose is better than getting some extra RAM? i don't get it. RAM is comparably cheap. besides, 11-18 MB of RAM consumption for a zope process really isn't much at all. jens On Wednesday, Sep 11, 2002, at 17:48 US/Eastern, Andreas Lauser wrote:
Hi List,
I'm doing shared zope hosting, where everbody gets his own instance (using the INSTANCE_HOME environment variable). but when I looked at the memory consumtion I noticed that every instance grabed between 11 and 18 Megabytes of RAM. Since our server is not too blessed with RAM I wondered, whether it was possible to reduce the RAM comsumtion of the instances.
I thought at doing a 'late' fork: start a supervisor process at first which loads python and initializes the default products. An Instance would fork this process and initialize the products special to it. In this scheme python's and most of zope's RAM pages are copy on write on most UNIX systems (-> they don't eat your mem). As a site effect, instance startup would be greatly accelerated.
Is something like this available, or planned? Maybe I can code this on my own, but at first I've got to gather a lot more experience with zope.
Thanks AND
On Thursday 12 September 2002 01:23, Jens Vagelpohl wrote:
so you really think that coming up with a setup like you propose is better than getting some extra RAM? i don't get it. RAM is comparably cheap.
that's true. It would be my solution of choice if I had physical access to my server. well maybe I've got to think about a more expensive machine.
besides, 11-18 MB of RAM consumption for a zope process really isn't much at all.
that depends on the view. after 20 instances between 220 and 360 MB are eaten. a bad thing if your server has only 256 MB. Furthermore this makes disk access slower as the "zope overhead" throws out disk cache pages. Regards AND -- "I used to think that the brain was the most wonderful organ in my body. Then I realized who was telling me this." -- Emo Phillips
On Thursday 12 Sep 2002 12:11 pm, Andreas Lauser wrote:
On Thursday 12 September 2002 01:23, Jens Vagelpohl wrote:
I thought at doing a 'late' fork: start a supervisor process at first which loads python and initializes the default products. An Instance would fork this process and initialize the products special to it. In this scheme python's and most of zope's RAM pages are copy on write on most UNIX systems (-> they don't eat your mem). As a site effect, instance startup would be greatly accelerated.
The problem with this schemes is python reference count. Most pages will be copied, because you cant touch an object without writing to its reference count.
besides, 11-18 MB of RAM consumption for a zope process really isn't much at all.
that depends on the view. after 20 instances between 220 and 360 MB are eaten. a bad thing if your server has only 256 MB. Furthermore this makes disk access slower as the "zope overhead" throws out disk cache pages.
IMHO trying to run 20 zope instances on a machine with just 256 MB of RAM is, err, insane. zope is not lightweight like you average apache process. it likes RAM. and it will take more and more as the content on your sites grow since the ZODB index is stored in RAM. jens
[Jens Vagelpohl]
IMHO trying to run 20 zope instances on a machine with just 256 MB of RAM is, err, insane.
Does anyone have some stats on: o Zope memory consumption quantified by (size of ZODB, other relevant factors). o Reports of number of Zope instances running on one machine compared to hardware. Cheers, // m -
Does anyone have some stats on:
o Zope memory consumption quantified by (size of ZODB, other relevant factors). o Reports of number of Zope instances running on one machine compared to hardware.
Cheers,
// m
there is a lot of factors playing into this:: - ZODB size - hit rate on your website - ZODB cache settings - version of zope - number and size of ZCatalogs used IMHO i think it's really hard to give meaningful numbers. jens
[Jens Vagelpohl]
there is a lot of factors playing into this::
- ZODB size - hit rate on your website - ZODB cache settings - version of zope - number and size of ZCatalogs used
IMHO i think it's really hard to give meaningful numbers.
I'd even be entertained and delighted to hear non-meaningful numbers, though. I say that partly in jest, but I guess I mean, "Any of the folks out there hosting Zope care to share whatever numbers they have on how many Zope instances they can fit on a machine?" It would be a bonus to have some information on the average Zope instance's profile a la the factors you mention as well as stats on the machine (RAM, disk space, SCSI vs. Ultra ATA, etc). cheers, // mark -
On Thu, Sep 12, 2002 at 08:58:06AM -0500, Mark McEahern wrote:
I'd even be entertained and delighted to hear non-meaningful numbers, though.
well, ok, here's what I can glean from my shell account at the company that hosts my zope sites for me: - linux 2.2.something - dual PIII 850 MHz - about 2 GB RAM (free shows 1962 meg) - about 80 instances of zope running single-threaded with pcgi - about 17 instances of zope running multithreaded zserver Memory usage: appended is a dump from "top -b", with the user name column deleted to protect the innocent; sorted by memory usage . Two issues with this list: some lines show memory in meg with an M suffix, and some do not, so the sort order is wrong; and, this being linux, each thread of a multithreaded process is shown in the list (and shown with its own memory stats, though they really use the same memory). -- Paul Winkler "Welcome to Muppet Labs, where the future is made - today!"
I'd even be entertained and delighted to hear non-meaningful numbers, though.
www.zopehosting.ch One of our machines which we think is representative has 1 Gigabyte RAM, 20 Zope instances, MySQL and Apache, no Squid, 36% of Memory are used (by all applications). -------------------------------------- Goeldi.com - Internet Services -------------------------------------- www.goeldi.com www.zopehosting.ch web@goeldi.com Tel +41-61-7330555 Fax +41-61-7330556 --------------------------------------
On Thursday 12 Sep 2002 12:56 pm, Jens Vagelpohl wrote:
zope is not lightweight like you average apache process. it likes RAM. and it will take more and more as the content on your sites grow since the ZODB index is stored in RAM.
If you use FileStorage, yes. FileStorage would be a bad choice for this project. All berkeley-based storage and DirectoryStorage do not have this problem. Also the new ZODB memory cache in Zope 2.6 is less wasteful with ram than previous versions. If Andreas is running zope in 11-18 MB of RAM I guess he is already using this. If not, there is an extra saving to be made.
participants (6)
-
Andreas Lauser -
Jens Vagelpohl -
Mark McEahern -
Paul Winkler -
Stephan Goeldi -
Toby Dickenson