Sizing a zope server
Hi, I plan to setup an apache/zope server for virtual hosting in an ISP environment. Are there any good docs outlining resource requirements of zope? Using ZODB it looks (from my initial foray) as if all data is stored in the file Data.fs. If Data.fs grows to 1/2 GB will my zope server still function? What if things grow to 2 GB? Thanks! -- Fraser Campbell <fraser@wehave.net> http://wehave.net/ Brampton, Ontario, Canada Linux 2.4.20 AuthenticAMD
Fraser Campbell wrote:
Hi, I plan to setup an apache/zope server for virtual hosting in an ISP environment. Are there any good docs outlining resource requirements of zope?
Using ZODB it looks (from my initial foray) as if all data is stored in the file Data.fs. If Data.fs grows to 1/2 GB will my zope server still function? What if things grow to 2 GB?
Thanks!
Zope is not a speed demon. You would fair better by putting Squid in front of Zope. I have seen 17GB data.fs's with no appreciate able degradation. It is wise to keep it smaller than this. Make sure Python and the OS is compatible with going over the 2GB limit. Python requires compile-time configuration parameters to allow it to handle files over 2GB. - Matt
On February 13, 2003 01:45 pm, Matt Burleigh wrote:
Zope is not a speed demon. You would fair better by putting Squid in front of Zope.
Can you get the same advantages with apache/mod_proxy as you would with squid or is squid just your preferred frontend? Apache gives me lots of other things and I'll no doubt run at least a few non-zope sites.
I have seen 17GB data.fs's with no appreciate able degradation. It is wise to keep it smaller than this. Make sure Python and the OS is compatible with going over the 2GB limit. Python requires compile-time configuration parameters to allow it to handle files over 2GB.
Yup, I'll look into that. The platform will be the latest stable version of Debian Linux (Linux kernel 2.4) on intel hardware. I'm 99% sure that the 2.4 kernel is fine with large files but I will definitely be keeping it in mind. -- Fraser Campbell <fraser@wehave.net> http://wehave.net/ Brampton, Ontario, Canada Linux 2.4.20 AuthenticAMD
Matt, Just out of curiosity, what kind of application required that much data? I am just starting on a web application that will have to handle a lot of operational data for power plant equipment, and I was hoping I could keep much of it in a ZODB. I saw a statement in the _Zope Web Application Construction Kit_ book that said "...the ZODB is not suitable for storing large amounts of data." I guess the meaning of "large amounts of data" depends on who you're talking to. :) Thanks, Alan Matt Burleigh wrote:
I have seen 17GB data.fs's with no appreciate able degradation. It is wise to keep it smaller than this. Make sure Python and the OS is compatible with going over the 2GB limit. Python requires compile-time configuration parameters to allow it to handle files over 2GB.
Alan McIntyre wrote:
I saw a statement in the _Zope Web Application Construction Kit_ book that said "...the ZODB is not suitable for storing large amounts of data." I guess the meaning of "large amounts of data" depends on who you're talking to. :)
That's just one of a number of sweeping statements in that rather pporly written book that are toally inaccurate. cheers, Chris
Chris Withers wrote at 2003-2-15 09:31 +0000:
Alan McIntyre wrote:
I saw a statement in the _Zope Web Application Construction Kit_ book that said "...the ZODB is not suitable for storing large amounts of data." I guess the meaning of "large amounts of data" depends on who you're talking to. :)
That's just one of a number of sweeping statements in that rather pporly written book that are toally inaccurate.
Things are changing over time.... Up to some time ago, I, too, was advicing not to put large objects (like big files) into the ZODB. Having a huge monolitic file calls for non-trivial backup strategies. However, with the advent of "DirectoryStorage"; I think, I will put everything into the ZODB (what is not relational mass data). Dieter
Alan McIntyre wrote:
Matt,
Just out of curiosity, what kind of application required that much data? I am just starting on a web application that will have to handle a lot of operational data for power plant equipment, and I was hoping I could keep much of it in a ZODB.
Its was a high activity web portal application. It was generally growing to 8GB per week. A pack reduced down to about 2GB. The resulting 17GB was from not packing over a two week period. ;) My experience is that ZODB is fairly robust and have significant trust in it to keep fairly large amounts of data.
On Thu, Feb 13, 2003 at 01:16:00PM -0500, Fraser Campbell wrote:
Using ZODB it looks (from my initial foray) as if all data is stored in the file Data.fs.
yes, that's the default.
If Data.fs grows to 1/2 GB will my zope server still function? What if things grow to 2 GB?
We have had our production Data.fs up to 2.5 GB at times. It works. But your OS & filesystem and Python must both be configured to support large files. However, making backups of a monolothic file that big is rather annoying, and we have had two cases of still-undiagnosed database corruption which we recovered from but it was very aggravating. For a more scaleable approach, we are investigating DirectoryStorage. google for it, it's very interesting. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE MATRIX PRINCESS! (random hero from isometric.spaceninja.com)
We have a couple of implementations with Data.fs files over 2GB and have no issues at all. Performance is still good and everything functions fine. By the way, you'll have to make sure the version of Zope you are using is compiled by a python version that contains large file support. Search the archives for more info. Kevin Fraser Campbell wrote:
Hi, I plan to setup an apache/zope server for virtual hosting in an ISP environment. Are there any good docs outlining resource requirements of zope?
Using ZODB it looks (from my initial foray) as if all data is stored in the file Data.fs. If Data.fs grows to 1/2 GB will my zope server still function? What if things grow to 2 GB?
Thanks!
I have a Data.fs now at 816.4 MB and it is doing fine. There are plenty of ways to keep down the size of the DB most of which have already been suggested.... - Let apache serve images - Store big files on the file system (or through products that allow that) - Pack your Data.fs frequently - Make sure your system has lots of RAM (Zope stores things in memory) - Use 2.6.1 which has better cache implementation - Use an outside cache (squid, mod_proxy) to let Zope do what Zope does best (not server up static, frequent files). BZ
Hi, I plan to setup an apache/zope server for virtual hosting in an ISP environment. Are there any good docs outlining resource requirements of zope?
Using ZODB it looks (from my initial foray) as if all data is stored in the file Data.fs. If Data.fs grows to 1/2 GB will my zope server still function? What if things grow to 2 GB?
Thanks! -- Fraser Campbell <fraser@wehave.net> http://wehave.net/ Brampton, Ontario, Canada Linux 2.4.20 AuthenticAMD
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
participants (8)
-
Alan McIntyre -
BZ -
Chris Withers -
Dieter Maurer -
Fraser Campbell -
Kevin Carlson -
Matt Burleigh -
Paul Winkler