The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time. We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate. Occasionally we can track down the offending URL and correct the problem, sometimes we cannot. Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4 What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client. Below is some information on our setup: 1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds 2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 30000 objects - 4 threads 2 failover apaches (Linux) - using pydirector for load balancing We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment? Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network
How about setting up persistent caches for your zeo clients, so that when the zeo server goes down you save some cache-reloading time when it restarts? Jonathan ----- Original Message ----- From: "Brian Brinegar" <brinegar@ecn.purdue.edu> To: <zope@zope.org> Cc: "David S Carmichael" <carmicha@ecn.purdue.edu>; "Andrew T Sydelko" <sydelko@ecn.purdue.edu>; "Christopher N Deckard" <cnd@ecn.purdue.edu> Sent: Thursday, May 25, 2006 10:49 AM Subject: [Zope] Preventing Zope Spinning?
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time.
We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate.
Occasionally we can track down the offending URL and correct the problem, sometimes we cannot.
Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4
What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client.
Below is some information on our setup:
1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds
2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 30000 objects - 4 threads
2 failover apaches (Linux) - using pydirector for load balancing
We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment?
Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Persistent caches can introduce odd errors, unfortunately. They can create inconsistencies between what a ZEO client thinks the database state is and what the ZEO server thinks. Not recommended for production setups. jens On 25 May 2006, at 15:56, Jonathan wrote:
How about setting up persistent caches for your zeo clients, so that when the zeo server goes down you save some cache-reloading time when it restarts?
Jonathan
----- Original Message ----- From: "Brian Brinegar" <brinegar@ecn.purdue.edu> To: <zope@zope.org> Cc: "David S Carmichael" <carmicha@ecn.purdue.edu>; "Andrew T Sydelko" <sydelko@ecn.purdue.edu>; "Christopher N Deckard" <cnd@ecn.purdue.edu> Sent: Thursday, May 25, 2006 10:49 AM Subject: [Zope] Preventing Zope Spinning?
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time.
We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate.
Occasionally we can track down the offending URL and correct the problem, sometimes we cannot.
Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4
What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client.
Below is some information on our setup:
1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds
2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 30000 objects - 4 threads
2 failover apaches (Linux) - using pydirector for load balancing
We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment?
Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEdcaBRAx5nvEhZLIRAucHAJ408RTF8Q9F7m1nBQxhO557TRRNwQCeKnKI f0B/KkAP42EIAgslBv31K68= =oRky -----END PGP SIGNATURE-----
Thanks for the warning! We haven't come across any of these 'odd errors', but we will get rid of our persistent caches just to be sure! Jonathan ----- Original Message ----- From: "Jens Vagelpohl" <jens@dataflake.org> To: "[Zope] Mailing List" <zope@zope.org> Sent: Thursday, May 25, 2006 11:00 AM Subject: Re: [Zope] Preventing Zope Spinning?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Persistent caches can introduce odd errors, unfortunately. They can create inconsistencies between what a ZEO client thinks the database state is and what the ZEO server thinks. Not recommended for production setups.
jens
On 25 May 2006, at 15:56, Jonathan wrote:
How about setting up persistent caches for your zeo clients, so that when the zeo server goes down you save some cache-reloading time when it restarts?
Jonathan
----- Original Message ----- From: "Brian Brinegar" <brinegar@ecn.purdue.edu> To: <zope@zope.org> Cc: "David S Carmichael" <carmicha@ecn.purdue.edu>; "Andrew T Sydelko" <sydelko@ecn.purdue.edu>; "Christopher N Deckard" <cnd@ecn.purdue.edu> Sent: Thursday, May 25, 2006 10:49 AM Subject: [Zope] Preventing Zope Spinning?
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time.
We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate.
Occasionally we can track down the offending URL and correct the problem, sometimes we cannot.
Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4
What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client.
Below is some information on our setup:
1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds
2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 30000 objects - 4 threads
2 failover apaches (Linux) - using pydirector for load balancing
We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment?
Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin)
iD8DBQFEdcaBRAx5nvEhZLIRAucHAJ408RTF8Q9F7m1nBQxhO557TRRNwQCeKnKI f0B/KkAP42EIAgslBv31K68= =oRky -----END PGP SIGNATURE----- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Jens Vagelpohl wrote:
Persistent caches can introduce odd errors, unfortunately. They can create inconsistencies between what a ZEO client thinks the database state is and what the ZEO server thinks. Not recommended for production setups.
Don't necessarily agree with this. If the clients get in a tiz, that's a bug that should be fixed. There are several production setusp where I use persistent client caches and I've had very few problems with them... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 30 May 2006, at 15:26, Chris Withers wrote:
Jens Vagelpohl wrote:
Persistent caches can introduce odd errors, unfortunately. They can create inconsistencies between what a ZEO client thinks the database state is and what the ZEO server thinks. Not recommended for production setups.
Don't necessarily agree with this. If the clients get in a tiz, that's a bug that should be fixed.
There are several production setusp where I use persistent client caches and I've had very few problems with them...
I'm sorry, this is not a matter of "agreeing". It is a matter of personally observed facts. jens -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEfFaNRAx5nvEhZLIRAhl/AKCsPm4lQeHr4JAAgWwOsRZGaMBt7QCfUo0E 1bx2oQocV666c3XajSnQBh0= =et2a -----END PGP SIGNATURE-----
Jens Vagelpohl wrote:
There are several production setusp where I use persistent client caches and I've had very few problems with them...
I'm sorry, this is not a matter of "agreeing". It is a matter of personally observed facts.
Yes, which is exactly what I'm providing ;-) I've seen big benefits from using persistent caches, and zero problems... okay? ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Jonathan, We have debated this, however the time necessary to revalidate the cache can be several minutes (maybe more) at startup for a 2 gig cache. If only one client was going down at a time this would not be a problem, however in the event that someone clicks reload until all of the clients are dead it would cause complete down time while the clients startup. We made the decision that having the server be slow was better than having it be completely unavailable for an extended period of time. I would like to prevent the clients from ever needing to be restarted. Brian Brinegar Web Systems Developer Engineering Computer Network Jonathan wrote:
How about setting up persistent caches for your zeo clients, so that when the zeo server goes down you save some cache-reloading time when it restarts?
Jonathan
----- Original Message ----- From: "Brian Brinegar" <brinegar@ecn.purdue.edu> To: <zope@zope.org> Cc: "David S Carmichael" <carmicha@ecn.purdue.edu>; "Andrew T Sydelko" <sydelko@ecn.purdue.edu>; "Christopher N Deckard" <cnd@ecn.purdue.edu> Sent: Thursday, May 25, 2006 10:49 AM Subject: [Zope] Preventing Zope Spinning?
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time.
We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate.
Occasionally we can track down the offending URL and correct the problem, sometimes we cannot.
Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4
What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client.
Below is some information on our setup:
1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds
2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 30000 objects - 4 threads
2 failover apaches (Linux) - using pydirector for load balancing
We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment?
Thank you, Brian Brinegar Web Systems Developer Engineering Computer Network _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Brian Brinegar wrote:
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time.
We end up in a situation where something will spin a client and a user will repeatedly resubmit the request until all of our zeo clients are spinning. Our clients are eventually killed, usually manually, and quickly come back up. We drop the zeo cache on a restart to improve startup speed, so when the clients do come back up we have 100% cache misses and the zeo server gets pounded resulting in slow performance until the client caches repopulate.
Occasionally we can track down the offending URL and correct the problem, sometimes we cannot.
Perhaps these issues will be addressed in future versions of Zope, we are currently running Zope 2.6.4
What I would like is some sort of timeout for requests, however I do not want to punish users with slow connections. Perhaps a way to kill off a specific request that is consuming excessive resources, without killing the entire client.
Below is some information on our setup:
1 zeo server (Solaris) - 82 gig datafs - transaction time out of 120 seconds
2 load balanced zeoclients (Linux) - 2 gig zeo cache - Database Cache 30000 objects - 4 threads
2 failover apaches (Linux) - using pydirector for load balancing
We are receiving appoximately 1 million hits per day, which from what I've read is not all that much. We probably have a higher number of DB writes than usual because of the number of developers/maintainers. Can anyone make suggestions for providing a more stable environment?
Enabling the "trace log" would help to find the errant requests. In Zope 2.6, I think that involved adding '-M <logfilename>' to the z2.py command line. There is a zope.conf stanza for it in Zope 2.7 and later. There is a 'requestprofiler.py' script which is useful for analyzing the log. I would strongly recommend upgrading to a more recent version of Zope and ZODB, as there have been a large number of stability-related fixes and features added since Zope 2.6 / ZODB 3.1, neither of which is being maintained at this point. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEdeIQ+gerLs4ltQ4RAqb3AJ4viBPiuD4kOmxvf9780zhRihKnLgCgu+4m OpMkS1V+6ZZoOynQDPTwYTI= =sMfD -----END PGP SIGNATURE-----
On Thu, May 25, 2006 at 10:49:10AM -0400, Brian Brinegar wrote:
The majority of the Purdue University Engineering web presence is provided via a cluster running ZEO. We offer hosting for every school, department, faculty, staff, and student in the College. Because of this we have a large number of content maintainers/developers on our system. We are running into problems with users writing bad code which spins or uploading huge files which seems to tie up the database for long periods of time.
Large files really hurt ZEO performance, both reads (since they're likely not in the cache, and if they are, they evict lots of other useful stuff), and for writes. I'd see about replacing all large files in the ZODB with files on an NFS mount. Chris M's Blob product might help, see http://plope.com/software/blob/ . You could use that as the basis for a replacement for OFS.File, and then remove everyone's permission to add OFS.File instances. Then write zopectl scripts to migrate existing Files. As for people writing bad code - that will always be a problem :) -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote at 2006-5-25 13:05 -0400:
... Large files really hurt ZEO performance, both reads (since they're likely not in the cache, and if they are, they evict lots of other useful stuff), and for writes.
Files use 64 kB chunks -- thus the problem you describe occurs with files I would call quite huge (several 10 MB and above). -- Dieter
Brian Brinegar wrote at 2006-5-25 10:49 -0400:
... What I would like is some sort of timeout for requests
This is very difficult: There is no safe way to abort a thread without its help -- an the thread might be in a condition where it does not recognize that its help is needed (e.g. it is waiting in a C extension). With version 2.3, Python grew a C-level (!) API function which sends an exception to a thread. In case, the thread is executing Python code, this exception may abort the thread (cleaning us as usually). However, when the thread executes C code, the exception is not seen until it reenters the Python interpreter. Therfore, this is only a partial solution... -- Dieter
participants (7)
-
Brian Brinegar -
Chris Withers -
Dieter Maurer -
Jens Vagelpohl -
Jonathan -
Paul Winkler -
Tres Seaver