[Zope-dev] How to make Zope fail nicely under high load?

[Edward Muller]

On Thu, 2004-02-12 at 07:19, Bjorn Stabell wrote:
> Toby wrote:
> > > One of the optimization we're thinking of is storing results of 
> > > ZCatalog searches (e.g., number of replies to postings) in volatile 
> > > variables so we don't have to run the catalog search at all.  We'd 
> > > like to use memory space shared between threads for this.  
> > > Using ZEO would require us to store this in the ZODB.
> > 
> > Storing that in ZODB would be a bad idea. Theres no reason to 
> > think that this cache would be faster than ZCatalog.
> >
> > I dont see why you would be *required* to store in ZODB... 
> > just dont share your cache between publisher threads on different
> > Zope instances.
> > 
> > (my apologies if this is obvious)
> 
> Okay, for example, since it's a BBS we need to have quick access to:
> 
> 	the latest poster
> 	the latest posting
> 	the latest registered member

Create a zope product that doesn't keep a history (otherwise the ZODB
will grow like nuts) and store these values in it whenever there is a
new port/registration. Then just read them when you need to.

Anyone see anything wrong with that approach?

> 
> All of these are complex queries (in the case of the latest registered
> member it's not even a query but a brute-force search).  What I think
> most ASP/PHP boards do is that they store these values in the database
> instead of querying for them all the time.
> 
> Since many of these can be found through queries, they don't really need
> persistent storage, but updates need to be seen by all threads; thus the
> requirement for ZODB if we use ZEO, but the possibility to use (even
> faster) shared memory if we don't use ZEO.
> 
> In effect we want to do something like this:
> 
> def getLatestPoster(self):
> 	if not hasattr(self, '_v_latestPoster'):
> 		self._v_latestPoster = whatever_query()
> 	return self._v_latestPoster
> 		return self._v_latestPoster
> 
> def setLatestPoster(self, poster):
> 	self._v_latestPoster = poster
> 
> But instead of having thread-local variables we wanted to use Dieter
> Maurer's SharedResource in order to share the cache between threads:
> 
> http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html
> 
> Regards,
-- 
Edward Muller - http://www.interlix.com - "Open Source Specialists"
Dedicated Zope Hosting - Web Hosting - Open Source Consulting
Network & PC Service & Support - Custom Programming
Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572
Jabber: edwardam at jabber.interlix.com - AIM: edwardam453 - ICQ: 287033