RE: [Zope-dev] How to make Zope fail nicely under high load?
Toby wrote:
One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB.
Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog.
I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances.
(my apologies if this is obvious)
Okay, for example, since it's a BBS we need to have quick access to: the latest poster the latest posting the latest registered member All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time. Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO. In effect we want to do something like this: def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster def setLatestPoster(self, poster): self._v_latestPoster = poster But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads: http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html Regards, -- Bjorn
using the _v_ variables won't work for this since different threads won't agree on who was latest. Its definitely a bad idea to do queries for these three data points. What about putting them in the temp_folder then you can still use ZEO with a bunch of clients that will all share the same data but you don't need to write the values to disk every time they change. Even putting them in a normal ZODB storage would be a _lot_ faster than doing a query. -EAD On Feb 12, 2004, at 8:19 AM, Bjorn Stabell wrote:
Toby wrote:
One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB.
Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog.
I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances.
(my apologies if this is obvious)
Okay, for example, since it's a BBS we need to have quick access to:
the latest poster the latest posting the latest registered member
All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time.
Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO.
In effect we want to do something like this:
def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster
def setLatestPoster(self, poster): self._v_latestPoster = poster
But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads:
http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html
Regards, -- Bjorn
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Thu, 2004-02-12 at 07:19, Bjorn Stabell wrote:
Toby wrote:
One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB.
Storing that in ZODB would be a bad idea. Theres no reason to think that this cache would be faster than ZCatalog.
I dont see why you would be *required* to store in ZODB... just dont share your cache between publisher threads on different Zope instances.
(my apologies if this is obvious)
Okay, for example, since it's a BBS we need to have quick access to:
the latest poster the latest posting the latest registered member
Create a zope product that doesn't keep a history (otherwise the ZODB will grow like nuts) and store these values in it whenever there is a new port/registration. Then just read them when you need to. Anyone see anything wrong with that approach?
All of these are complex queries (in the case of the latest registered member it's not even a query but a brute-force search). What I think most ASP/PHP boards do is that they store these values in the database instead of querying for them all the time.
Since many of these can be found through queries, they don't really need persistent storage, but updates need to be seen by all threads; thus the requirement for ZODB if we use ZEO, but the possibility to use (even faster) shared memory if we don't use ZEO.
In effect we want to do something like this:
def getLatestPoster(self): if not hasattr(self, '_v_latestPoster'): self._v_latestPoster = whatever_query() return self._v_latestPoster return self._v_latestPoster
def setLatestPoster(self, poster): self._v_latestPoster = poster
But instead of having thread-local variables we wanted to use Dieter Maurer's SharedResource in order to share the cache between threads:
http://www.dieter.handshake.de/pyprojects/zope/SharedResource.html
Regards,
-- Edward Muller - http://www.interlix.com - "Open Source Specialists" Dedicated Zope Hosting - Web Hosting - Open Source Consulting Network & PC Service & Support - Custom Programming Phone: 417-862-0573 - Cell: 417-844-2435 - Fax: 417-862-0572 Jabber: edwardam@jabber.interlix.com - AIM: edwardam453 - ICQ: 287033
participants (3)
-
Bjorn Stabell -
Edward Muller -
Erik A.Dahl