spider cache blowout [was RE: [ZODB-Dev] Generational Caching]

9 Jun 2005


      ...
-----Original Message-----
...
From: Chris McDonough [mailto:chrism@plope.com]
...
Sent: Wednesday, 8 June 2005 11:56 AM
...
To: Dylan Jay
...
Cc: zodb-dev@zope.org
...
Subject: Re: [ZODB-Dev] Generational Caching
...

...
One way to potentially soften the impact of cache busting by spiders
...
might be to allow Zope to choose a particular ZODB connection based on
...
request parameters (like sessionid or requesting ip address, or most
...
likely user agent in the case of "legitimate" spiders). This is a
...
modification to Zope that would be largely independent of ZODB,
...
however, so might be more appropriate to discuss on zope-dev.
I think this sounds like a very smart idea. I'll forward this to the 
zope-dev list.
...
On Wed, 2005-06-08 at 10:20 +1000, Dylan Jay wrote:
...
...
I've been thinking about the problems of memory use. One use case
...
...
that
...
seems
...
...
hard is web crawlers. That will wake up a lot of objects for just a
...
single
...
...
read possibly getting rid of many often used objects from the zodb
...
cache.
...
...
Has anyone investigated the idea of a generational cache?
...
...
...
...
Another idea I had was asymetric cache sizes. I believe that zope
...
...
uses a single thread if all requests are serial and just uses the
...
...
other
...
connections
...
...
when similtanious requests are handled. Since some threads will be
...
...
less
...
used
...
...
than others perhaps there could be cache settings so memory can be
...
...
minimised.
...
...
...
-- 
Dylan Jay
---
http://www.meetmemap.com - makes giving directions easy
http://www.pretaweb.com - Update-it-yourself, low cost websites
http://www.eatmanifesto.com - against all dumbing down of food
Y!:dylan_jay ICQ:520341 MSN:djsq@hotmail.com Ph:+61423198306

Dylan Jay

tags

participants (1)