[Zope-dev] spider cache blowout [was RE: [ZODB-Dev] Generational Caching]

[Dylan Jay]

> -----Original Message-----

> From: Chris McDonough [mailto:chrism at plope.com]

> Sent: Wednesday, 8 June 2005 11:56 AM

> To: Dylan Jay

> Cc: zodb-dev at zope.org

> Subject: Re: [ZODB-Dev] Generational Caching

>

> One way to potentially soften the impact of cache busting by spiders

> might be to allow Zope to choose a particular ZODB connection based on

> request parameters (like sessionid or requesting ip address, or most

> likely user agent in the case of "legitimate" spiders). This is a

> modification to Zope that would be largely independent of ZODB,

> however, so might be more appropriate to discuss on zope-dev.

I think this sounds like a very smart idea. I'll forward this to the 
zope-dev list.


> On Wed, 2005-06-08 at 10:20 +1000, Dylan Jay wrote:

> > I've been thinking about the problems of memory use. One use case

> > that

> seems

> > hard is web crawlers. That will wake up a lot of objects for just a

> single

> > read possibly getting rid of many often used objects from the zodb

> cache.

> > Has anyone investigated the idea of a generational cache?

> >

> > Another idea I had was asymetric cache sizes. I believe that zope

> > uses a single thread if all requests are serial and just uses the

> > other

> connections

> > when similtanious requests are handled. Since some threads will be

> > less

> used

> > than others perhaps there could be cache settings so memory can be

> > minimised.

> >

>






-- 
Dylan Jay
---
http://www.meetmemap.com - makes giving directions easy
http://www.pretaweb.com - Update-it-yourself, low cost websites
http://www.eatmanifesto.com - against all dumbing down of food
Y!:dylan_jay ICQ:520341 MSN:djsq at hotmail.com Ph:+61423198306