[ZODB-Dev] Cache Query (why doesn't RAM usage ever drop?)

[Toby Dickenson]

On Monday 28 October 2002 6:48 pm, Chris Withers wrote:
> Toby Dickenson wrote:
> > On Friday 25 October 2002 1:12 pm, Chris Withers wrote:
> >>apart from the online help now being out of
> >>date ;-)
> >
> I just hit the help button:
> http://x/HelpSys?help_url=3D/Control_Panel/Products/OFSP/Help/Database-=
Manage
>ment_Database.stx

I dont see anything wrong with the 2.6 version????

> Sadly, I see the most crucial problem hasn't really been addressed: you=
 can
> still top a server by having, say, a random python script that drags in=
 a
> stupid number of objects between transaction boundaries.

A malicious script? yes.  I think this is a bogus argument for a change t=
o=20
ZODB; a malicious script could also bring the server down by creating lar=
ge=20
numbers of non-persistent objects

A well-intentioned script? you just need to call the garbage collecter=20
intermittantly (plus use subtransactions if you are modifying large numbe=
rs=20
of objects) and everything will be happy.

> I wonder if it'd
> be possible to set a hard limit on the memory used by Zope and somehow =
get
> the cache to enforce it?

The problem is not the hardness of the limit, but rather that the limit i=
s=20
applied at the right time. Any automatic enforcement mid-transaction is=20
liable to cause problems (for example, think about _v_ attributes)

> So if I've got a 0.5Mb text file, it doesn't have to be loaded into mem=
ory
> to be indexed?

If it is using Zope's 'File' object then the 500k block-o-bytes will have=
 been=20
split into 7 64k chunks, and each chunk stored in a ZODB object.

It all needs to be loaded into memory for indexing, but each chunk is man=
aged=20
seperately by the ZODB cache.

> >>and in there anything that can be doen to make that happen?
> >
> > Restart Zope?
>
> That's the current thinking, but that's a but scummy...

I can suggest a change to your indexing loop that will make this happen=20
automatically. Would that help? ; -)