[ZODB-Dev] Cache Query (why doesn't RAM usage ever drop?)

[Chris Withers]

Toby Dickenson wrote:
>>Not malicious, consider this:
>>
>>for brain in context.some_catalog(an_index='fish'):
>>   object = brain.getObject()
>>   object.doSomeMaintenence()
>>
>>...now if an_index goes walkies, you've just loaded all the obejcts you've
>>cataloged into memory by mistake.
> 
> Yes. IMO this script *needs*:
> 
> 1. either batching, as is traditional for a search results page. The search 
> might be huge, but memory useage will be controlled if the batch size is 
> reasonable:

This isn't a search results page, it's an object maintenance script. I find thsi 
pattern extremely useful.

> 2. or a garbage collection tickler as part of the loop. See ZCatalog's 
> "reindex all" implementation for an example.

Can this tickling be done in a TTW script?

> Probably not, but I dont do much ttw work.

Touché. Lots of people do, and they're the most likely to topple their servers 
this way. I've had just such a thing going on this morning, on 2.6.0, from 
someone accidentally doing objectValues or some such on a BTreeFolder that 
contains 6,000 odd big chunky objects.


>>how would raising a MemoryError (as happens when you run of RAM for real)
>>affect these?
> 
> Not a bad idea. This could be done easily and efficiently in Connection.py

:-)

> Of course it still doesnt help with the case of memory usage by non-persistent 
> objects.

Well, a partial fix is better than no fix, and persistent objects are the things 
I'm worrying about.

>>>I can suggest a change to your indexing loop that will make this happen
>>>automatically. Would that help? ; -)
>>
>>What would the change be?
> 
> sys.exit(1)

Hmmm... somewhat less re-rentrant than I'd hoped ;-)

Anyway, what can I do to help make the raising of a MemoryError a reality?

That said, I'd prefer to see code that dumps an old object out of the cache to 
bring the new one in rather than a MemoryError.

Sure, this could lead to processor loading, but that's better than 
machine-dying-due-to-running-out-of-RAM or confused end users being presented 
with MemoryErrors when the machine has plenty of memory left.

cheers,

Chris