[Zope3-Users] MemoryError Evolving a ZODB

[Jeroen Michiel]

I found the actual issue!
I am running a client/server setup: the client (C++ tool) runs tests on
devices and pushes the results to the server (Grok) with REST requests.
There was a mechanism that the server validated the results, and could
reject them if they didn't validate, so the client would rerun the test and
try to push it again. Also, if pushing results for some other reason failed
the client would keep trying. However, there was a bug in the server that if
validation failed, the wrong error was returned, but the results were
committed to the DB. So the client kept trying, and the server kept storing
but rejecting with the wrong error...

I had found and solved this issue a long time ago on my test setup while
developing, but I had no idea it had actually happened on my live setup,
too. 

With the circular references, I ended up with one massive set of data, that
didn't fit into memory while doing findObjectsProviding. I managed to remove
the redundant data in chunks and committing in between. Even committing
didn't drop the memory usage, so I had to restart the script multiple times.

Now my DB migration is running OK!

thx!


Jeroen Michiel wrote:
> 
> I was thinking along the same lines.
> I have found a spot in the code with circular references, and indeed
> (using heapy) it seems those are the objects (which happen to be quite big
> also) taking most of the memory. The main problem is now to get rid of
> them while staying within memory boundaries. It's a part of the code I
> implemented first in this site, and it was also the time I started working
> with Zope/Grok, so I would approach things quite differently now, knowing
> what I know about the ZODB that I didn't know then...
> I guess multiple transactions will be needed, indeed.
> 
> It originally was just a matter of indexing objects, but wanting to get
> rid of the circular references, it has become a db migration...
> 
> Thanks for the advice, I'll keep you posted on how I fixed it (if I ever
> do), might be interesting for other people, too.
> 
> Regards,
> Jeroen
> 
> 
> Alexandre Garel-2 wrote:
>> 
>> Le 12/12/2012 09:39, Jeroen Michiel a écrit :
>>> Thanks for the reply!
>>>
>>> I already tried
>>> transaction.savepoint()
>>> every minute, but that didn't help: I only saw the memory usage dropping
>>> the
>>> first time, but never after.
>>>
>>> I changed the code to what you suggested, but it still doesn't seem to
>>> help.
>>> Something must be wrong somewhere along the line, but I don't have a
>>> clue
>>> where to begin looking.
>>> Would using something like guppy (or heapy, or what it's called) reveal
>>> something?
>>>
>>> Could it be something about objects with circular references not being
>>> able
>>> to be garbage-collected?
>>> The objects in my DB are quite complex, so something like that might
>>> actually be happening.
>> 
>> Hello
>> 
>> my suggestions might be silly, but in case… :
>> 
>> 1- is that that you modify a lot of objects (and big objects) in which 
>> case savepoints may not save you (as my wild guess is that savepoints 
>> will only drop objects participating in computation but not modified). 
>> If it's just re-indexing it's strange as the only thing changing would 
>> normally be the index.
>> 
>> 2- if it's a kind of migration for your database, do you really need to 
>> have it done in one transaction. Could you save your database, run your 
>> migration in multiple commit (transaction.commit() instead of 
>> transaction.savepoint()) then if it goes wrong, restore old file and if 
>> it's ok, well it's ok :-)
>> 
>> Hope it helps,
>> 
>> Alex
>> 
>> -- 
>> Alexandre Garel
>> 06 78 33 15 37
>> 
>> _______________________________________________
>> Zope3-users mailing list
>> Zope3-users at zope.org
>> https://mail.zope.org/mailman/listinfo/zope3-users
>> 
>> 
> 
> 
-- 
View this message in context: http://old.nabble.com/MemoryError-Evolving-a-ZODB-tp34784598p34809613.html
Sent from the Zope3 - users mailing list archive at Nabble.com.