[Zope3-Users] MemoryError Evolving a ZODB

[Adam GROSZER]

On 12/11/2012 05:33 PM, Jeroen Michiel wrote:
>
> Hi,
>
> I'm having serious trouble getting my DB evolved to a new version. I'm
> runnong a Grok 1.4 site using ZODB 3.10.2
> The problem happens when I add a new index to a new catalog.
> As soon as the index is added, a subscriber from zop.catalog (I believe)
> will automatically loop over all objects in the DB trying to index them. For
> some reason, it apparently tries to keep all these objects into memory,
> while only a very small part of them are effectively need indexing, and even
> then, indexing shouldn't touch them.
> After some time of running I see the process taking 1.9GB of mem on windows,
> (or 3G on linux), and then I first get these errors:
>
> 2012-12-11 16:52:56,617 ERROR [ZODB.Connection] Couldn't load state for
> 0x0a45a2
> Traceback (most recent call last):
>    File
> "c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\Connection.py",
> line 856, in setstate
>      self._setstate(obj)
>    File
> "c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\Connection.py",
> line 910, in _setstate
>      self._reader.setGhostState(obj, p)
>    File
> "c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\serialize.py",
> line 612, in setGhostState
>      state = self.getState(pickle)
>    File
> "c:\users\jm.traficon-int\.buildout\eggs\zodb3-3.10.2-py2.6-win32.egg\ZODB\serialize.py",
> line 605, in getState
>      return unpickler.load()
> MemoryError
>
> I have not a single clue why it would need that much memory.
> I tried using savepoints, but that doesn't help.
> How can I see what exactly is eating all that memory, where do I start
> debugging this?
>
> ANY help appreciated!
>

Well it loads too many objects in a single transaction.
Doing this after some iterations (10k?, depends on your object sizes) 
helps usually:

def forceSavepoint(anyPersistentObject=None):
     transaction.savepoint(optimistic=True)

     if anyPersistentObject is not None:
         #and clear picklecache
         conn = anyPersistentObject._p_jar
         conn.cacheGC()


-- 
Best regards,
  Adam GROSZER
--
Quote of the day:
A liberal is someone too poor to be a capitalist and too rich to be a 
communist.