[ZODB-Dev] Recursion Error with ape and ZCatalog

[Shane Hathaway]

Klaus Wölfel wrote:
> I experience two problems with the otherwise great Ape storage.

I'm pleased to hear Ape is being put to good use!

> I've got a 
> ZCatalog with 22 indexes, the half of them being textindexes. When indexing 
> more than about 3000 objects I get the following error:
> 
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 104, in publish
>   Module Zope.App.startup, line 221, in commit
>   Module ZODB.Transaction, line 233, in commit
>   Module ZODB.Transaction, line 348, in _commit_objects
>   Module apelib.zodb3.connection, line 263, in commit
>   Module apelib.core.io, line 152, in serialize
>   Module apelib.core.serializers, line 93, in serialize
>   Module apelib.zodb3.serializers, line 214, in serialize
> RuntimeError: maximum recursion depth exceeded

This doesn't happen when you index a smaller number of objects?

> I tried it with ape fs storage and with the sql storage with mysql, the latter 
> on both linux and win.
> 
> The other problem occurs when I use ZCtextIndex instead TexindeexNG as 
> textindex. Then after adding only some object and then clearing the Catalog i 
> get this error:
> 
> Traceback (innermost last):
>   Module ZPublisher.Publish, line 104, in publish
>   Module Zope.App.startup, line 221, in commit
>   Module ZODB.Transaction, line 233, in commit
>   Module ZODB.Transaction, line 348, in _commit_objects
>   Module apelib.zodb3.connection, line 263, in commit
>   Module apelib.core.io, line 152, in serialize
>   Module apelib.core.serializers, line 93, in serialize
>   Module apelib.zodb3.serializers, line 238, in serialize
> RuntimeError: Unable to pickle the '_catalog' attribute, <Catalog instance at 
> 4142e500>, of <ZCatalog instance at 413bf830> at '9'.  Cannot pickle 
> <extension class Acquisition.ImplicitAcquirerWrapper at 408ab500> objects.
> 
> The same error occurs when adding a normal "old" textindex.

Interesting.  The catalog is trying to store an acquisition wrapper, 
which is a bug in the catalog.  However, I thought I fixed this one a 
while back.  I wonder if I could make that error message display just a 
little more info.  (BTW that OID of '9' is nice, isn't it?  Ape OIDs are 
really simple now.)

> I suspect that ape doesn't know how to store either the ZCatalog, the Catalog, 
> the  ZCTextIndex or the required lexicon, so I added this to the apeconf.xml 
> file:
> 
> <mapper name="anyfile">
>  <use-for class="Products.ZCatalog.Catalog.Catalog" />
>  <use-for class="Products.ZCatalog.ZCatalog.Catalog" />
>  <use-for class="Products.ZCTextIndex.ZCTextIndex.ZCTextIndex" />
>  <use-for class="Products.ZCTextIndex.ZCTextIndex.PLexicon" />
> </mapper>
> 
> but it didn't help. Can anybody give me a hint what I'm doing wrong here?

You're doing nothing wrong.  ZCTextIndex apparently needs a minor fix 
somewhere.  (Standard ZODB hides this error.)

> Then I have some questions: In my Product all about 10,000 objects are stored 
> in an BTree so the ape fs-storge stores them all in one file. Is this a 
> problem - does this file needs to get somehow packed like Data.fs? Is the 
> whole file loaded in memory when accessing one of the objects?

Yes, by default Ape loads and stores database-like things as one big 
file.  We should look at serializing BTrees in a more scaleable manner.

Shane