[Zope] ZCatalog len problem (resolved)

[Jonathan]

> ----- Original Message ----- 
> From: "Andreas Jung" <lists at zopyx.com>
> To: "Jonathan" <dev101 at magma.ca>; <zope at zope.org>
> Sent: Saturday, March 24, 2007 10:43 AM
> Subject: Re: [Zope] ZCatalog len problem
>
> --On 24. März 2007 10:16:46 -0400 Jonathan <dev101 at magma.ca> wrote:
>
>>> I have a ZCatalog and in the ZMI 'Catalog' tab it reports:
>>>
>>> Returns:
>>>
>>> len(sRes)=  1820
>>> cnt=  1842
>>>
>>
>> The result remains the same after reindexing the whole catalog?
>
> Yes.  I have an application that has 5 different ZCatalogs. The same 
> problem occurs in 3 of them. It seems that a ZCatalog with less than ~500 
> records is ok, but the len problem increases with the number of records in 
> the ZCatalog:
>
> actual        len        diff
>
> 108           108       0
> 322           322       0
> 613           607       6
> 1842         1820     22
> 3628         3590     38
>
>
>
>>> It looks like the __len__ method in Catalog.py is not returning the
>>> correct value - it returns self._length() which is set by the following
>>> method (also in Catalog.py):
>>>
>>>     def migrate__len__(self):
>>>        """ migration of old __len__ magic for Zope 2.8 """
>>>         if not hasattr(self, '_length'):
>>>             n = self.__dict__['__len__']()
>>>             del self.__dict__['__len__']
>>>             self._length = BTrees.Length.Length(n)
>>>
>>
>> Is this migration code for pre-2.8 catalogs. This change was necessary
>> because the new ExtensionClass implementation did not longer work
>> with the scary implementation of the __len__ functionality of the Catalog
>> class.
>>
>> If the result remains the same after reindexing the catalog there might 
>> be
>> an issue with the increment/decrement calls of the _length counter
>> with catalogObject/uncatalogObject...however no idea why...
>
> Clearing the catalog and then running 'Find Objects' returns the same 
> incorrect number every time (at least the error is consistent!). I have 
> also loaded records into the ZCatalogs via a python script/external method 
> and the record counts are exactly the same as with the 'Find Objects'.

I wrote a bunch of unittests and could not recreate the problem, so i 
reinstalled zope and rebuilt the application infrastructure (ZCatalogs, etc) 
and reloaded all the data from scratch and the problem went away!

Good idea of using the unittests to track down the problem, thanks Andreas.

Jonathan