[Zope-dev] zcatalog and versions

[Oliver Bleutgen]

Thanks for the fast reply Casey.

Casey Duncan wrote:
> On Thursday 27 September 2001 12:48 pm, Oliver Bleutgen allegedly wrote:
>> Hi,
>> I'm resending this to zope-dev because on zope
>> nobody answered, it would be very nice if someone
>> could step up with a small hint.
>>
>>
>>
>> Can somenone briefly explain what exactly gets
>> locked in zope 2.3.3's catalog when it tries to
>> index an object which is hold in a version?
>> The whole catalog?

> Any Btree buckets (in the indexes) which get changed get locked, which can
> effectively lock out other changes to the catalog. This is a limitation of
> the way the indexes are implemented, for which, sadly, there is no easy
> fix.


>>
>> I found some answers which indicate that it has
>> to do with the catalog when we see version lock errors
>> where there shouldn't be any (from a naive POV).
>> I would like to know how far reaching these problems
>> are, because I'm currently rewriting an application,
>> and I might be able to work around that.

> I would suggest that if any objects are reindexed in a version, that no
> cataloged objects should be reindexed in any other version until that
> version
> is saved.

I hope I understand you correctly, but I'd say that if there's already 
a locked bucket, I've lost. There's no guarantee how long this 
particular version will stay uncommitted. I would also have to check 
anyhow whether there is a lock somewhere in the catalog's index.

> You could also get around that by deferring the indexing until the version
> is
> commited, but this will take some coding on your part. 

This seems easier - if I'm a little bit lax about when the indexing occurs.
Like making the object only index/reindex/unindex itself if it's not in
a version, and combining that with a nightly cronjob which reindexes
all (non-versioned) documents. 

With "some coding on your part" you mean making Version.py more intelligent?

Like instead of just doing commitVersion(s,'') doing the following:

1) search for objects which have been deleted in the 
  version
2) search for objects which want to catalog themselves 
  and are locked in version s
3) unindex the objects found in 1)
4) commit version
5) index/reindex objects found in 2)

Where it's not clear to me how to prevent that 3) and 5) will
not itself get versioned.


> Ultimately I think
> that ZCatalog should do this for you, or at least somehow let indexes have
> concurrent versioning (any volunteers?)

I should say that I really don't grasp this ZODB voodoo, but I 
suspect that this will also be not too easy. AFAIK, the decision 
to write "in" a version (and in which) is taken deep down in 
zope's innards.

> I am thinking about writing a fishbowl proposal for ZCatalog upgrades
> sometime next month, and this is one potential problem areas to address
> there, especially as things like the CMF make it more ubiquitous

If I don't misunderstand how versions work (not unlikely), it might
be necessary in the end rather to improve versions than the zcatalog 
in order to remedy this specific problem.

thanks again,
oliver