[Zope-dev] Bulletproof ZCatalog proposal

[Shane Hathaway]

On Thursday 07 June 2001 12:17, Phillip J. Eby wrote:
> At 09:34 AM 6/7/01 -0400, Shane Hathaway wrote:
> >One thing I didn't make clear in the proposal is that I'm interested
> > in repurposing ZCatalog as a general ZODB indexing mechanism and
> > essentially moving it down from the application layer to the
> > database layer.  Catalogs would be updated automatically (in a lazy
> > fashion to avoid performance penalties).  There would potentially
> > be a large number of catalogs, though, so the number of conflicts
> > would increase significantly.
> >
> >But I think I have a solution for all of the issues in conflict
> >resolution.  If you, or anyone else, is also interested in this,
> > please show support (if only by saying "please do this"!) :-)  I
> > can't work on it unless I have a reason to.
>
> Sounds cool.  At one time Ty and I thought we might do something like
> this by adding to a linked list of "pending updates" which the
> catalog would use to alter its search results, until the list got
> "too long", at which point it would post the actual updates.

That sounds like what I have in mind.

> The
> only catch was that this would still produce conflicts at the head
> end of the linked list.  :(  Of course, that was in the days before
> ZODB conflict resolution.  Nowadays, you could probably implement it
> as a simple sequence object with the conflict resolution method
> implemented.  But then there'd be the question of how to resolve
> conflicts if more than one thread or ZEO client decided to apply the
> queued changes...  a 100 conflicts vs 1 situation.  Ugh.

I was thinking the queue would only last the duration of a transaction 
and that the queue would be thread-specific.

> Anyway, I'd be really interested in seeing "a solution for all the
> issues in conflict resolution".

I was thinking that certain types of objects would be committed by the 
transaction manager before all others.  In this case, the catalog (or a 
special object in the catalog) would be committed first.  It would 
resolve all conflicts in the contained indices before they occur by 
replaying the changes in the persisted queues from the transaction 
history, then setting the _p_serial attributes to convince the storage 
that the conflicts have already been resolved.

> Will it help with arguments, too? 
> How about world peace?  ;)

Okay, maybe we haven't solved that. :-)

Shane