[Zope-dev] Bulletproof ZCatalog proposal

[Phillip J. Eby]

At 07:07 PM 6/7/01 -0400, Shane Hathaway wrote:
>"Phillip J. Eby" wrote:
>> That is, in ZPatterns one can specify triggers such as:
>> 
>> WHEN OBJECT DELETED, CHANGED CALL
>> someCatalog.manage_uncatalog(self.absolute_url(1))
>> WHEN OBJECT ADDED, CHANGED CALL
>> someCatalog.manage_catalog(self,self.absolute_url(1))
>
>After I read this again I realized what you were saying.  This
>capability of ZPatterns is very brittle, don't you think?

Yep.  That's why I've previously described ZPatterns as a "hack".  :)


>If the
>catalog is updated manually before the special ZPatterns object is added
>to the queue, the behavior is undefined AFAIK--either the later changes
>to the catalog will be ignored, will cause a conflict, or some objects
>will be written twice in the same transaction.

True.  But this behavior is avoidable through the use of subtransaction
commits, in the event that someone has to have transactions which update a
ZCatalog directly.  Usually, when someone is using catalogs with ZPatterns,
they use triggers to do all the updates and don't touch the catalog manually.

Note that I'm not saying this still isn't a hack.  But it's the best I
could do without either fixing the multi-commit issue in ZODB, or with some
kind of priority scheme.


>However, if we could specify transaction commit priorities, and the
>ZPatterns update came first, auto-indexing came second, and everything
>else followed, I think it would work.  Or perhaps ZPatterns should be
>able to register things that occur *before* the two-phase commit
>protocol.

Yep.  One of the last two times I spoke with Jim in person (either the
January DC visit or IPC 8, I forget which), he said something about it
maybe being a good idea to have some kind of priority system like that.
I'd love to see something like it exist, if it would make some of
ZPatterns' hackery unnecessary.

The implementation could consist of two subscription queues: ruleAgents and
indexingAgents.  ZCatalog would register in indexingAgents, and ZPatterns
objects would register in one or the other, usually ruleAgents.  (I can
think of some circumstances where it would be nice to use the
indexingAgents queue, but right now ZPatterns apps have to work around this
by defining their rules in execution priority order.)

Upon being told to perform a transaction or subtransaction commit, the
transaction would notify all the ruleAgents, and then all the
indexingAgents.  Objects could still subscribe to either queue while this
notifying is taking place.  (So that triggered actions could cause indexish
objects to register as indexingAgents, not to mention causing updated
objects to fire additional triggers.)

Once all agents in a queue are notified, that queue should be cleared so
that notifications are on a per-subtransaction basis.  Once both queues are
cleared, normal transaction behavior goes forward.

Hm.  That's simpler than I thought it was going to be.  Shoot, I can even
see how to implement it as a runtime patch, that would've been simpler than
all the shenanigans ZPatterns goes through to fake out the transaction
machinery...  and it's a better
implementation.  Ah well.  At the time I wanted to avoid trying to convince
Jim to add machinery to transactions "just" for ZPatterns, given that
ZPatterns wasn't a particularly established product at the time.

Let me know if you guys put something like this in, though, and I'll
definitely look at reworking ZPatterns to use the mechanism.  It could
potentially cut a lot of code out and improve the robustness at the same time.