[ZODB-Dev] Ordering before commit hooks

[Jim Fulton]

Dieter Maurer wrote:
> Jim Fulton wrote at 2005-8-31 10:20 -0400:
> 
>>...
>>
>>>Thus my proposal could be seen as well as a request to change (or to
>>>make more flexible) the policy already in place. (Which was my use case)
>>
>>Except that, as Tim and I have pointed out, the order gimmic didn't
>>deal at all well with the use case of making something go last.
> 
> 
> Did you read my message?

Yes.  Did you read my response> It was in the same note
I wrote to Julien, so you probably missed it. Sorry.

>   The "order gimmic" provides a global scale along which
>   an *integrator* can align hooks according to their
>   relative needs (if they do not conflict altogether).
> 
>   He (the integrator!) assigns orders to all hooks and can
>   let individual hooks go at the start, the end, in the middle
>   whereever it is necessary -- in a an (quite) easy and
>   understandable way (the "import order" of packages might
>   be an approximation -- but far less understandable and flexible
>   and more error prone).

To support this, you, will need some ZCML directives.
building an add-on that provides a registry for ordered
callbacks is a very small effort in comparisn.

> 
> I am starting to get a bit tired with this discussion.

Me too.

>   I live since a long time with enhanced versions of Zope/ZODB.
>   This can stay so, if improvements take too much effort
>   to get them into the core.

I'm sorry to hear that.  There are a number of reasons for this.
I'm not trying to create obsticals.  I am extremely interested
in getting ZODB to be much cleaner.  We are chipping away at
making ZODB simpler and easier to understand.  For that reason,
I'm gaurding the APIs pretty aggressively.

I'm happy to see contributions coming in.  The initial callback
capability was contributed, and I'm glad to have it.  The fairly
large contribution of Blob support is also very welcome.

>   Another alternative would be to publish a standard extension package, that
>   provides the (trivial) ordering.
>   The only difficulty with this: It provides a *global*
>   coordination facility; it would need to be available
>   in all systems that anticipate a hook coordination need.
>   The only advantage the ZODB has wrt. this solution:
>   It is available everywhere.

Only because it would be shared API. An add-on could be available
everywhere too. It just needs to be published.  I don't even object
to this being published with Zope if people do indeed find it useful.

> 
>>...
>>
>>>Well yes and no. What if I want to go first instead ? Or in the middle
>>>of 2 independently (external to my system) defined hooks ?
>>
>>What if you want one callback to go immediately after another?  What
>>if you want a callback to be called only if another wasn't called (or
>>was called). These are also legitimate policies that the order gimmick
>>doesn't address.  They are policies that can be handled by a higher-level
>>plug-in.
> 
> 
> That's true -- apart from the fact that you have not yet shown
> the real life use cases for these hypothetical policies.

Actually, no one has shown real life use cases for ordering.
The only real use case I've seen is for going last, and the
order gimmick doesn't support it well.

> It appears to me that these policies entail significant knowledge
> among the various hooks about themselves. Such cases may
> need specific solutions.

Exactly.

> 
> Our use cases (at least mine) for the "order gimmick" come from a common 
> root: the wish to perform a summary operation once before
> the commit rather then partial operations over and over again
> during the transactions: e.g.:
> 
>   *  one summary SQL UPDATE for the complete modified record
>      rather than an SQL UPDATE for each attribute modification
> 
>   *  one summary "index_object" rather than lots
>      of "index_object" calls for each individual modification.

These are valid use cases. This is why we added the callback
mechanism. Neither of these require the order gimmick.

> The have order requirements (they should come after all possible
> modifications that should go into the summary operation) but
> they are not too strong.

Ordering doesn't help this at all.  The callback mechanism
itself provides the ability to execute special operations,
like indexing after all normal operations.

If you think that callback operations might cause additional
changes that might cause indexing, then you can try to get the
indexing to go last, but ordering gimmick dosn't do that.


> The "order gimmick", while not universal, provides an easy
> and understandable facility to handle a large group of such
> use cases.

Huh? It doesn't help these use cases at all.

> 
>>...
>>
>>>Jim, honestly where is the complication ?
>>
>>The API gets an extra argument who's use is far from clear.  People
>>reading the API would rightly wonder why they would use it.
> 
> 
> Ordering the execution of hooks is not that unfamiliar.
> Doing it by means of numbers, neither...

It's not familiar to me.  It's also not effective.  I made the mistake
of using this hack in zope.configure and it obfuscated the code.
All I wanted was to cause certain actions to be executed after
normal non-delayed actions. An API that allowed me to say that
would have been clearer than what I did. (order=9999999).

> 
>>What
>>does it gain them?
> 
> 
> There are few developpers who will use the hook, many hooks
> will not need an explicit ordering. A developper who has
> specific needs with guess what "order" is for ;-)

That's what I'm afraid of. The gimmick will encourage code
that missuses order.  There will be snippets of code that
set an order. The reader of the code won't know if they thought
the intent was to go last, or to fit into some mysterious
non-obvious integer scheme. For example Julien used 100 to spell
"last". Reading the code, I never would have guessed that.
I would be wondering where to find the code with an order of
200. :)

> 
>>If they wanted something to go last (or first),
>>what argument should they provide?
> 
> 
> Contrary to your opinion, hooks will probably rarely have the
> need to be called as the very first or the very last one.
> Usually, the requirements will be: after I was executed
> some (hook) specific things should not happen anymore (because,
> otherwise, I would need to run again).

You might be right, but the only credible use case I've seen
is to go last.  The example Julien has something going first
and something going last.  Looking at the code, I don't see
why the thing he wanted to go first needed to do so.
The example you gave above had subscribers that wanted to go last.
What evidence to I have of other orderings?  Julien did give
a "future need" to order two specific systems.  IMO, having
one system implement that requirement explicitly by invoking
the other would be much clearer than having the
requirement implemented implicitly through integers orders.

> 
>>...
>>Right. You need to use a common API. So define one and get people to use
>>it. You *want* the common API to be defined by ZODB.
> 
> 
> Indeed, because then (almost) all ZODB users with hook synchronization
> needs can use it.

All people with hook synchronization needs can use an add on. It doesn't
need to be in ZODB for them to do this.

> 
>>That's too low level.
> 
> 
> The amount of "common components" available for the resource *coordination*
> of independently developed packages is not so large.
> 
> 
> If it does not go into the ZODB, then almost surely, competing
> solutions will spring up that locally (e.g. for CPS, Plone, MySelf, Archetypes,
> ...) will do the job, causing pain when packages from these different worlds
> are integrated.

Then propose a common framework.  I have no problem with a common
framework. Put it in Zope if you want. It doesn't belong
in ZODB.

> 
> It would be nice, if a common solution could be established
> at or below CMF (as most of the worlds above are above it) -- but
> after this discussion, I fear, it will not happen...

Why? (That was rhetorical :) There are lots of layers above ZODB
where this can go.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org