[Zope-dev] how bad are per-request-write-transactions

[Paul Everitt]

ForkedStorage, I like it simply for the coolness of the name. :^)

But it sparked a different kind of idea, leveraging a pattern that might 
emerge in Zope 3.

Let's say we had a queue in Zope.  We could asynchronously send changes 
into the queue.  Later, based on some policy (e.g. idle time, clock 
ticks, etc.), those changes would be enacted/committed.

Imagine the queue itself is in a different storage, likely 
non-versioned.  Imagine that the queue is processed every N seconds.  It 
takes all the work to do and performs it, but in a subtransaction.

Thus you might send the queue ten increments to a counter, but only one 
will be committed to the main storage.

To make programmers have to think less about the queue (send in the 
object reference, the method to use, and the parameters), you could make 
it look like a special form of subtransactions.  That is, you say:

   tm.beginQueuingTransactions()
   self.incrementCounter()
   self.title='Simple change'
   self.body = upload_file
   tm.endQueueingTransactions()

At the transaction level, all enclosed changes are queued for later 
commit.  You don't have to think any differently than rather object 
state management.

This pattern applies better when you have a lot of document cataloging 
to be done.  A separate process can wake up, make a ZEO connection, and 
process the queue.  I don't think that indexing documents *has* to be a 
transactional part of every document save.

Under this cron-style approach, you also pay less of a conflict-error 
penalty, as you can increase the backoff period.  There's no web browser 
on the other end, impatiently waiting for their flaming logo. :^)

Ahh well, fun to talk about.  Maybe this time next year we can repeat 
the conversation. :^)

--Paul

Shane Hathaway wrote:
> Jeremy Hylton wrote:
> 
>>>>>>> "CM" == Chris McDonough <chrism@zope.com> writes:
>>>>>>
>>>>>>
>>
>>   >> Completely agreed.  My disagreement is portraying the counter
>>   >> problem as impossible with the zodb.  I think some people, as
>>   >> evidenced by some of the responses, are willing to live with the
>>   >> tradeoffs.  Other people will find managing a log file on disk to
>>   >> be a more manageable solution.
>>
>>   CM> It would be best to make make a dual-mode undoing and nonundoing
>>   CM> storage on a per-object basis.
>>
>> I'd really like to do this for ZODB4, but it seems hard to get it into
>> FileStorage, without adding automatic incremental packing to
>> FileStorage.
>>
>> Example: Object A is marked as save enough revisions to do a single
>> undo.  When a transaction updates A and makes older revisions
>> unnecessary, there's no obvious way to remove them without doing a
>> pack.  We could write a garbage collector that removed unneeded things
>> (as opposed to packing everything to a particular date), but it
>> doesn't seem very useful if it needs to be run manually.
> 
> 
> One idea I've been floating in my head is the idea of a "forked" 
> storage, where some objects are stored in an undoable storage and others 
> are stored in a non-undoable storage.  I could try to explain it in 
> English but pseudocode is easier:
> 
> 
> class ForkedStorage:
> 
>     def __init__(self, undoable_storage, non_undoable_storage):
>         self.undoable = undoable_storage
>         self.non_undoable = non_undoable_storage
> 
>     def store(self, oid, data, serial):
>         if not serial or serial == '\0' * 8:
>             # For new objects, choose a storage.
>             want_undo = self.wantUndoableStorage(data)
>             if want_undo:
>                 storage = self.undoable
>             else:
>                 storage = self.non_undoable
>         else:
>             # For existing objects, use the storage chosen previously.
>             if self.undoable.load(oid):
>                 storage = self.undoable
>             else:
>                 storage = self.non_undoable
>         storage.store(oid, data, serial)
> 
>     def load(self, oid):
>         data, serial = self.undoable.load(oid)
>         if not data:
>             data, serial = self.non_undoable.load(oid)
>             if not data:
>                 raise POSException, 'data not found'
>         return data, serial
> 
>     def wantUndoableStorage(self, data):
>         u = cpickle.Unpickler()
>         module, name = u.loads(data)
>         class_ = getattr(__import__(module), name)
>         if getattr(class_, '_p_undoable', 1):
>             return 1
>         else:
>             return 0
> 
> 
> Only a simple idea. :-)
> 
>> Also, how would you specifiy the object's packing policy?  I'm
>> thinking an _p_revision_control attribute or something like that.  If
>> the attribute exists on an object, it sets a particular policy for
>> that object.  
> 
>  > > Do individual transactions need to play in this game, too?  I'm
> 
>> imagining a use case where an object is marked as "no revisions" but
>> you want to be able to undo a particular transaction.  I'm not sure if
>> that means :
>>
>>     - you can undo the transaction, but the "no revisions" object
>>       keeps its current state.
>>
>>     - you can undo the transaction, and because the transaction is
>>       specially marked as undoable, there actually is a revision
>>
>>     - you can't undo the transaction
>>
>> The first choice seems appropriate for a counter (I think), but I'm
>> not sure if it makes sense for all possible revision-less objects.
> 
> 
> The first choice also makes sense for a catalog.  Here's another 
> possible variation: transactions that involve *only* non-undoable 
> objects are non-undoable; all other transactions are undoable and revert 
> the revision of non-undoable objects as well.
> 
> Shane
> 
> 
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev@zope.org
> http://lists.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce
> http://lists.zope.org/mailman/listinfo/zope )