[Zope-dev] how bad are per-request-write-transactions

[Shane Hathaway]

Jeremy Hylton wrote:
>>>>>>"CM" == Chris McDonough <chrism@zope.com> writes:
>>>>>
> 
>   >> Completely agreed.  My disagreement is portraying the counter
>   >> problem as impossible with the zodb.  I think some people, as
>   >> evidenced by some of the responses, are willing to live with the
>   >> tradeoffs.  Other people will find managing a log file on disk to
>   >> be a more manageable solution.
> 
>   CM> It would be best to make make a dual-mode undoing and nonundoing
>   CM> storage on a per-object basis.
> 
> I'd really like to do this for ZODB4, but it seems hard to get it into
> FileStorage, without adding automatic incremental packing to
> FileStorage.
> 
> Example: Object A is marked as save enough revisions to do a single
> undo.  When a transaction updates A and makes older revisions
> unnecessary, there's no obvious way to remove them without doing a
> pack.  We could write a garbage collector that removed unneeded things
> (as opposed to packing everything to a particular date), but it
> doesn't seem very useful if it needs to be run manually.

One idea I've been floating in my head is the idea of a "forked" 
storage, where some objects are stored in an undoable storage and others 
are stored in a non-undoable storage.  I could try to explain it in 
English but pseudocode is easier:


class ForkedStorage:

     def __init__(self, undoable_storage, non_undoable_storage):
         self.undoable = undoable_storage
         self.non_undoable = non_undoable_storage

     def store(self, oid, data, serial):
         if not serial or serial == '\0' * 8:
             # For new objects, choose a storage.
             want_undo = self.wantUndoableStorage(data)
             if want_undo:
                 storage = self.undoable
             else:
                 storage = self.non_undoable
         else:
             # For existing objects, use the storage chosen previously.
             if self.undoable.load(oid):
                 storage = self.undoable
             else:
                 storage = self.non_undoable
         storage.store(oid, data, serial)

     def load(self, oid):
         data, serial = self.undoable.load(oid)
         if not data:
             data, serial = self.non_undoable.load(oid)
             if not data:
                 raise POSException, 'data not found'
         return data, serial

     def wantUndoableStorage(self, data):
         u = cpickle.Unpickler()
         module, name = u.loads(data)
         class_ = getattr(__import__(module), name)
         if getattr(class_, '_p_undoable', 1):
             return 1
         else:
             return 0


Only a simple idea. :-)

> Also, how would you specifiy the object's packing policy?  I'm
> thinking an _p_revision_control attribute or something like that.  If
> the attribute exists on an object, it sets a particular policy for
> that object.  
 > > Do individual transactions need to play in this game, too?  I'm
> imagining a use case where an object is marked as "no revisions" but
> you want to be able to undo a particular transaction.  I'm not sure if
> that means :
> 
>     - you can undo the transaction, but the "no revisions" object
>       keeps its current state.
> 
>     - you can undo the transaction, and because the transaction is
>       specially marked as undoable, there actually is a revision
> 
>     - you can't undo the transaction
> 
> The first choice seems appropriate for a counter (I think), but I'm
> not sure if it makes sense for all possible revision-less objects.

The first choice also makes sense for a catalog.  Here's another 
possible variation: transactions that involve *only* non-undoable 
objects are non-undoable; all other transactions are undoable and revert 
the revision of non-undoable objects as well.

Shane