[Zope-dev] how bad are per-request-write-transactions

[Casey Duncan]

Paul Everitt wrote:
> 
> I don't agree that high write is always forbidden.  I think there are 
> plenty of cases where this can work.  It simply becomes unworkable much 
> sooner than other data systems (e.g. a relational database or FS-based 
> solution).

I agree, but I loath to approve of any solution which demands a write 
for every read of an object.

> For instance, think about bloat for a second.  Let's be crazy and say it 
> takes 100 bytes to store an integer representing a count.  Let's say you 
> write once a second.  That's under 7Mb a day (per counter).  Combined 
> with hourly packing, that might be well within limits.

Yes, but the counter is not the only thing written. The whole containing 
object is written out to the storage. Now that doesn't include binary 
data (such as image and file data), but it does include any primitive 
data stored in the object's attributes (strings, lists, dicts, etc).

Hourly packing seems like a blunderbus solution to the bloat problem. 
You can't tell me that won't kill performance...

> Let's take the next step and say that you can live with a little 
> volatility in the data.  You write an object that caches ten seconds 
> worth of writes.  Whenever a write comes in at the over-ten-second mark, 
> you write the _v_ attribute to the persistent attribute.  There's an 
> order of magnitude improvement.

Only if you run single threaded. For multi-threaded Zope apps (the 
default), you would need to use a transient object which introduces its 
own complexities.

> Finally, you store all your counters in a non-versioned storage.  Now 
> you have *no* bloat problem. :^)

Right, the transient object or something else that writes to disk. Now 
you have to make sure the counters can be related to the object 
robustly. Bookkeeping... This is certainly a possibility, I would 
hesitate to argue for it on the notion it is less complex though.

> Regarding performance, maybe his application isn't doing 50 
> requests/second and he'd be willing to trade the slight performance hit 
> and bloat for a decrease in system complexity.

That could be a good trade, I just wanted to make sure the issues were 
known.

> All of the above has downsides as well.  My point, though, is that we 
> shouldn't automatically dismiss the zodb as inappropriate for *all* 
> high-write situations.  In fact, with Andreas and Matt Hamilton's 
> TextIndexNG, you might even be able to write to catalogued applications 
> at a faster rate than one document per minute. :^)

Of course not, but the obvious and easiest solution (just incrementing a 
counter on the objects on every read) is probably not the best solution.

-Casey