[Zope-dev] how bad are per-request-write-transactions

[Paul Everitt]

I don't agree that high write is always forbidden.  I think there are 
plenty of cases where this can work.  It simply becomes unworkable much 
sooner than other data systems (e.g. a relational database or FS-based 
solution).

For instance, think about bloat for a second.  Let's be crazy and say it 
takes 100 bytes to store an integer representing a count.  Let's say you 
write once a second.  That's under 7Mb a day (per counter).  Combined 
with hourly packing, that might be well within limits.

Let's take the next step and say that you can live with a little 
volatility in the data.  You write an object that caches ten seconds 
worth of writes.  Whenever a write comes in at the over-ten-second mark, 
you write the _v_ attribute to the persistent attribute.  There's an 
order of magnitude improvement.

Finally, you store all your counters in a non-versioned storage.  Now 
you have *no* bloat problem. :^)

Regarding performance, maybe his application isn't doing 50 
requests/second and he'd be willing to trade the slight performance hit 
and bloat for a decrease in system complexity.

All of the above has downsides as well.  My point, though, is that we 
shouldn't automatically dismiss the zodb as inappropriate for *all* 
high-write situations.  In fact, with Andreas and Matt Hamilton's 
TextIndexNG, you might even be able to write to catalogued applications 
at a faster rate than one document per minute. :^)

--Paul

Casey Duncan wrote:
> This will kill performance, especially concurrent use of the site. It 
> will also cause large amounts of database bloat. Do you need real time 
> numbers, or is a delay (such as 24 hours) acceptable?
> 
> If you can stand a delay, another approach would be to write a script 
> which scans the z2.log file (or another log that you generate on page 
> hits) each night and in a single transaction updates a counter on each 
> object hit.
> 
> If you use the z2.log, no additional writing is needed to the FS, and 
> you get the benefit of easy access to the counts directly from the 
> objects, without degrading performance or db bloat.
> 
> -Casey
> 
> Ivo van der Wijk wrote:
> 
>> Hi,
>>
>> How bad are per-request transactions in a non-ZEO environment? I.e.
>> each request on a folder or its subobjects will cause a write transaction
>> (somewhat like a non-fs counter, but worse as it happens for all 
>> subobjects)
>>
>> And if this is really bad, are there any workarounds except for writing
>> to the filesystem?
>>
>> Cheers
>>
>>     Ivo
>>
>>
> 
> 
> 
> 
> _______________________________________________
> Zope-Dev maillist  -  Zope-Dev@zope.org
> http://lists.zope.org/mailman/listinfo/zope-dev
> **  No cross posts or HTML encoding!  **
> (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce
> http://lists.zope.org/mailman/listinfo/zope )