Paul Everitt wrote:
I don't agree that high write is always forbidden. I think there are plenty of cases where this can work. It simply becomes unworkable much sooner than other data systems (e.g. a relational database or FS-based solution).
I agree, but I loath to approve of any solution which demands a write for every read of an object.
For instance, think about bloat for a second. Let's be crazy and say it takes 100 bytes to store an integer representing a count. Let's say you write once a second. That's under 7Mb a day (per counter). Combined with hourly packing, that might be well within limits.
Yes, but the counter is not the only thing written. The whole containing object is written out to the storage. Now that doesn't include binary data (such as image and file data), but it does include any primitive data stored in the object's attributes (strings, lists, dicts, etc). Hourly packing seems like a blunderbus solution to the bloat problem. You can't tell me that won't kill performance...
Let's take the next step and say that you can live with a little volatility in the data. You write an object that caches ten seconds worth of writes. Whenever a write comes in at the over-ten-second mark, you write the _v_ attribute to the persistent attribute. There's an order of magnitude improvement.
Only if you run single threaded. For multi-threaded Zope apps (the default), you would need to use a transient object which introduces its own complexities.
Finally, you store all your counters in a non-versioned storage. Now you have *no* bloat problem. :^)
Right, the transient object or something else that writes to disk. Now you have to make sure the counters can be related to the object robustly. Bookkeeping... This is certainly a possibility, I would hesitate to argue for it on the notion it is less complex though.
Regarding performance, maybe his application isn't doing 50 requests/second and he'd be willing to trade the slight performance hit and bloat for a decrease in system complexity.
That could be a good trade, I just wanted to make sure the issues were known.
All of the above has downsides as well. My point, though, is that we shouldn't automatically dismiss the zodb as inappropriate for *all* high-write situations. In fact, with Andreas and Matt Hamilton's TextIndexNG, you might even be able to write to catalogued applications at a faster rate than one document per minute. :^)
Of course not, but the obvious and easiest solution (just incrementing a counter on the objects on every read) is probably not the best solution. -Casey