[Zope] non-undoable storage

[Paul Winkler]

On Mon, Apr 18, 2005 at 02:38:01PM +0200, Milos Prudek wrote:
> I have a couple hundred Articles (class instances) in my ZODB. Most 
> instances are 200 kB, some are much larger. One of the Article 
> properties is Number of times the article was read. Any increment causes 
> ZODB to grow by the size of the given instance, due to the fact that 
> ZODB will append a new version of the whole 200 kB or 2MB instance just 
> because a single byte was changed.
> 
> Consequently ZODB grows several hundred MB in a single day even if no 
> new article is added.
> 
> This calls for some form of non-undoable storage.
> 
> One possibility is to move the "NumberOfReaders" attribute from ZODB to 
> my SQL server. Not a problem, but I'm not sure what is the best way to 
> maintain connection between ZODB instance (it can appear at different 
> places in ZODB directory structure and it can be moved from place to 
> place with Cut and Paste in ZMI) and SQL table. Connection means SQL key 
> column. Instance ids cannot be used as a SQL key, because they are 
> nonunique - each folder has a sequence of instances numbered 1...x. So 
> there are over ten instances named "1". Instance addresses = URLs could 
> be used but I would have to write my own methods / interfaces for moving 
> Articles in the directory structure to maintain the connection between 
> ZODB and SQL table.
> 
> Is there an obvious elegant solution that I am missing?

You might consider replacing the NumberOfReaders attribute with a
first-class persistent object.  e.g. you could use a PersistentList or
IIBTree or some such.  This would prevent Zope from saving the entire
Article object when that one attribute changes. You'd still get some
bloat from all the historical revisions of the attribute, but it would
be MUCH less.  

e.g. something like:

from ZODB import PersistentList

class Article(...):

    def __init__(self):
        ...
        self._numberOfReaders = PersistentList([0])

    def getNumberOfReaders(self):
        return self._numberOfReaders[0]

    def incrementNumberOfReaders(self):
        # do NOT set self._p_changed = 1
        self._numberOfReaders[0] = += 1 


You could then have NumberOfProperties become a ComputedAttribute
to allow client code to keep accessing it as an attribute.
(this is a read-only "property" in recent versions of Python,
not to be confused with Zope's "properties"; but Zope doesn't
support pythonic properties yet.)

e.g.:

    NumberOfReaders= ComputedAttribute(getNumberOfReaders)


But AFAIK, ComputedAttributes don't support write methods.
So client code can't write "someArticle.NumberOfReaders += 1".
But then, you wouldn't be able to do that with a SQL-based
solution either.
 
-- 

Paul Winkler
http://www.slinkp.com