[Zope-dev] Re: AdaptableStorage

[Shane Hathaway]

seb bacon wrote:
> Shane Hathaway wrote:
> 
>> Chris Withers wrote:
>>
>>> Shane Hathaway wrote:
>>>
>>>> performance.  There needs to be a way for applications that modify 
>>>> the database to tell Zope about the modification, so Zope can reset 
>>>> its caches.
>>>
>>>
>>>
>>> But, IIRC, the last time this was discussed on a mailing list you had 
>>> some cool ideas to sovle the problem, right?
>>
>>
>>
>> Yes, but I want to hear other people's ideas first.  What do you think?
> 
> 
> Isn't this a different problem for each kind of storage, e.g. MD5 hash 
> for ext2, transaction ID for foo...?  Or are you referring to a 
> different aspect of the problem?

I'm thinking about "real-time" updates.  When the underlying data 
changes, you'd like Zope to see the change immediately.  If indefinite 
delays are OK, then AdaptableStorage already does enough: it raises a 
ConflictError if you try to write changes based on old data.

The idea I like the most for relational databases is to ask the RDBMS 
what the ID of the last transaction was.  If Zope missed a transaction, 
it should flush all caches.  This will work if the database is 
infrequently changed by external applications, or if Zope is accessed 
infrequently.

If external applications make a lot of changes, however, and Zope needs 
good performance at the same time, then both Zope and the external 
applications need to update a per-object transaction ID.  Then, at the 
beginning of transactions, Zope would invalidate only the recently 
updated objects.  Hmm, perhaps smarter RDBMSs could make it easy to keep 
transaction IDs updated using triggers.  (This solution could also 
replace both ZEO and ZRS, BTW. ;-) )

On the filesystem, the problem seems much more difficult, since there 
are no transactions.  You'd like the kernel to send Zope a message 
anytime someone modifies a file in a certain hierarchy, but that would 
require kernel hacking.

For that case, I'm thinking that requiring external apps to touch a 
special file somewhere might be the right thing.  At the beginning of 
each transaction, if Zope sees a change to the file, it flushes its cache.

> While reading the referenced thread on the subject, I found your 
> description of the product design here:
> 
> http://lists.zope.org/pipermail/zope-dev/2002-August/016981.html
> 
> Could this go in the docs/ directory of the product?  The design, while 
> very clean, doesn't lend itself to immediate understanding on a cursory 
> view of the source...

I'm hoping to present a complete tutorial on AdaptableStorage at PyCon 
DC 2003.  I'll integrate those notes.  Thanks for pointing them out--I'd 
forgotten about them.  The names are changed somewhat, but the basic 
design is the same.

Shane