Chris Withers wrote:
Shane Hathaway wrote:
<snip>
Wow! That was very very cool. I look forward to helping make this all a reality in Zope 3 ;-)
I'm sure I left a few things out, so ask questions about the unclear parts. It's probably more info than you were expecting. ;-)
Hmm, okay, the only problem I can think that still needs solving is this: Say you're serializing to a relational database. All well and good.
...until another app comes along and modifies your relational table (pretty common usecase, othwerwise people wouldn't care as much about the storage layer...)
How does the serializer/deserializer find out something has changed and propogate this back up, invalidating any cached object, etc?
I've been trying out a limited-duration cache strategy. The simplest approach is to clear the cache between transactions. Alternatively, you can clear the cache periodically. For a lot of applications this is adequate. Another approach, if you can afford it, is triggers. To do this, you have to know a lot about your database, since there is no standard way. But as I learned from AdaptableStorage, no matter what you do, you need to have conflict detection. Otherwise Zope will merrily delete a directory you just added externally, for example. This was a hard problem to solve until I stumbled on the idea of putting half of an MD5 sum in _p_serial instead of the date. (Only half because the current C code only allows 8 bytes in _p_serial. ;-) ) This works well enough to detect nearly all conflicts, even though it might not be the speediest solution. And as it turned out, as long as I had conflict detection, it didn't matter as much that the database didn't have the most recent state all the time. Zope detected conflict errors and retried the transactions, and the second time always worked. It was good. :-) Finally, here's a theoretical solution that I think would be ideal for a lot of people: if we could just ask the RDBMS for its current transaction ID, Zope could keep track of the last transaction ID it knew about, and clear the caches when another process made a change. This solution may yield the highest performance for Zope-centric applications. It would also remove the need for my MD5 hack. ;-) Shane