[ZODB-Dev] Idea for DirectoryStorage, GFS DLM, ZODB and ZEO

[Tim Peters]

[Bisaillon, Brian (MBS)]
> I have briefly discussed an idea with Toby Dickenson (the author
> of DirectoryStorage) that perhaps it is possible to use the Global File
> System (GFS) Distributed Lock Manager (DLM) based on OpenDLM to manage
> locks for DirectoryStorage.  The goal would be to basically be able to run
> two or more ZEO servers against the same DirectoryStorage data and
> properly handle situations where two or more ZEO servers are trying to
> modify an object at the same time.
>
> Toby said that looking solely at the storage layer, DirectoryStorage has
> been designed for exactly this type of customization.  He said that the
> BaseFilesystem.py interface is basically where he thinks I would need
> to slot in.  Then he says keep BaseFilesystem.py and Full.py, and
> replace LocalFilesystem.

I doubt anyone here would argue with anything Toby says about
DirectoryStorage <wink>.

> However, he goes on to say that I would need to address more than just
> the storage layer if I wanted to run more than one storage concurrently.
> He says that some parts of concurrency control is implemented in the DB
> layer, not the storage and that the DB keeps a cache of objects, and that
>> cache needs to be invalidated when objects change.

It may be harder than just that:  a Connection object maintains an in-memory
LRU object cache (sometimes called "the pickle cache"), and ZEO clients
maintain a second-level on-disk object cache.  So an object request made by
a ZEO client first looks in the Connection's cache; failing that, it then
looks in ZEO's cache; and failing that it then asks the ZEO server for the
data (and the data it gets back normally ends up in both the ZEO client
cache and the Connection's cache then).  Note that guaranteeing cache
consistency is typically the hardest part of any distributed system with
caches.

> Then he says that ZEO coordinates the distribution of those invalidations

The ZEO server sends invalidation messages to its clients, yes.  I suppose
that if that part gets handled "correctly" (whatever that means, exactly, in
this case), the Connection caches will take care of themselves.

> and that I might get some help from this mailing list.  This is why I am
> here now.  Any thoughts?

It's interesting, but I doubt anyone who understands ZEO well enough has
enough spare time to help -- in which case you'll have to stare at the code.
I'd recommend looking at the ZODB 3.3 code instead of code in the 3.2 line,
because 3.2 is in bugfix-only mode and 3.3's code is cleaner and better
commented in some respects.  I don't know whether DirectoryStorage is
compatible with 3.3, though (the storage API grew in 3.3, to accommodate the
addition of Multiversion Concurrency Control).  

> Please CC my home e-mail address at HYPERLINK
> "mailto:brian_bisaillon at rogers.com"brian_bisaillon at rogers.com when
> replying.  Thanks a lot!