[ZODB-Dev] Idea for DirectoryStorage, GFS DLM, ZODB and ZEO

[Bisaillon, Brian (MBS)]

Hello,

I have briefly discussed an idea with Toby Dickenson (the author of
DirectoryStorage) that perhaps it is possible to be able to run two or more
ZEO servers against the same DirectoryStorage data at the same time and
handle situations where two or more ZEO servers are trying to modify objects
either at the same time or not.

Toby said that I would need to address more than just the Storage layer of
ZODB if I wanted to run more than one storage concurrently. He said that
some parts of concurrency control are implemented in the DB layer of ZODB,
not the Storage layer, and that the DB keeps a cache of objects that needs
to be invalidated when objects change. He also said that ZEO coordinates the
distribution of those invalidations and that I might get some help from this
mailing list so this is why I am here now.

Originally, I had thought that this problem was caused by filesystem locking
issues and I was wrong in that original assumption. I commented out the
following code under 'def engage(self,synchronous=0):' in Localfilesystem.py
that is a part of DirectoryStorage:

        #try:
        #    self._lock()
        #except:
        #    raise DirectoryStorageError('Storage is locked by another
process')

I configured the first ZEO server to use IDs of 1 and the second ZEO server
to use IDs of 2 for storage related configuration so that they would not
conflict when I tried to start them both at the same time and against the
same DirectoryStorage data. They both started successfully after I had
commented out the above source code. Then I configured one Zope instance per
ZEO server. They both started successfully as well.

I then tried to add a new Plone Site to through the first Zope instance and
that worked. Then I tried to access it on the second Zope instance and it
didn't work. The second ZEO server had failed with a POSKeyError. However,
it seemed as though the second ZEO server just didn't have any way of
knowing that I had added an object through the first ZEO server because it
looked like it was trying to compare an old copy of something (possibly the
client cache?) to the current DirectoryStorage data. Therefore, I simply
restarted the second ZEO server and its associated Zope instance, browsed to
the newly created object, and this time it worked!

Therefore, I have a feeling this goes back to what Toby was saying about how
ZEO coordinates object invalidation and maybe it even has something to do
with the ZEO client cache. I have not yet tried to disable the client cache
on the second ZEO server to see if that helps and I have not tried to use
read-only on the second ZEO server yet. It would be nice if I could create
new objects on both Zope instances and have the ZEOs continue to run against
the same DirectoryStorage data. Then you could have that data sitting on a
clustered filesystem like GFS (which is what I have setup at home for
testing). After that of course, I'd have to test what happens if one tries
to create new objects on both Zope instances at the same time.

Another thing I was thinking was that perhaps it would be possible to
somehow take advantage of OpenDLM that is a part of GFS for locking at the
filesystem level.

Anyway, does anyone have any thoughts on what I've discussed here? Please CC
brian_bisaillon at rogers.com when replying.

Thanks a lot!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20050314/82a92e09/attachment.htm