RE: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely r evis ion based storage for Zope ?)
Kapil, Right now, the svn transactions are entirely contained within a single fileops operation: for example a "mkdir" connects to a transaction root, performs the necessary operations, and commits, all in one shot. Last night I took some more time to try and learn more about Ape's functionning (Where events come from, which interfaces are used for what, and TPC), so I'm starting to understand more ... The more I learn, the more I think closer integration between SVN txn's and Ape's TPC would be a good place to start before looking at adding features like history support and so on: defining a model for what happens in svn for each TPC related call (connect, vote, finish), and then as Shane had said, look at IFSReader/IFSWriter (Which I now call ISubversionReader/ISubversionWriter :P) to match. Right now the fs implementation stores "script commands" that are cummulated upon connect() (I think?), validated as best as possible upon vote() and run upon finish(). I don't see why this couldn't be adapted to SVN txn's ... connect() = start a txn, vote() = validation (what this entails needs to be defined, could involve delta operations, revision number matching, etc ... ?), finish() = commit the svn txn. Because we're within an svn transaction, there would be no need for fs style script command accumulation however, which is nice. J.F. -----Original Message----- From: zope-dev-bounces@zope.org [mailto:zope-dev-bounces@zope.org]On Behalf Of Kapil Thangavelu Sent: April 14, 2004 6:59 AM To: Shane Hathaway Cc: Jean-Francois.Doyon@ccrs.nrcan.gc.ca; zope-dev@zope.org Subject: Re: Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?) On Tue, 2004-04-13 at 12:01, Shane Hathaway wrote:
On Tue, 13 Apr 2004, Kapil Thangavelu wrote:
since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site.
This is only true of FileStorage. Some other storage could implement ZODB versions with merging capability rather than locking.
good point, just because hasn't been doesn't mean it can't ;-) although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog?
i'm also curious how you dealt with svn transactions as part of the ape integration work to date.
The same way it tries to impose transactions on the filesystem: in the vote phase, Ape looks for possible problems and aborts early if it detects
anything that will cause the transaction to fail. Obviously, this provides no guarantee, but covers many cases.
i was more curious how jean-francois was doing the svn ops in fileops, as svn is fundamentally a transactional store (as opposed to the fs), ie is there some record boundary notion of ape signalling the end of serialization for an object set, or was each operation being conducted in a separate svn transaction. -kapil _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Wed, 14 Apr 2004 Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
Right now the fs implementation stores "script commands" that are cummulated upon connect() (I think?), validated as best as possible upon vote() and run upon finish(). I don't see why this couldn't be adapted to SVN txn's ... connect() = start a txn, vote() = validation (what this entails needs to be defined, could involve delta operations, revision number matching, etc ... ?), finish() = commit the svn txn.
The transaction script is necessary for exactly one thing: translation between OIDs and paths. I agonized over this for months, but translating OIDs to paths simply requires complex machinery like this. Also, there is only one reason we have to translate OIDs to paths: opening files by inode number is disallowed (and probably unreliable) in most operating systems. In Subversion, is it possible to read/write a versioned object by unique identifier rather than by path? If so, it's a great blessing and we don't need a transaction script. Shane
On Wed, 2004-04-14 at 19:28, Shane Hathaway wrote:
In Subversion, is it possible to read/write a versioned object by unique identifier rather than by path? If so, it's a great blessing and we don't need a transaction script.
not really, yes every svn node has an a node id, but there is no way to retrieve a particular node from the svn fs via node id. it is possible to compare to compare and see if their related, (part of the node id is a copy id which gets incremented when a node is copied). in short not really. -kapil
On Wed, 2004-04-14 at 10:04, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
Kapil,
Right now, the svn transactions are entirely contained within a single fileops operation: for example a "mkdir" connects to a transaction root, performs the necessary operations, and commits, all in one shot.
ok, thats what about what i expected. not ideal but it works.
Last night I took some more time to try and learn more about Ape's functionning (Where events come from, which interfaces are used for what, and TPC), so I'm starting to understand more ...
The more I learn, the more I think closer integration between SVN txn's and Ape's TPC would be a good place to start before looking at adding features like history support and so on: defining a model for what happens in svn for each TPC related call (connect, vote, finish), and then as Shane had said, look at IFSReader/IFSWriter (Which I now call ISubversionReader/ISubversionWriter :P) to match.
part of the reason i never used ape as a means for svn integration was was that this imo, mix of high level application operations with zodb storage level operations never seemed a proper fit as it required bypassing the storage interface for richer semantic operations. ie i see reverting a revision, tagging a content tree, or diffing mulitiple revisions as application operations. still, there have been the some compelling ideas here about using zodb/ape as an interface.
Right now the fs implementation stores "script commands" that are cummulated upon connect() (I think?), validated as best as possible upon vote() and run upon finish(). I don't see why this couldn't be adapted to SVN txn's ... connect() = start a txn, vote() = validation (what this entails needs to be defined, could involve delta operations, revision number matching, etc ... ?), finish() = commit the svn txn.
not exactly, you just start a txn, do whatever work, and commit in vote. see my previous message outlining txn integration difficulties, and why this is not ideal but currently the best option.
Because we're within an svn transaction, there would be no need for fs style script command accumulation however, which is nice.
sure. should we move this discussion to an ape specific mailing list? -kapil
On Sun, 18 Apr 2004, Kapil Thangavelu wrote:
part of the reason i never used ape as a means for svn integration was was that this imo, mix of high level application operations with zodb storage level operations never seemed a proper fit as it required bypassing the storage interface for richer semantic operations. ie i see reverting a revision, tagging a content tree, or diffing mulitiple revisions as application operations. still, there have been the some compelling ideas here about using zodb/ape as an interface.
FWIW, Ape is also designed to be useful as an import/export mechanism. In theory, if you set up a mapper configuration for a set of classes, you've gained both a database bridge and a clean format for importing and exporting objects. I imagine that would benefit many kinds of applications. In practice, I haven't tried it yet. :-)
should we move this discussion to an ape specific mailing list?
I'm happy with zope-dev and zodb-dev, but if there's a good reason to make a new list, that's fine. Shane
On Mon, 2004-04-19 at 13:33, Shane Hathaway wrote:
On Sun, 18 Apr 2004, Kapil Thangavelu wrote:
should we move this discussion to an ape specific mailing list?
I'm happy with zope-dev and zodb-dev, but if there's a good reason to make a new list, that's fine.
its a knowledge discovery issue, there's a lot of useful info ape info floating around on diseparate mailing lists which makes it hard to get a hold of, having a single mailing focused on ape would make it much easier for folks to get up to speed and see all the cool stuff that can be done and that people are doing with ape ;-) cheers, -kapil
participants (3)
-
Jean-Francois.Doyon＠CCRS.NRCan.gc.ca -
Kapil Thangavelu -
Shane Hathaway