Zope + Ape + Subversion (was: RE: [Zope-dev] Using a truely revis ion based storage for Zope ?)
G'Day, Well, step one is done ... I now have Zope + Ape using Subversion as it's "filesystem" !! This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem. Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support: - Use zope's username for SVN logging. - History/Undo support: View past revisions of an object, and revert to such a past revision. - Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?) In the longer term, there's great opportunity for: - "Built-in" conflict management and resolution: No more need for a "SafetyBelt" type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error. - Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort. - Leveraging SVN's property management. Content in SVN has properties, much like Zope does. I haven't looked at it yet, but I've noticed ".properties" file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too ! In the realm of the wishful thinking, there's even more: Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually get to is publsihing something of a given revision, while editing another. One potential paradigm for distiguishing between the two modes of operation could be to use anonymous vs. authenticated. This is not useful to everyone, but can be in certain circumstances, most notably where authenticated = authors/developpers and anonymous = normal users. This however requires ZMI interfaces, and in my case CMF ones as well ... This would be global though ... Eventually it'd be nice to have per object control of this stuff. Andy McKay says it can't be done, anybody care to contradict him ? :P I image I'd have to monkey pathc something DEEP in the Zope code base, but I find the mix-in class that's the commonn denominator ... why not ? Right now I've been working within the "fs" module of apelib. I'm going to split it off into a seperate one so that it's a clean merge with Ape's code base, in case someone wants to try it, or it eventually makes into Ape's default distribution ?? ;) Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome ! Thanks, J.F. -----Original Message----- From: zope-dev-bounces@zope.org [mailto:zope-dev-bounces@zope.org]On Behalf Of Shane Hathaway Sent: April 8, 2004 11:20 AM To: Jean-Francois.Doyon@ccrs.nrcan.gc.ca Cc: zope-dev@zope.org Subject: Re: [Zope-dev] Using a truely revision based storage for Zope ? Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
I've started looking at the ZODB and APE packages to try and get some understanding of how the whole storage interaction works, but it'll take me some time to figure it all out ... So I thought I'd get feedback on the idea first ...
Sounds great! If I were you, I would start by replacing Ape/lib/apelib/fs/fileops.py with something that uses the Subversion module. It's a very simple module and you should be able to get pretty far that way. Shane _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Mon, 12 Apr 2004 Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
Well, step one is done ... I now have Zope + Ape using Subversion as it's "filesystem" !!
That's fantastic! I'll write a more detailed reply soon. :-) Shane
On Mon, 2004-04-12 at 18:03, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
G'Day,
Well, step one is done ... I now have Zope + Ape using Subversion as it's "filesystem" !!
cool!
This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem.
Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support:
- Use zope's username for SVN logging.
using AccessControl.getSecurityManager().getUser().getId() and setting it up as a revision prop ( or directly when creating the repo transaction) should do it.
- History/Undo support: View past revisions of an object, and revert to such a past revision.
perhaps the code for this would be helpful, (remove image for code link) http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevision...
- Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?)
zope version support isn't really all that worthwhile, esp in a cmf context. in general zope's version support (or zodb more particularly) is a database level feature masquerading as an application one. since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. otoh, integration with zope's transaction manager would be a good thing, although its problematic to integrate between svn and zope txn models, more on that in a moment.
In the longer term, there's great opportunity for:
- "Built-in" conflict management and resolution: No more need for a "SafetyBelt" type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error.
i don't know that conflict management is really useful in this context. svn like zope relies on optimistic concurrency control, and currently doesn't support dav locking (which zope does). ie, it will just throw an error if the content has been changed since the transaction began. the 'normal' concurrency control of svn is better but dependent on using the working copy (wc) layer, which is additional programming and storage overhead. so at the layer of the svn client this is already done and working and good, but integrating this functionality into zope is a bit harder without wc overhard. this also makes the transaction integration becomes harder because both zope and svn are using what amounts to optimistic concurrency control which makes it impossible afaics, to get real txn integration, ie in zope's two phase commit strategy, the last chance for a participant to safely abort is tpc_vote, but there is no real way of knowing if the svn txn will suceed or not until its tried. if it is tried at this stage and succeeds then there is the possibility of a latter txn participant failing the tpc_vote and the txn being aborted, and if waits till tpc_finish (last part of two phase commit) and the svn txn fails it can hose the composite txn integrity. once svn supports dav locks, doing txn integration via resource locking as part of tpc_vote (or earlier) would be possible, till then.. i dunno, i can't see a way around this for real txn integration. i'm also curious how you dealt with svn transactions as part of the ape integration work to date.
- Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort.
to me this is one of the fundamental benefits of using svn, giving users the ability to use interfaces like tortoisesvn (win shell extensions) or mac finder extensions to interact directly with content.
- Leveraging SVN's property management. Content in SVN has properties, much like Zope does. I haven't looked at it yet, but I've noticed ".properties" file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too !
definitely!
In the realm of the wishful thinking, there's even more:
Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually get to is publsihing something of a given revision, while editing another. One potential paradigm for distiguishing between the two modes of operation could be to use anonymous vs. authenticated. This is not useful to everyone, but can be in certain circumstances, most notably where authenticated = authors/developpers and anonymous = normal users. This however requires ZMI interfaces, and in my case CMF ones as well ... This would be global though ... Eventually it'd be nice to have per object control of this stuff. Andy McKay says it can't be done, anybody care to contradict him ? :P I image I'd have to monkey pathc something DEEP in the Zope code base, but I find the mix-in class that's the commonn denominator ... why not ?
to me this seems a separate point then svn usage at all, its really about content view paradigms for revision workflow. the two predominant ones are using staging to deploy different sites representing different parts of the publication cycle/wf, and the other which afaik, is only widely used by silva is view multiplexing where the view of a content is responsible for retrieving the proper version to render a display for, although afaicr silva support for such is initimately tied to its workflow and versioning paradigm.
Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome !
i've done some thinking and work with integration of svn as content in zope as well, though using a different software stack than ape. some notes/ideas culled from a dev diary i kept while working on it that might be useful. - using svn hook scripts for event notification back into zope (content reindex), although it should be careful about modifying the content otherwise some client which just committed might be put out of date. - transform layers (using portal transforms) with a private content type registry... although integrating at the ape layer this might not be as useful. cheers, -kapil
Thanks, J.F.
-----Original Message----- From: zope-dev-bounces@zope.org [mailto:zope-dev-bounces@zope.org]On Behalf Of Shane Hathaway Sent: April 8, 2004 11:20 AM To: Jean-Francois.Doyon@ccrs.nrcan.gc.ca Cc: zope-dev@zope.org Subject: Re: [Zope-dev] Using a truely revision based storage for Zope ?
Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
I've started looking at the ZODB and APE packages to try and get some understanding of how the whole storage interaction works, but it'll take me some time to figure it all out ... So I thought I'd get feedback on the idea first ...
Sounds great! If I were you, I would start by replacing Ape/lib/apelib/fs/fileops.py with something that uses the Subversion module. It's a very simple module and you should be able to get pretty far that way.
Shane
On Tue, 13 Apr 2004, Kapil Thangavelu wrote:
since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site.
This is only true of FileStorage. Some other storage could implement ZODB versions with merging capability rather than locking.
i'm also curious how you dealt with svn transactions as part of the ape integration work to date.
The same way it tries to impose transactions on the filesystem: in the vote phase, Ape looks for possible problems and aborts early if it detects anything that will cause the transaction to fail. Obviously, this provides no guarantee, but covers many cases. Shane
On Tue, 2004-04-13 at 12:01, Shane Hathaway wrote:
On Tue, 13 Apr 2004, Kapil Thangavelu wrote:
since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site.
This is only true of FileStorage. Some other storage could implement ZODB versions with merging capability rather than locking.
good point, just because hasn't been doesn't mean it can't ;-) although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog?
i'm also curious how you dealt with svn transactions as part of the ape integration work to date.
The same way it tries to impose transactions on the filesystem: in the vote phase, Ape looks for possible problems and aborts early if it detects anything that will cause the transaction to fail. Obviously, this provides no guarantee, but covers many cases.
i was more curious how jean-francois was doing the svn ops in fileops, as svn is fundamentally a transactional store (as opposed to the fs), ie is there some record boundary notion of ape signalling the end of serialization for an object set, or was each operation being conducted in a separate svn transaction. -kapil
On Wed, 14 Apr 2004 06:59:05 -0400 Kapil Thangavelu <hazmat@objectrealms.net> wrote:
On Tue, 2004-04-13 at 12:01, Shane Hathaway wrote:
On Tue, 13 Apr 2004, Kapil Thangavelu wrote:
since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site.
This is only true of FileStorage. Some other storage could implement ZODB versions with merging capability rather than locking.
good point, just because hasn't been doesn't mean it can't ;-)
although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog?
Catalogs are actually fairly easy to handle, at least in theory. The state of the catalog directly depends (and is derived from) the current state of the cataloged objects. So merging the catalog should mean: merge the content first and then reindex it.
From this pov, it would be possible to get cataloging right in the current version machinery as well. This machinery would need to allow objects to override the locking behavior in some way. In the case of the catalog it would be fine for a current catalog and the one in the version to ignore each other (i.e., changing the one in the version does not lock the current one). Then when the version is saved/merged, the affected objects are recataloged.
In practical terms this would mean that the versioned catalog would need to keep track of the uids that had been cataloged, uncataloged and reindexed in the version. The "merge" would mean cataloging, uncataloging and reindexing those same uids in the current catalog. -Casey
On Wed, 14 Apr 2004 09:50:17 -0400 Casey Duncan <casey@zope.com> wrote: [..]
In practical terms this would mean that the versioned catalog would need to keep track of the uids that had been cataloged, uncataloged and reindexed in the version. The "merge" would mean cataloging, uncataloging and reindexing those same uids in the current catalog.
A complicating factor to this though is that it would need to be done *after* all of the other content merges. That implies some sort of dependancy graph, *bleah*. I've never been terribly fond of the idea of catalogs as top-level instance space objects (i.e., peers to the content they catalog), and things like this are one of the reasons why. In any case it might be good enough for an object to be able to say "merge me last". Other toolsy and servicey sorts of objects that depend on content (like a workflow state repository) might want to do so as well. As long as the state of the tools doesn't depend on each other, then it's ok. That last sentence is making me a little ill, however ;^) -Casey
On Wed, 14 Apr 2004, Kapil Thangavelu wrote:
although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog?
By my understanding, outside the storage, different versions are completely independent. You could even simulate multiple databases using versions. It is the storage that knows how to create new versions, merge versions, and prevent objects being used in a version from being changed. Mounted databases try to use the same version as the root connection. To merge a catalog, you could bring one of the stored catalogs to life within the storage and replay the sequence of transactions that occurred in the other catalog. This could be tricky, and opening a catalog within a ZEO server would make the ZEO server less stable. I think there might be other ways to look at it. One way to simplify would be to put catalogs in their own storage, separated from the content they index. That way, you could (ab)use the transaction machinery to abort or commit indexes as you see fit within the catalog conflict resolver. I've experimented with this before, and it seemed promising. Shane
On Thu, 15 Apr 2004 20:40:09 -0400 (EDT) Shane Hathaway <shane@zope.com> wrote:
On Wed, 14 Apr 2004, Kapil Thangavelu wrote:
although i wonder if there is some hand waving in progress here that i can't see. i guess my semantic notion of versions has been that of long lived transactions, and is there a better means of thinking of them? how do they play across with multiple mounted zodbs? what would something like merge mean in the context of a catalog?
By my understanding, outside the storage, different versions are completely independent. You could even simulate multiple databases using versions. It is the storage that knows how to create new versions, merge versions, and prevent objects being used in a version from being changed.
Mounted databases try to use the same version as the root connection.
To merge a catalog, you could bring one of the stored catalogs to life within the storage and replay the sequence of transactions that occurred in the other catalog. This could be tricky, and opening a catalog within a ZEO server would make the ZEO server less stable.
Ok, running the catalog on the storage server. Now that's crazy talk ;^) running-awayly y'rs, -Casey
Hi all, I've read your discussion about version control, it seems a cool thing you guys making good progress. Btw, can I ask is the Ape using Subversion in Zope stable? how able CMF stuff? I wanna make/find a document management system which can provide some kinda version control in Plone. Thanks Arthur On Tue, 13 Apr 2004 04:06:26 -0400, Kapil Thangavelu wrote
On Mon, 2004-04-12 at 18:03, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
G'Day,
Well, step one is done ... I now have Zope + Ape using Subversion as it's "filesystem" !!
cool!
This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem.
Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support:
- Use zope's username for SVN logging.
using AccessControl.getSecurityManager().getUser().getId() and setting it up as a revision prop ( or directly when creating the repo transaction) should do it.
- History/Undo support: View past revisions of an object, and revert to such a past revision.
perhaps the code for this would be helpful, (remove image for code link) http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevision.... png
- Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?)
zope version support isn't really all that worthwhile, esp in a cmf context. in general zope's version support (or zodb more particularly) is a database level feature masquerading as an application one. since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. otoh, integration with zope's transaction manager would be a good thing, although its problematic to integrate between svn and zope txn models, more on that in a moment.
In the longer term, there's great opportunity for:
- "Built-in" conflict management and resolution: No more need for a "SafetyBelt" type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error.
i don't know that conflict management is really useful in this context. svn like zope relies on optimistic concurrency control, and currently doesn't support dav locking (which zope does). ie, it will just throw an error if the content has been changed since the transaction began. the 'normal' concurrency control of svn is better but dependent on using the working copy (wc) layer, which is additional programming and storage overhead. so at the layer of the svn client this is already done and working and good, but integrating this functionality into zope is a bit harder without wc overhard.
this also makes the transaction integration becomes harder because both zope and svn are using what amounts to optimistic concurrency control which makes it impossible afaics, to get real txn integration, ie in zope's two phase commit strategy, the last chance for a participant to safely abort is tpc_vote, but there is no real way of knowing if the svn txn will suceed or not until its tried. if it is tried at this stage and succeeds then there is the possibility of a latter txn participant failing the tpc_vote and the txn being aborted, and if waits till tpc_finish (last part of two phase commit) and the svn txn fails it can hose the composite txn integrity.
once svn supports dav locks, doing txn integration via resource locking as part of tpc_vote (or earlier) would be possible, till then.. i dunno, i can't see a way around this for real txn integration.
i'm also curious how you dealt with svn transactions as part of the ape integration work to date.
- Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort.
to me this is one of the fundamental benefits of using svn, giving users the ability to use interfaces like tortoisesvn (win shell extensions) or mac finder extensions to interact directly with content.
- Leveraging SVN's property management. Content in SVN has properties,
much
like Zope does. I haven't looked at it yet, but I've noticed ".properties" file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too !
definitely!
In the realm of the wishful thinking, there's even more:
Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually get to is publsihing something of a given revision, while editing another. One potential paradigm for distiguishing between the two modes of operation could be to use anonymous vs. authenticated. This is not useful to everyone, but can
be
in certain circumstances, most notably where authenticated = authors/developpers and anonymous = normal users. This however requires ZMI interfaces, and in my case CMF ones as well ... This would be global though ... Eventually it'd be nice to have per object control of this stuff. Andy McKay says it can't be done, anybody care to contradict him ? :P I image I'd have to monkey pathc something DEEP in the Zope code base, but I find the mix-in class that's the commonn denominator ... why not ?
to me this seems a separate point then svn usage at all, its really about content view paradigms for revision workflow. the two predominant ones are using staging to deploy different sites representing different parts of the publication cycle/wf, and the other which afaik, is only widely used by silva is view multiplexing where the view of a content is responsible for retrieving the proper version to render a display for, although afaicr silva support for such is initimately tied to its workflow and versioning paradigm.
Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome !
i've done some thinking and work with integration of svn as content in zope as well, though using a different software stack than ape. some notes/ideas culled from a dev diary i kept while working on it that might be useful.
- using svn hook scripts for event notification back into zope (content reindex), although it should be careful about modifying the content otherwise some client which just committed might be put out of date.
- transform layers (using portal transforms) with a private content type registry... although integrating at the ape layer this might not be as useful.
cheers,
-kapil
Thanks, J.F.
-----Original Message----- From: zope-dev-bounces@zope.org [mailto:zope-dev-bounces@zope.org]On Behalf Of Shane Hathaway Sent: April 8, 2004 11:20 AM To: Jean-Francois.Doyon@ccrs.nrcan.gc.ca Cc: zope-dev@zope.org Subject: Re: [Zope-dev] Using a truely revision based storage for Zope ?
Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
I've started looking at the ZODB and APE packages to try and get some understanding of how the whole storage interaction works, but it'll take me some time to figure it all out ... So I thought I'd get feedback on the idea first ...
Sounds great! If I were you, I would start by replacing Ape/lib/apelib/fs/fileops.py with something that uses the Subversion module. It's a very simple module and you should be able to get pretty far that way.
Shane
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Open WebMail Project (http://openwebmail.org)
granted the svn integration has lots of coolness factors including richer client integration, but things like zope version control (zope.org cvs) are already production quality and not so hard to integrate in for basic version control. -kapil On Tue, 2004-04-13 at 14:13, Arthur Chan Chi Chuen wrote:
Hi all,
I've read your discussion about version control, it seems a cool thing you guys making good progress. Btw, can I ask is the Ape using Subversion in Zope stable? how able CMF stuff? I wanna make/find a document management system which can provide some kinda version control in Plone.
Thanks Arthur On Tue, 13 Apr 2004 04:06:26 -0400, Kapil Thangavelu wrote
On Mon, 2004-04-12 at 18:03, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
G'Day,
Well, step one is done ... I now have Zope + Ape using Subversion as it's "filesystem" !!
cool!
This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem.
Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support:
- Use zope's username for SVN logging.
using AccessControl.getSecurityManager().getUser().getId() and setting it up as a revision prop ( or directly when creating the repo transaction) should do it.
- History/Undo support: View past revisions of an object, and revert to such a past revision.
perhaps the code for this would be helpful, (remove image for code link) http://zope.org/Members/k_vertigo/Products/CMFSubversionBrowser/FileRevision.... png
- Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?)
zope version support isn't really all that worthwhile, esp in a cmf context. in general zope's version support (or zodb more particularly) is a database level feature masquerading as an application one. since objects modified in a version are in essence locked from participating in other transactions, actions like modifying content in a version in a cmf site amounts to locking the catalog from changes outside of the version, which amounts to shutting down write activities to a cmf site. otoh, integration with zope's transaction manager would be a good thing, although its problematic to integrate between svn and zope txn models, more on that in a moment.
In the longer term, there's great opportunity for:
- "Built-in" conflict management and resolution: No more need for a "SafetyBelt" type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error.
i don't know that conflict management is really useful in this context. svn like zope relies on optimistic concurrency control, and currently doesn't support dav locking (which zope does). ie, it will just throw an error if the content has been changed since the transaction began. the 'normal' concurrency control of svn is better but dependent on using the working copy (wc) layer, which is additional programming and storage overhead. so at the layer of the svn client this is already done and working and good, but integrating this functionality into zope is a bit harder without wc overhard.
this also makes the transaction integration becomes harder because both zope and svn are using what amounts to optimistic concurrency control which makes it impossible afaics, to get real txn integration, ie in zope's two phase commit strategy, the last chance for a participant to safely abort is tpc_vote, but there is no real way of knowing if the svn txn will suceed or not until its tried. if it is tried at this stage and succeeds then there is the possibility of a latter txn participant failing the tpc_vote and the txn being aborted, and if waits till tpc_finish (last part of two phase commit) and the svn txn fails it can hose the composite txn integrity.
once svn supports dav locks, doing txn integration via resource locking as part of tpc_vote (or earlier) would be possible, till then.. i dunno, i can't see a way around this for real txn integration.
i'm also curious how you dealt with svn transactions as part of the ape integration work to date.
- Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort.
to me this is one of the fundamental benefits of using svn, giving users the ability to use interfaces like tortoisesvn (win shell extensions) or mac finder extensions to interact directly with content.
- Leveraging SVN's property management. Content in SVN has properties,
much
like Zope does. I haven't looked at it yet, but I've noticed ".properties" file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too !
definitely!
In the realm of the wishful thinking, there's even more:
Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually get to is publsihing something of a given revision, while editing another. One potential paradigm for distiguishing between the two modes of operation could be to use anonymous vs. authenticated. This is not useful to everyone, but can
be
in certain circumstances, most notably where authenticated = authors/developpers and anonymous = normal users. This however requires ZMI interfaces, and in my case CMF ones as well ... This would be global though ... Eventually it'd be nice to have per object control of this stuff. Andy McKay says it can't be done, anybody care to contradict him ? :P I image I'd have to monkey pathc something DEEP in the Zope code base, but I find the mix-in class that's the commonn denominator ... why not ?
to me this seems a separate point then svn usage at all, its really about content view paradigms for revision workflow. the two predominant ones are using staging to deploy different sites representing different parts of the publication cycle/wf, and the other which afaik, is only widely used by silva is view multiplexing where the view of a content is responsible for retrieving the proper version to render a display for, although afaicr silva support for such is initimately tied to its workflow and versioning paradigm.
Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome !
i've done some thinking and work with integration of svn as content in zope as well, though using a different software stack than ape. some notes/ideas culled from a dev diary i kept while working on it that might be useful.
- using svn hook scripts for event notification back into zope (content reindex), although it should be careful about modifying the content otherwise some client which just committed might be put out of date.
- transform layers (using portal transforms) with a private content type registry... although integrating at the ape layer this might not be as useful.
cheers,
-kapil
Thanks, J.F.
-----Original Message----- From: zope-dev-bounces@zope.org [mailto:zope-dev-bounces@zope.org]On Behalf Of Shane Hathaway Sent: April 8, 2004 11:20 AM To: Jean-Francois.Doyon@ccrs.nrcan.gc.ca Cc: zope-dev@zope.org Subject: Re: [Zope-dev] Using a truely revision based storage for Zope ?
Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
I've started looking at the ZODB and APE packages to try and get some understanding of how the whole storage interaction works, but it'll take me some time to figure it all out ... So I thought I'd get feedback on the idea first ...
Sounds great! If I were you, I would start by replacing Ape/lib/apelib/fs/fileops.py with something that uses the Subversion module. It's a very simple module and you should be able to get pretty far that way.
Shane
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Open WebMail Project (http://openwebmail.org)
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On 12 Apr 2004 at 18:03, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
Anyways, I'm just rambling by now ... Comments, thoughts and constructive criticism welcome !
This sounds wonderful! Regarding versions. Perhap you can have a control panel setting that selects the branch to publish "by default". The ZMI could offer another mechanism that uses cookies to select the desired "branch or version" That is, a branch name or "PREV, HEAD" etc. But, I use jEdit and FTP for most of my Zope work, so would really like to have a way to access version/branches via FTP. Maybe that can be done by having pseudo subdirectories named after revisions or branches. So /myFolder/.Prev/index_html would do what you expect (note you can't save to old versions anyway ..) -- Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements
On Mon, 12 Apr 2004 Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
This is step one because, as Shawn suggested (Thanks for the pointer, that's what I needed!), this simply means that Zope uses SVN purely as a filesystem.
Because of subversion's nature, I want to look at 2 things beyond this that traditional filesystems don't support:
To take this to the next level, you need to implement IFSReader and IFSWriter instead of the more basic file operations interface. See Ape/lib/apelib/fs/interfaces.py. In IFSReader/IFSWriter, you're a little closer to objects, and you work with OIDs instead of paths. I expect that history and version support will require us to augment IFSReader or add another interface.
- Use zope's username for SVN logging. - History/Undo support: View past revisions of an object, and revert to such a past revision. - Zope Version support: SVN is fully transactional and atomic, this should allow for support of Zope versions (I think ?)
These seem doable.
In the longer term, there's great opportunity for:
- "Built-in" conflict management and resolution: No more need for a "SafetyBelt" type approach. Right now I haven't looked at this at all. I plan to implement smart merging where possible (It might work already actually, I just need to test it). True conflicts (Where a merge can't be accomplished withouth user interaction) would raise some sort of conflict error.
This could be complicated, because after the merge you have to be sure Zope caches the merged state rather than either of the intermediate states. The idea has potential, though!
- Editing Zope content objects through interaction with the svn repository. I can checkout the repository, edit some objects, and chek them back in, never interacting with Zope directly ... I've already tried this ! Works great for text based content types such as PageTemplates or DTML Documents and so on ... I even did it with a JPG, though because the properties hold width and height, you get some weird looking pictures :) The concept is valid though. There may someday be a way to leverage this functionality better with a purpose built client of some sort.
Sounds neat. Fortunately, Ape doesn't have to be aware of this. :-)
- Leveraging SVN's property management. Content in SVN has properties, much like Zope does. I haven't looked at it yet, but I've noticed ".properties" file appearing ... I'm guessing those are the Zope properties, which would be better handled by subversion's property mechanism. And properties are versioned too !
I think you'll find this easy to do once you implement IFSReader/IFSWriter.
In the realm of the wishful thinking, there's even more:
Right now, HEAD (Latest/youngest revision) is always used and worked with. The really powerful feature I want to eventually get to is publsihing something of a given revision, while editing another. One potential paradigm for distiguishing between the two modes of operation could be to use anonymous vs. authenticated. This is not useful to everyone, but can be in certain circumstances, most notably where authenticated = authors/developpers and anonymous = normal users. This however requires ZMI interfaces, and in my case CMF ones as well ... This would be global though ... Eventually it'd be nice to have per object control of this stuff. Andy McKay says it can't be done, anybody care to contradict him ? :P I image I'd have to monkey pathc something DEEP in the Zope code base, but I find the mix-in class that's the commonn denominator ... why not ?
Well, CMFStaging is designed to address this need. To do this at the database level, you could use versions.
Right now I've been working within the "fs" module of apelib. I'm going to split it off into a seperate one so that it's a clean merge with Ape's code base, in case someone wants to try it, or it eventually makes into Ape's default distribution ?? ;)
You can add a new module, perhaps 'svnops.py', to the 'fs' package. Then we should make the choice between fileops and svnops configurable. Shane
participants (6)
-
Arthur Chan Chi Chuen -
Brad Clements -
Casey Duncan -
Jean-Francois.Doyon＠CCRS.NRCan.gc.ca -
Kapil Thangavelu -
Shane Hathaway