Re: [Zope-dev] FTP interface being worked on?
"Chris McDonough" <chrism@digicool.com> writes:
I don't think it's reasonable or wise to impose any "master structure" for filesystem serialization of bodies of objects. Each instance (or perhaps each class) should define how best to serialize itself to disk. Representations between classes are likely to be radically different. A place for standardization is in the "properties" file(s) which accompany each object rep... this is likely to be XML or another structured variant.
Is there a motivation for using serialization to provide an editable "god's eye view" of all or part of a Zope site? That is, provide a two-way XML serialization at any stage of the managed hierarchy of a site? That way, XML tools could be used as stream editors at any level, to the extent that the serialization is understandable. So, for example, we'd have ways to not just alter a template or content that gets templatized, but the containers that organize them, and related metadata or content that isn't as near. This links well with the standardization that you mention - objects can have arbitrary serialization formats, but if certain attributes that we're interested in are recognizable, those attributes could be edited on a containerwide level. So, if we had arbitrary objects that we wanted to have an effect on content or display, the same XML tools could be used to manage them all, limited only by the ability of those tools to slog through a level of the hierarchy. This is kind of stream-of-consciousness talk, and might be more silly than realistic; certainly, if the objects didn't guard how they could be updated, a misconfigured tool could waste them without warning. There's a similar project called psilib that was discussed in xml.com: <http://www.xml.com/pub/a/2000/03/22/psi/index.html> -- Karl Anderson karl@digicool.com
I think the only good reasons we have right now for having filesystem-compatible serialization are to make Zope content editable via common tools in a way that makes sense to people not used to (or comfortable with) the object database, and to give people a plausible way to put a Zope site under source control. Are you thinking that we would build client-side tools to recognize an XML representation of a subpart of a site? ----- Original Message ----- From: "Karl Anderson" <karl@digicool.com> To: "Chris McDonough" <chrism@digicool.com> Cc: "John D. Heintz" <jheintz@isogen.com>; "Fred Wilson Horch" <fhorch@ecoaccess.org>; <zope-dev@zope.org> Sent: Thursday, March 22, 2001 8:17 PM Subject: Re: [Zope-dev] FTP interface being worked on?
"Chris McDonough" <chrism@digicool.com> writes:
I don't think it's reasonable or wise to impose any "master structure" for filesystem serialization of bodies of objects. Each instance (or perhaps each class) should define how best to serialize itself to disk. Representations between classes are likely to be radically different. A place for standardization is in the "properties" file(s) which accompany each object rep... this is likely to be XML or another structured variant.
Is there a motivation for using serialization to provide an editable "god's eye view" of all or part of a Zope site?
That is, provide a two-way XML serialization at any stage of the managed hierarchy of a site? That way, XML tools could be used as stream editors at any level, to the extent that the serialization is understandable.
So, for example, we'd have ways to not just alter a template or content that gets templatized, but the containers that organize them, and related metadata or content that isn't as near.
This links well with the standardization that you mention - objects can have arbitrary serialization formats, but if certain attributes that we're interested in are recognizable, those attributes could be edited on a containerwide level. So, if we had arbitrary objects that we wanted to have an effect on content or display, the same XML tools could be used to manage them all, limited only by the ability of those tools to slog through a level of the hierarchy.
This is kind of stream-of-consciousness talk, and might be more silly than realistic; certainly, if the objects didn't guard how they could be updated, a misconfigured tool could waste them without warning.
There's a similar project called psilib that was discussed in xml.com: <http://www.xml.com/pub/a/2000/03/22/psi/index.html>
-- Karl Anderson karl@digicool.com
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Hi Folks, I posted this to the Wiki... but it's not "in-your-face" like email, so I never know if anyone reads it. Here are a few, possibly random, but nonetheless concrete, thoughts of mine on the matter. I'm looking at all this from the perspective of someone who is using the current xml/zexp code to manage objects in CVS today, and hoping that whatever evolutionary/revolutionary process that occurs here not completely break all my existing stuff. It seems to me that the current import/export mechanism is actually pretty close to what we need for serialization. It has some nice properties: A) All objects are faithfully encoded and saved on the filesystem in a text format that any configuration management system can use and track. It seems to me that this property is critical to maintain. B) Whole object hierarchies can be saved in a simple file, moved to any other zope instance (via checkin/checkout) and instantiated. (This is needed for import/export to be useful) C) It's possible (through a hack[1]) to make simple objects 'diff'able so that all the metadata that's in the current xml export rep doesn't get in the way too much. (this is almost irrelavent, but may be useful as an interem step to make this part of the native import/export code... ) There are also a few not-so-nice properties: D) The current xml representation use by xml import/export is morally binary, in the sense that you can't easily edit/manage it with ordinary editors etc. E) There is no simple way to separate the contents of a containerish object from the container. I think that both of these shortcomings could be addresses using Chris M's idea of letting objects handle their own serialization. (e.g., Folders could save their meta-data in a 'special' file, or directory and then proceed to call the serialization methods of their contents). However I think there would be benefits to working this into the current import/export scheme. Imagine the following: A 'serialization' interface is invented that objects could implement that preserve A-C, and, optionally, fix D-E. If no such interface is found, the current export method is invoked. (I suppose this could also be done by adding a default implementation of the interface that simply does what the current import/export code does.) Something like this maybe? class FSObject: """ Abstract filesystem Object that 'does the right thing'. Might be implemented as StringIO, or impose special security restrictions, or whatever. """ def chdir(.. ) def mkdir(.. ) def open(.. ) class Serializable: Serialize(self, serialize_contents_separately=None, use_fs_object=None): """ Serialize this object. If serialize_contents_separately is not None then create a 'special' file for meta_data, and create a new directory for self and subobjects. After saving meta_data, call Serialize on each content object in turn, passing appropriate arguments through.. """ Anyway.. it seems to me that this sort of approach would have the benefit of using what's available today to the best advantage while permitting tools to be developed that would work well with whavever serialization process that evolves for all the Zope objects that are out there.. take care, -steve [1] See the ZCVMixin product: ZCVSExIm.py
"CM" == Chris McDonough <chrism@digicool.com> writes:
CM> I think the only good reasons we have right now for having CM> filesystem-compatible serialization are to make Zope content CM> editable via common tools in a way that makes sense to people CM> not used to (or comfortable with) the object database, and to CM> give people a plausible way to put a Zope site under source CM> control. CM> Are you thinking that we would build client-side tools to CM> recognize an XML representation of a subpart of a site? CM> ----- Original Message ----- From: "Karl Anderson" CM> <karl@digicool.com> To: "Chris McDonough" CM> <chrism@digicool.com> Cc: "John D. Heintz" CM> <jheintz@isogen.com>; "Fred Wilson Horch" CM> <fhorch@ecoaccess.org>; <zope-dev@zope.org> Sent: Thursday, CM> March 22, 2001 8:17 PM Subject: Re: [Zope-dev] FTP interface CM> being worked on? >> "Chris McDonough" <chrism@digicool.com> writes: >> >> > I don't think it's reasonable or wise to impose any "master > >> structure" for filesystem serialization of bodies of > objects. >> Each instance (or perhaps each class) should > define how best >> to serialize itself to disk. > Representations between classes >> are likely to be radically > different. A place for >> standardization is in the > "properties" file(s) which >> accompany each object rep... this > is likely to be XML or >> another structured variant. >> >> Is there a motivation for using serialization to provide an >> editable "god's eye view" of all or part of a Zope site? >> >> That is, provide a two-way XML serialization at any stage of >> the managed hierarchy of a site? That way, XML tools could be >> used as stream editors at any level, to the extent that the >> serialization is understandable. >> >> So, for example, we'd have ways to not just alter a template or >> content that gets templatized, but the containers that organize >> them, and related metadata or content that isn't as near. >> >> This links well with the standardization that you mention - >> objects can have arbitrary serialization formats, but if >> certain attributes that we're interested in are recognizable, >> those attributes could be edited on a containerwide level. So, >> if we had arbitrary objects that we wanted to have an effect on >> content or display, the same XML tools could be used to manage >> them all, limited only by the ability of those tools to slog >> through a level of the hierarchy. >> >> This is kind of stream-of-consciousness talk, and might be more >> silly than realistic; certainly, if the objects didn't guard >> how they could be updated, a misconfigured tool could waste >> them without warning. >> >> There's a similar project called psilib that was discussed in >> xml.com: <http://www.xml.com/pub/a/2000/03/22/psi/index.html> >> >> -- >> Karl Anderson karl@digicool.com >> >> _______________________________________________ Zope-Dev >> maillist - Zope-Dev@zope.org >> http://lists.zope.org/mailman/listinfo/zope-dev ** No cross >> posts or HTML encoding! ** (Related lists - >> http://lists.zope.org/mailman/listinfo/zope-announce >> http://lists.zope.org/mailman/listinfo/zope ) >> CM> _______________________________________________ Zope-Dev CM> maillist - Zope-Dev@zope.org CM> http://lists.zope.org/mailman/listinfo/zope-dev ** No cross CM> posts or HTML encoding! ** (Related lists - CM> http://lists.zope.org/mailman/listinfo/zope-announce CM> http://lists.zope.org/mailman/listinfo/zope )
Steve Spicklemire wrote:
I posted this to the Wiki... but it's not "in-your-face" like email, so I never know if anyone reads it.
Thanks for sending this to e-mail. (I never read the Wikis -- I mean to, but never find the time.)
I'm looking at all this from the perspective of someone who is using the current xml/zexp code to manage objects in CVS today
Can you tell me how you do that? Our big problem is that we can't import over existing objects. Are we doing something wrong? Here's our set up: We have developers all over the world that can check code in and out of SourceForge. We'd like our developers to be able to work on our site using their own local instances of Zope, export everything from their Zope ODB to their local filesystem, then use CVS to check in all the changes they've made to SourceForge. Then to keep in synch, everyone else can update their local copies of the source tree, and import the updates into their local instances of Zope. The problem: imports raise all sorts of conflict errors -- we basically have to delete everything before importing it.
It seems to me that the current import/export mechanism is actually pretty close to what we need for serialization.
I agree. We also have the hooks for a file system like representation already built in with the FTP interface. So I think an evolution rather than a revolution it called for. I want to comment on the rest of your message, but will have to do that later in the week. Thanks for sharing your ideas. Fred -- Fred Wilson Horch mailto:fhorch@ecoaccess.org Executive Director, EcoAccess http://ecoaccess.org/ P.O. Box 2823, Durham, NC 27715-2823 phone: 919.419-8354
Hi Fred,
"Fred" == Fred Wilson Horch <fhorch@ecoaccess.org> writes:
Fred> Steve Spicklemire wrote: >> I'm looking at all this from the perspective of someone who is >> using the current xml/zexp code to manage objects in CVS today Fred> Can you tell me how you do that? Our big problem is that we Fred> can't import over existing objects. Are we doing something Fred> wrong? I use ZCVSMixin (http://www.zope.org/Members/sspickle/ZCVSMixin). Fred> Here's our set up: Fred> We have developers all over the world that can check code in Fred> and out of SourceForge. We'd like our developers to be able Fred> to work on our site using their own local instances of Zope, Fred> export everything from their Zope ODB to their local Fred> filesystem, then use CVS to check in all the changes they've Fred> made to SourceForge. Then to keep in synch, everyone else Fred> can update their local copies of the source tree, and import Fred> the updates into their local instances of Zope. Fred> The problem: imports raise all sorts of conflict errors -- Fred> we basically have to delete everything before importing it. ZCVSMixin does the loading of objects into folderish containers so conflict errors are not a problem. This naturally means that you can destroy things if you're not careful. (Motto: a tool that doesn't let you destroy things now & then.. is a weak tool.) -steve
Wanted to follow up on Steve's points. He wrote in part:
[...] It seems to me that the current import/export mechanism is actually pretty close to what we need for serialization.[...]
A) All objects are faithfully encoded and saved on the filesystem in a text format that any configuration management system can use and track. [...]
B) Whole object hierarchies can be saved in a simple file, moved to any other zope instance (via checkin/checkout) and instantiated. [...]
C) It's possible (through a hack[1]) to make simple objects 'diff'able so that all the metadata that's in the current xml export rep doesn't get in the way too much. [...]
There are also a few not-so-nice properties:
D) The current xml representation use by xml import/export is morally binary, in the sense that you can't easily edit/manage it with ordinary editors etc.
E) There is no simple way to separate the contents of a containerish object from the container.
[...] Imagine the following:
A 'serialization' interface is invented that objects could implement that preserve A-C, and, optionally, fix D-E. If no such interface is found, the current export method is invoked. [...]
I don't know if we need just one serialization interface that tries to solve all five issues. We currently have two serialization interfaces in Zope: 1) the FTP interface, and 2) the XML export interface. Seems to me that the FTP interface could be generalized as the "lossy" serialization interface that is "morally human readable" and the XML export interface could be tweaked a bit as "lossless" but "morally binary". The FTP interface would be used when you want a representation that can be edited by hand. It solves C, D and E, but not A and B. The XML interface would be used when you want a representation that can be manipulated programmatically. It solves A and B, but not C, D and E. FWIW, I'm working on tweaking the XML export/import code to serialize object hierarchies as directories and files, rather than exporting a single file. Fred P.S. The link Karl sent (http://www.thetwowayweb.com/theXmlFiles) is interesting. -- Fred Wilson Horch mailto:fhorch@ecoaccess.org Executive Director, EcoAccess http://ecoaccess.org/ P.O. Box 2823, Durham, NC 27715-2823 phone: 919.419-8354
Hi Fred,
"Fred" == Fred Wilson Horch <fhorch@ecoaccess.org> writes:
Fred> Wanted to follow up on Steve's points. Fred> I don't know if we need just one serialization interface Fred> that tries to solve all five issues. Ok.. Fred> We currently have two serialization interfaces in Zope: Fred> 1) the FTP interface, and 2) the XML export interface. Fred> Seems to me that the FTP interface could be generalized as Fred> the "lossy" serialization interface that is "morally human Fred> readable" and the XML export interface could be tweaked a Fred> bit as "lossless" but "morally binary". Fred> The FTP interface would be used when you want a Fred> representation that can be edited by hand. It solves C, D Fred> and E, but not A and B. Fred> The XML interface would be used when you want a Fred> representation that can be manipulated programmatically. It Fred> solves A and B, but not C, D and E. Hmm.. maybe I'm misuderstanding... which would/could you use for version control? It still seems to me that a blend of these could be developed that would work in all cases. An object could have a human readable part/aspect and 'the rest' could be captured as an xml bloblet. The 'human editable' part could just be what you get on the FTP interface (as you say), and the rest could be saved/restored to a 'auxiliary' file that would track other aspects of the object. Since objects could implement their own serialization, they could decide what aspects belonged in which part. Otherwise I can't help feeling that we'd have problems with duplicated versions, some xml/zexp, some human-readable that would inevitably stomp on eachother in any version control scenario. If all this could be worked in to the existing FTP/export/import system... there would be a minimum impact on existing interfaces. Fred> FWIW, I'm working on tweaking the XML export/import code to Fred> serialize object hierarchies as directories and files, Fred> rather than exporting a single file. Cool.. this sounds like a promising approach. I'd be interested in testing this.. thanks! -steve Fred> Fred Fred> P.S. The link Karl sent Fred> (http://www.thetwowayweb.com/theXmlFiles) is interesting. Fred> -- Fred Wilson Horch mailto:fhorch@ecoaccess.org Executive Fred> Director, EcoAccess http://ecoaccess.org/ P.O. Box 2823, Fred> Durham, NC 27715-2823 phone: 919.419-8354
Steve Spicklemire wrote:
Fred> FWIW, I'm working on tweaking the XML export/import code to
Fred> serialize object hierarchies as directories and files, Fred> rather than exporting a single file.
Cool.. this sounds like a promising approach. I'd be interested in testing this..
Indeed :-) A slight further tweak... I'd love to be able to tick a group of objects and hit the export button. Maybe the stuff Fred is looking at could be bent to do that? cheers, Chris
Hi Steve, You wrote:
Fred> We currently have two serialization interfaces in Zope: Fred> 1) the FTP interface, and 2) the XML export interface.
Hmm.. maybe I'm misuderstanding... which would/could you use for version control?
The XML interface.
It still seems to me that a blend of these could be developed that would work in all cases. An object could have a human readable part/aspect and 'the rest' could be captured as an xml bloblet. The 'human editable' part could just be what you get on the FTP interface (as you say), and the rest could be saved/restored to a 'auxiliary' file that would track other aspects of the object. Since objects could implement their own serialization, they could decide what aspects belonged in which part. Otherwise I can't help feeling that we'd have problems with duplicated versions, some xml/zexp, some human-readable that would inevitably stomp on eachother in any version control scenario. If all this could be worked in to the existing FTP/export/import system... there would be a minimum impact on existing interfaces.
That was my original sentiment, too. But Chris McDonough's proposal (http://dev.zope.org/Wikis/DevSite/Proposals/RepresentingObjectsOnTheFilesyst...) seems to allow developers to ignore any 'auxiliary' information that is not easy to serialize. I've come around to the viewpoint that the FTP interface (which in my opinion is what Chris' proposal is replacing) could just be used for editing objects one at a time. You can afford not to represent some information in this situation. It is just like through the web editing, but allows you to use emacs or Microsoft Word (or whatever client side tools you have that work with files on your filesystem).
Fred> FWIW, I'm working on tweaking the XML export/import code to Fred> serialize object hierarchies as directories and files, Fred> rather than exporting a single file.
Cool.. this sounds like a promising approach. I'd be interested in testing this..
The reason I'm working on this is that I think the XML serialization is what developers should use for source control / configuration management. There are really only two major problems with the existing XML serialization functionality that prevent us from using it with CVS: 1) everything is serialized to a single monolithic XML export file, and 2) you cannot update existing objects from an XML export file Hopefully I will have some code soon based on the existing export code that allows you to export any part of the object tree as a hierarchy of directories and files in Python Pickle Markup Language (ppml - the format used by the existing XML export), and import ppml files to update existing objects in the ZODB. By the way -- is it me, or is the current Import/Export interface broken? I tried to select multiple objects to export, but I can only get the first one to actually be exported. Fred -- Fred Wilson Horch mailto:fhorch@ecoaccess.org Executive Director, EcoAccess http://ecoaccess.org/ P.O. Box 2823, Durham, NC 27715-2823 phone: 919.419-8354
Hi!
By the way -- is it me, or is the current Import/Export interface broken? I tried to select multiple objects to export, but I can only get the first one to actually be exported.
It's not just you.. but never thought about whether this might be a bug ;-) -- christian -- COM.lounge http://comlounge.net/ communication & design info@comlounge.net
By the way -- is it me, or is the current Import/Export interface broken? I tried to select multiple objects to export, but I can only get the first one to actually be exported.
Nope, that's the way it's supposed to work, I think :-) It's why I got happy when I saw your earlier post; it looks like it could create a way around this... cheers, Chris
participants (6)
-
Chris McDonough -
Chris Withers -
cs＠comlounge.net -
Fred Wilson Horch -
Karl Anderson -
Steve Spicklemire