October 2001 - Zope-Dev

Twisted, medusa, ZServer, and VFS's
by Donovan Baarda 31 Oct '01

31 Oct '01

G'day, I've just been looking at Twisted Python for the past hour or two, read the mailing list archives, and have some comments and questions. I've been working on using Medusa for serving a virtual mirror via http and ftp. I'm at the point where I'm close, but I'm starting to re-think some stuff, in particular my choice of Medusa. First my impressions of the three contenders; Twisted, Medusa, ZServer. Please correct me if I'm wrong in the following summaries; Medusa seems to be the daddy of them all. It's the oldest, which has benefits and problems. It is a little messy from its evolution, but seems pretty mature. It uses an async select loop to drive everything. It uses "asynchat" derived class objects to communicate on sockets. Data can be sent by pushing "producers" onto asynchat objects. Producers can be complex objects that produce data, execute callbacks, whatever. It's ftp and http server classes use a primative VFS to serve from. The asyncore and asynchat modules it is built on are now part of Python. Twisted seems to be a from-the-ground up re-invention of Medusa. It's newer, but surprisingly it's bigger, dispite it's apparently less mature feature set. It is similar in structure to Medusa, but simplifys it by dispensing with producers. It can use a variety of event-loops, including Tk and GTK, or it's own. It doesen't have a VFS (yet) so its ftp and http servers serve from the underlying os filesystem. ZServer grew out of Medusa. It uses the same basic underlying architecture, but throws in threads to get around the problem of delayed producers blocking the event loop. I'm not sure how tightly tied to Zope it is, but its http and ftp servers generally serve from a ZODB database, presumably wrapped in a Medusa VFS, though I have a feeling they might have changed that. ZServer also supports webDAV serving. It is possible that some of the enhancements could be merged back into Medusa, but probably it has changed so much it would be difficult. My problem with Medusa is its http and ftp servers assume that the VFS can deliver files wrapped in producers without blocking. I've fixed this by creating a patch for Medusa's asynchat that adds support for a ready() method to producers, so they can block without blocking the event loop. I'm currently in the process of writing Medusa VFS's for ftp and eventualy http backends. In the process I've also found that the Medusa ftp server is not as full featured as I want. In my search for Python VFS's I found PyVFS (http://www.pycage.de/) This is modelled on the Gnome/MC VFS, so it supports '/dir/somefile.tar.gz#tgz:/somepath' style paths to look inside tar, tgz, ftp, whatever. The various different VFS backends are loaded dynamicly as pluggins. These pluggins execute as a seperate process that are communicated with over a channel. The API is too simplistic for me, with files being "projected" out of the VFS to a local file to be manipulated/used. I don't like the pluggin-process-channel architecture either. In the reading of the Twisted mailing list, I saw a comment to the affect that the Medusa VFS was an example of how _not_ to do it, which lead to using webDAV as the API for a VFS. My gut feeling is DAV is a cool protocol for a VFS backend, but I dunno about using it as the primary API. Sure, it supports meta- data etc, but the reality is the API that is most widely used and understood is the POSIX filesytem API, as exposed in Python by the os and os.path modules. My solution for a VFS has been, upto now, based on Medusa's, but extending it to be more like os and os.path. So far it's a filesystem class with most of the os and os.path methods. One of the derived classes is a mountable_filesystem that allows you to mount other VFS filesystems off it. At this point I'm tempted to make a vfs module that emulates os and os.path so that you can mount whatever vfs's you want first, and then just replace all your os.* calls with vfs.* calls. Note that the one catch would be open() would need to be replaced with vfs.open(). I'm sort of fishing for general suggestions, comments, and interest. I'm at the point where I've just convinced myself my vfs is worth finishing, and my ready () patch to asynchat is worth updating, but I'm not sure what to use as the http and ftp server front-end, though I'm still leaning towards medusa. It looks like Twister is not ready, and ZServer would be too hard to seperate from Zope. PS... I'm not on the zope-dev list but I am on the twister and medusa lists. The zope-dev list is too much non-ZServer stuff and that's all I'm interested in. So zope-dev'ers, please reply to me or one/both of the other lists directly. -- ABO: finger abo(a)minkirri.apana.org.au for more information.

25 70

Re: [Zope-dev] Any interest in a set of Zope products to support CVS-versioned, XML/XSLT-based Zope development?
by Anthony Baxter 31 Oct '01

31 Oct '01

You might want to look at the fishbowl project on this that was recently announced: http://dev.zope.org/Projects/Wikis/DevSite/Projects/ZopeVersionControl/ (URL copied from Brian Lloyd's earlier email - with a 7s delay in the AU-US net link at the moment, I can't check it's correct, unfortunately...) Anthony -- Anthony Baxter <anthony(a)interlink.com.au> It's never too late to have a happy childhood.

1 0

improving str(REQUEST)
by seb bacon 31 Oct '01

31 Oct '01

The REQUEST __str__ representation can be useful for tracking down bugs, and is a useful didactic tool for newbies. I wanted to get feedback on some ways it has occured to me that it might be improved. If it turns out to be a good idea, I'll do a fishbowl proposal (one day...) In its simplest form, it would be nice if it were a collapsable tree, so you could examine just the bits you were interested in. I think it would also be really handy to extend this concept, for example adding nodes to the AUTHENTICATED_USER branch, which show local roles, roles, etc. Perhaps automatically, through introspection? seb

4 4

Re: [Parsed-XML-Dev] Re: [Zope-dev] ParsedXML in Zclass methods loses permissions on Zope restart
by Karl Anderson 30 Oct '01

30 Oct '01

Martijn Faassen <faassen(a)vet.uu.nl> writes: > Brad Clements wrote: > > I'm still casting around for a suggestion on where I can go to fix this. > > > > I have a ParsedXML object in the methods list of a ZClass Product. > > > > The "Access Contents Information" Permission Mapping always get's reset to "blank" in > > the ParsedXML object when Zope restarts. > > > > Is this a problem with ZClasses, or with ParsedXML? > > Taking a look at ParsedXML, it looks like it just does a very simple > permission definition, so it doesn't *look* like there's something > wrong there. > > What it sounds like is that something that should be persistent isn't > for some reason. Either an object attribute is changed without > the permission mechanism being informed of the change, or it's > for some reason an attribute on a nonpersistent is changed. This at least > would explain why the permission mapping magically goes away.. I don't understand the situation very well, but Martijn's explanation sounds familiar. Remember, there's only one persistent Zope object in a ParsedXML instance, and that's the document proxy object. All of the proxy subobjects are transient, and that includes proxy documents that are navigated to from proxy subobjects. I.E. if "app.foo.PXML" is the ParsedXML instance, then setting Zope attributes on "app.foo.PXML.firstChild()" and "app.foo.PXML.firstChild.parentNode()" won't persist. Same goes if you traverse to these nodes via the web-exposed URLs of these methods. Perhaps you're trying to persist changes to Zope attributes on those nodes? -- Karl Anderson kra(a)monkey.org http://www.monkey.org/~kra/

1 0

(no subject)
by biz＠ebookleader.org 30 Oct '01

30 Oct '01

1 0

ParsedXML in Zclass methods loses permissions on Zope restart
by Brad Clements 30 Oct '01

30 Oct '01

I'm still casting around for a suggestion on where I can go to fix this. I have a ParsedXML object in the methods list of a ZClass Product. The "Access Contents Information" Permission Mapping always get's reset to "blank" in the ParsedXML object when Zope restarts. Is this a problem with ZClasses, or with ParsedXML? I put this in the collector 3 or 4 months ago, and have asked a few times on different lists.. Got no responses. I now must fix it, can't put it off any longer. Any suggestions where to look in the source for where Permission mappings are stored? Perhaps ZClasses stores the permission mappings, but ParsedXML overrides the restored value somehow? Zope 2.3.3 (binary release, python 1.5.2, linux2-x86) (Using last released beta parsed XML) Brad Clements, bkc(a)murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements

2 1

Re: [Zope-dev] fsrecover error
by Scott Burton 29 Oct '01

29 Oct '01

Thanks, that worked. I got the message: Nothing to recover... So, I used Tranalyzer and followed the instructions from here: http://www.zope.org/Members/itamar/CorruptedZODB Now I get an entirely different error: 2001-10-29T20:49:25 ERROR(200) ZODB Couldn't load state for '\000\000\000\000\000\000\000\002' Traceback (innermost last): File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/Connection.py, line 508, in setstate File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 593, in load (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 569, in _load (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) KeyError: ------ 2001-10-29T20:49:25 PANIC(300) z2 Startup exception ------ 2001-10-29T20:49:25 ERROR(200) ZODB Couldn't load state for '\000\000\000\000\000\000\000\002' Traceback (innermost last): File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/Connection.py, line 508, in setstate File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 593, in load (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 569, in _load (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) KeyError: Traceback (innermost last): File /home/zopesite/Zope-2.3.0-src/z2.py, line 566, in ? File <string>, line 1, in ? File /home/zopesite/Zope-2.3.0-src/lib/python/Zope/__init__.py, line 144, in ? File /home/zopesite/Zope-2.3.0-src/lib/python/OFS/Application.py, line 384, in initialize File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/Connection.py, line 508, in setstate File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 593, in load (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 569, in _load (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) KeyError: >From: Jens Vagelpohl <jens(a)zope.com> >To: "Scott Burton" <mrzope(a)hotmail.com> >CC: zope-dev(a)zope.org >Subject: Re: [Zope-dev] fsrecover error >Date: Mon, 29 Oct 2001 15:36:18 -0500 > >from where are you running it? i believe you must change into ><zope>/lib/python and run it from that location so that all the imports >can be found. > >jens > > >On Monday, October 29, 2001, at 03:20 , Scott Burton wrote: > >>I have a corrupted Data.fs and I am attempting to use fsrecover.py to fix >>it. Here is the error I am getting on the startup: >>2001-10-29T18:57:18 PANIC(300) ZODB FS FS21 ERROR: /home/zopesite/Zope-2. >>3.0-src/var/Data.fs data record exceeds transaction record at 246204937L >> >>------ >>2001-10-29T18:57:18 PANIC(300) z2 Startup exception >>Traceback (innermost last): >> File /home/zopesite/Zope-2.3.0-src/z2.py, line 566, in ? >> File <string>, line 1, in ? >> File /home/zopesite/Zope-2.3.0-src/lib/python/Zope/__init__.py, line >>110, >> in ? >> File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line >>308, in __init__ >> (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) >> File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line >>1669, in read_index >> File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line >>219, in panic >>CorruptedTransactionError: /home/zopesite/Zope-2.3.0-src/var/Data.fs data >>record exceeds transaction record at 246204937L >> >>Here is the error I get when running fsrecover: >>Traceback (innermost last): >> File "lib/python/ZODB/fsrecover.py", line 92, in ? >> import FileStorage >> File "/home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py", >>line 193, in ? >> from TimeStamp import TimeStamp >>ImportError: No module named ExtensionClass >> >>I have searched the lists and have not found a solution. Any ideas? >> > > >_______________________________________________ >Zope-Dev maillist - Zope-Dev(a)zope.org >http://lists.zope.org/mailman/listinfo/zope-dev >** No cross posts or HTML encoding! ** >(Related lists - >http://lists.zope.org/mailman/listinfo/zope-announce >http://lists.zope.org/mailman/listinfo/zope ) _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp

2 1

fsrecover error
by Scott Burton 29 Oct '01

29 Oct '01

I have a corrupted Data.fs and I am attempting to use fsrecover.py to fix it. Here is the error I am getting on the startup: 2001-10-29T18:57:18 PANIC(300) ZODB FS FS21 ERROR: /home/zopesite/Zope-2.3.0-src/var/Data.fs data record exceeds transaction record at 246204937L ------ 2001-10-29T18:57:18 PANIC(300) z2 Startup exception Traceback (innermost last): File /home/zopesite/Zope-2.3.0-src/z2.py, line 566, in ? File <string>, line 1, in ? File /home/zopesite/Zope-2.3.0-src/lib/python/Zope/__init__.py, line 110, in ? File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 308, in __init__ (Object: /home/zopesite/Zope-2.3.0-src/var/Data.fs) File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 1669, in read_index File /home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py, line 219, in panic CorruptedTransactionError: /home/zopesite/Zope-2.3.0-src/var/Data.fs data record exceeds transaction record at 246204937L Here is the error I get when running fsrecover: Traceback (innermost last): File "lib/python/ZODB/fsrecover.py", line 92, in ? import FileStorage File "/home/zopesite/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py", line 193, in ? from TimeStamp import TimeStamp ImportError: No module named ExtensionClass I have searched the lists and have not found a solution. Any ideas? _________________________________________________________________ Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp

2 1

CollectorReplacment online
by Ken Manheimer 29 Oct '01

29 Oct '01

We finally have a new version of the Zope bug collector in place, at: http://new.zope.org/Collector Note that it's *new*.zope.org not the main www.zope.org site. This is because the new collector is built on the CMF, which is part what will be the new site (once it's commissioned). I realize this has taken longer to land than any of us would like - we're eager to resume collect and addressing the important bugs, and i know some of you have important ones you're yearning to submit. In the interest of reducing further delays, we're jamming this new collector into production - we've done what we could to shake out the major bugs (many thanks particularly to magnus heino for patiently continuing to give it a whirl!), but i expect there are more to fall out as it gets really used. Please bear with us - and submit your Zope bugs! (There's a separate collector for collector-specific issues: http://new.zope.org/Members/klm/ColDev Submit collector bugs and suggestions there. If the collector is so whacked you can't even submit your issue, email the description directly to me. We'll try to deal with it promptly! The collector is an installed product on new.zope.org - anyone with an account there can create an instance of their own to play with, in their member folder, and the product is checked into the Zope public CVS repository (cvs.zope.org) as CMF/CMFCollector.) Happy reporting!-) -- Ken Manheimer klm(a)zope.com

1 0

validate() trouble in Zope-2.5.0a1
by Steve Spicklemire 29 Oct '01

29 Oct '01

I don't know if I'm doing something wrong, but when I try to unpack and run Zope-2.5.0a1 I can't seem to authenticate manage_workspace. I haven't snooped much, but it seems that "options" in Management.py comes back empty: > /usr/local/etc/Zope2d/lib/python/App/Management.py(147)manage_workspace() -> try: (Pdb) print options [] (Pdb) l 142 """ 143 import pdb 144 pdb.set_trace() 145 146 options=self.filtered_manage_options(REQUEST) 147 -> try: 148 m=options[0]['action'] 149 if m=='manage_workspace': raise TypeError 150 except: 151 raise Unauthorized, ( 152 'You are not authorized to view this object.') When I look into self.filtered_manage_options(REQUEST) it seems that "getSecurityManager.validate" always throws an exception "this function takes no keyword arguments" > /usr/local/etc/Zope2d/lib/python/App/Management.py(130)filtered_manage_options( ) -> if validate(container=self, value=o): (Pdb) n TypeError: 'this functio...ord arguments' > /usr/local/etc/Zope2d/lib/python/App/Management.py(130)filtered_manage_options( ) -> if validate(container=self, value=o): (Pdb) n > /usr/local/etc/Zope2d/lib/python/App/Management.py(132)filtered_manage_options( ) -> except: (Pdb) l 127 if o is None: continue 128 129 try: 130 if validate(container=self, value=o): 131 result.append(d) 132 -> except: 133 if not hasattr(o, '__roles__'): 134 result.append(d) 135 136 return result (Pdb) print __exception__ (<class exceptions.TypeError at 0x80ec60c>, 'this function takes no keyword arguments') (Pdb) Anyway.. no more time to play now.. any idea what's up? thanks, -steve

3 2