December 2001 - Zope-Dev

My thoughts on the development process
by Martijn Faassen 10 Dec '01

10 Dec '01

Hi there, I've read parts of the open letter threads just now. There's a lot of talk about how if only we have better tools the whole process will go better and Zope will get more contributors. That's a typical hacker response, and I do this myself as well. Throwing more technology at a problem doesn't always make a problem go away. And though technological solutions to social problems are nice if you can have them, and we should look for them, they don't always work. I'm not convinced more technology will make the dead fish problem go away. I think the contributing process is in fact too heavyweight. It should be easier for people to get in drastic changes to Zope. The only way for people to take more responsibility if they can actually have it. Only a few people will take it, but that's more than what is possible now, with possibly the single exception of my taking responsibility for ParsedXML. And until recently I was still in the position of doubting whether I really had it formally, not just de-facto. I kept asking for approval and guidelines from the official maintainers, but they were too busy (no blame to them), so I went on anyway and did a release eventually. I dread having to go through the fishbowl to add in my 'node path' implementation to ParsedXML. I've done the design work, I've implemented most of it, and I feel I'd have mostly wasted time writing a fishbowl proposal. I hadn't even explored the problem enough to be able to do that. I needed to prototype it to understand it. I've discussed some issues with people locally and and on the Zope-XML mailing list. And I'll probably release a version in a few days. Perhaps adding Formulator to the Zope core would be nice eventually. But going through the fishbowl bureaucracy would take forever. I only have so much time to spend on it, and I'd rather spend time improving the product itself. And now look at how the Zope core is actually being developed. Sure, there's lots of stuff in the fishbowl about what the Zope future should be like. Plenty of stuff, though some stuff is rather hard to find. But I have a lot of praise for what the Zope Corp people have accomplished it it; it's a lot better than having no such thing at all, even if it's only used as a notification service in part. The main thinking about the directions of Zope is not done in the fishbowl or on the lists, it's in the minds of the talented people at Zope Corp and in the brainstorm sessions they hold together. That's the natural way people work. I work that way too. Such a process can occur on mailing lists as well, but it's very hard to break into it. I've tried several times. I'll keep trying as I'm convinced it's possible, but it takes a lot of persistence. Time will tell. On the Zope-XML list I just post regular updates about my thinking to encourage discussion, and sometimes that works. So what am I trying to get at with this mail? One thing is that the process is too heavy-weight right now. The other thing is that the core coders at Zope Corp are in a position nobody else is in, and that should change. They are the only ones that can get around the fishbowl if they so desire. They can use the fishbowl in effect as a notification service. Not that they want to; I don't doubt their good intentions for one minute. But I want to be able to use it like that too when I need to. Others should be able to as well. I think I and a few other contributors are slowly getting to that position, but it happens too slowly and takes way too much persistence now. So let's trade in some risks to the Zope core development (rash action and messed up stuff happening once every while), in exchange for a lot more active contributors. Regards, Martijn

6 15

Python Conference News
by Joya Subudhi 10 Dec '01

10 Dec '01

Python 10 Conference News Tim Berners-Lee, Director of the World Wide Web Consortium, and Andrew Koenig, Principal Research Staff Member at AT&T Shannon Laboratory, will give the keynote addresses. Tim Berners-Lee will speak on "Webizing Python" and Andrew Koenig will discuss "Notes from a Polygot Outsider." The entire program of the Refereed Paper Track is complete. You can view the titles and authors of all the presentations within each session and check out nominations for the Best Paper Award on the Python 10 Web site. The Tutorials Day program is also complete. Silver Sponsors of Python 10 are: Hostway Corporation: http://www.hostway.com and New Riders Publishing: http://www.newriders.com Registration for Python 10 is open. Save with early bird registration rates when you register online before January 7, 2002. To register and for conference details, visit: http://www.python10.org The Tenth International Python Conference February 4th through February 7th The Hilton Alexandria Mark Center Alexandria, Virginia

1 0

100k+ objects, or...Improving Performance of BTreeFolder...
by sean.upton＠uniontrib.com 10 Dec '01

10 Dec '01

Interesting FYI for those looking to support lots of cataloged objects in ZODB and Zope (Chris W., et al)... I'm working on a project to put ~350k Cataloged objects (customer database) in a single BTreeFolder-derived container; these objects are 'proxy' objects which each expose a single record in a relational dataset, and allow about 8 fields to be indexed (2 of which, TextIndexes). Some informal stress tests using 100k+ _Cataloged_ objects in a BTreeFolder in Zope 2.3.3 on my PIII/500/256mb laptop are proving to be successful, but not without some stubborn investigation and a few caveats. BTreeFolder, using ObjectManager APIs, frankly, just won't scale for bulk-adds of objects to folders. I was adding CatalogAware objects to my folder (including index_object()). After waiting for bulk-add processes to finish after running for 2 days, I killed Zope and started trying to optimize, figuring that the problem was related to Catalog and my own RDB access code, and got nowhere (well, I tuned my app, bu this didn't solve my problem), then went to #zope, got a few ideas, and ended up with the conclusion that my problem was not Catalog-related, but related to BTreeFolder; I initially thought it was a problem with the C-Based generic BTree implementation scaling well past 10k objects, but felt I couldn't point the finger at that before some more basic stuff was ruled out. The easiest thing to do in this case, was to figure out what was heavily accessing the BTree via its dictionary-like interface, and the thought occurred to me that there might be multiple has_key checks, security stuff, and the like called by ObjectManager._setObject(), and I was right. I figured a switch to use the simple BasicBTreeFolder._setOb() for my stress tests might reveal an increase in speed, and... ...it works, acceptably, no less, on my slow laptop for 100,000 objects. It took ~50 minutes to do this on meager hardware with a 4200 RPM ide disk, and I figure a bulk add process like this on fast, new hardware (i.e. something with upwards of 22k pystones and lots of RAM) with a dedicated server for my RDB, would likely take 1/5th this time, or about 10 minutes (by increasing both MySQL performance, and Zope performance); combine this with ZEO and have a dedicated node do this, and I think this is a small amount of proof of Zope's ability to scale to many objects. (See my caveats at the bottom of this message, though). After days of frustration, I'm actually impressed by what I found: My data-access APIs are very computationally expensive, since they establish a MySQLdb cursor object for each call and execute a query; these data access methods used in bulk adding 100k objects after using _setOb() during Cataloging via index_object() (the transaction done all in memory for now, but likely moved to subtransactions soon to support up to 4x that data). So far, the moral of the story: use _setOb(), not _setObject() for this many objects! I haven't seen any material documenting anything like this for BTreeFolder, so I figured I would share with zope-dev what I found in the hopes that developers creating products with BTreeFolder and/or future implementations of BTreeFolder might take this into account, in docs, if nothing else. Caveats: - I'm using FileStorage and an old version of Zope (2.3.3). I can't say how this will perform with Python 2.1/Zope 2.[4/5]. I imagine that one would want to pack the storage between full rebuilds or have very, very fast storage hardware. - Catalog searches without any limiting queries to indexes will simply be too slow for practical use with this many objects, so they need to be forbidden with a permission to prevent accidental over-utilization of system resources or DOS-style attacks. Otherwise, Catalog searches on my slow hard drive seem acceptable. - I'm not too concerned with BTreeFolder __getattr__() performance penalties, though I modified BTreeFolder.__getattr__ just in case to remove the 'if tree and tree.has_key(name)', replacing with try/except; I'm not sure if this helps/hinders, because my stress-test code uses _getOb() instead. - objectIds() doesn't work; or, more accurately, at first glance, <dtml-var "_.len(objectIds())"> doesn't work; I haven't tested anything else. I would like to find out why this is, and fix it. I suppose that there is something done in ObjectManager that BTreeFolder's simple _setOb() doesn't do. If anyone wants to help me figure out the obvious here, I'd appreciate it. ;) - I don't think un-indexed access of records is likely to be very practical with this many, esp. if things like objectIds() are broken, which increases the value of Catalog, and I think that what my experiences here with this project are showing is that Catalog indexing isn't as expensive/slow as I initially thought it would be. That said, I'm sure there can be improvements in Catalog as often is discussed here recently, but for now, I think I'm happy. :) - I Haven't compared these results with OFS.Folder.Folder yet. I'm too lazy/busy to comparison test. - I'm relatively sure that, in my app, the text index BTrees in the Catalog are very 'bushy' (more so than normal) because I am indexing people's full names, and street addresses, which means there are less common words than indexing, say, an every-day document. - Also, I want to make it clear that if I had a data access API that needed more than simple information about my datasets (i.e. I was trying to do reporting on patterns, like CRM-ish types of applications), I would likely wrap a function around indexes done in the RDB, not in Catalog. My requires no reporting functionality, and thus really needs no indexes, other than for finding a record for customer service purposes and account validation purposes. The reason, however, that I chose ZCatalog was for full text indexing that I could control/hack/customize easily. My slightly uninformed belief now is that for big datasets or "enterprise" applications (whatever that means), I would use a hybrid set of (faster) indexes using the RDB's indexes where appropriate (heavily queried fields), and ZCatalog for TextIndexes (convenient). I'm sure inevitable improvements to ZCatalog (there seems to be community interest in such) will help here. - I wonder if "directory-storage" combined with ReiserFS might make for an interesting future ZODB choice for this sort of app. Sean ========================= Sean Upton Senior Programmer/Analyst SignOnSanDiego.com The San Diego Union-Tribune 619.718.5241 sean.upton(a)uniontrib.com =========================

5 7

Re: [Zope-dev] zope collector is broken
by Ken Manheimer 10 Dec '01

10 Dec '01

I'm sorry, i'm unable to reprovoke the problem! Is there anything unusual about how you were submitting? Did you try more than once, to see if the problem is consistent? On Fri, 07 Dec 2001 11:43:52 -0500, Chris Deckard wrote: > I tried submitting a new issue to the Zope collector at 11:42AM > EST. Here's the traceback and error message: > > Zope Error > Zope has encountered an error while publishing this resource. > > Error Type: WorkflowException Error Value: No workflow > provides the "request" action. > [...] > WorkflowException: (see above) -- Ken klm(a)zope.com

5 4

Is it possible to get REQUEST from dtml tag class.
by Y. Cheng 10 Dec '01

10 Dec '01

Hi, There is a bug in dtml-tree, the bug should be in file: Zope-2.5.0b2-src/lib/python/TreeDisplay.py we need to know BASEPATH1 around line 626 which is: ------------------------------ if exp: ptreeData['tree-item-expanded']=1 output('<A NAME="%s" HREF="%s?%stree-c=%s#%s">' '<IMG SRC="%s/p_/mi" ALT="-" BORDER=0></A>' % (id, root_url, param, s, id, script)) else: output('<A NAME="%s" HREF="%s?%stree-e=%s#%s">' '<IMG SRC="%s/p_/pl" ALT="+" BORDER=0></A>' % (id, root_url, param, s, id, script)) output('</TD>\n') ------------------------------ sorry I don't know how to fix this, thank for your great work for zope. Yuan-Chen Cheng

1 0

Zope and Curl
by Dirk Datzert 10 Dec '01

10 Dec '01

Hi, has any zoper played with the new internet technology curl from http://www.curl.com ? Regards, Dirk

2 1

Re: [Zope-dev] RAMCacheManager and gzip
by Adrian Hungate 10 Dec '01

10 Dec '01

There is zlib support in Python, and it is used in Medusa already. ZServer does not call it, but the compression stuff is there. There is one good reason not to use zlib though, Micro$oft Internet Distorter does not support content-encoding: gzip and for content-encoding: compress it expects data to be compressed using the Unix compress tool... (I think someone reported this already... oh well) This one needs more work. Netscape 4.7 (Latest version of communicator) does not support HTTP1.1 at all, so it can not do content-encoding at all. (This was a complete supprise to me - Netscape list this as their current version of communicator!) I have to download the latest Mozilla and Netscape 6.2 and see if they work properly. I suspect this is a non-starter as the support is not reliably present in the clients. I don't see any reason to continue this as a project - even if Mozilla and Netscape 6.x support content encoding properly, and even if the smaller browsers (Noteably Opera I guess) support it, so many people use MSID that this would be a major problem - All pages would fail to display as MSID reports that it does HTTP1.1 !!! I do have another server-side caching project that I am about to start looking at though, so we might still see some speed improvements. Adrian... -- Adrian Hungate All views expressed in this email are those of the whole world, however some people don't realise this yet. ----- Original Message ----- From: "JanStiller T-Online" <JanStiller(a)t-online.de> To: "Adrian Hungate" <adrian(a)haqa.co.uk> Sent: Sunday, December 09, 2001 6:33 PM Subject: RE: [Zope-dev] RAMCacheManager and gzip > ;o) > > I think that zlib would be the best, too. But, I think that - while trying > mod_gzip - I read somewhere that zlib isn't thread-safe. Maybe you want to > check this out... > > I primarily asked because I have no experience in making Python calls to a > .so, so I spawn an os.system everytime - which doesn't matter that much > because my cache lasts for several days, but it's nonetheless :o( > > > ciao Jan > > > > > > -----Original Message----- > > From: Adrian Hungate [mailto:adrian@haqa.co.uk] > > Sent: Samstag, 8. Dezember 2001 00:42 > > To: JanStiller T-Online > > Subject: Re: [Zope-dev] RAMCacheManager and gzip > > > > > > I intend to use zlib, (Infact I may just slot in a > > ZServer.medusa.producers.compressed_producer somewhere) why is there a > > better one that I can be sure exists on every platform? > > > > Adrian... > > > > -- > > Adrian Hungate > > > > All views expressed in this email are those of the whole world, > > however some > > people don't realise this yet. > > > > ----- Original Message ----- > > From: "JanStiller T-Online" <JanStiller(a)t-online.de> > > To: "Adrian Hungate" <adrian(a)haqa.co.uk> > > Sent: Thursday, December 06, 2001 10:35 PM > > Subject: RE: [Zope-dev] RAMCacheManager and gzip > > > > > > > > > > definitely yes. But don't you think, optionally caching the > > zipped results > > > would be interesting? If you have a shop, that doesn't offer up to date > > > information about product availability (like I do), storing the results > > > would be a cool thing IMHO. You save many, many compress-calls > > per day and > > > caching about 10,000 items takes according to my first tests about 15M > > RAM. > > > I find this idea quite appealing, if your database selects are > > expensive. > > > > > > btw: Which compression function do you plan to use? > > > > > > > > > looking forward Jan > > > > > > p.s.: I have a quick'n'dirty prototype running, if you're interested... > > > > > > > > > > -----Original Message----- > > > > From: Adrian Hungate [mailto:adrian@haqa.co.uk] > > > > Sent: Donnerstag, 6. Dezember 2001 22:30 > > > > To: JanStiller T-Online; zope-dev(a)zope.org > > > > Subject: Re: [Zope-dev] RAMCacheManager and gzip > > > > > > > > > > > > Just to let you know, I am looking at patching ZServer using > > my PatchKit > > > > product to do "Content-Encoding: ..." stuff... anyone interested > > > > in testing > > > > when I get it running? > > > > > > > > (Probably a couple of days before I get a running version...) > > > > > > > > Adrian... > > > > > > > > -- > > > > Adrian Hungate > > > > > > > > All views expressed in this email are those of the whole world, > > > > however some > > > > people don't realise this yet. > > > > > > > > ----- Original Message ----- > > > > From: "JanStiller T-Online" <JanStiller(a)t-online.de> > > > > To: <zope-dev(a)zope.org> > > > > Sent: Wednesday, October 31, 2001 4:50 AM > > > > Subject: [Zope-dev] RAMCacheManager and gzip > > > > > > > > > > > > > Hi, > > > > > > > > > > Is it possible to marry the RAMCacheManager and gzip? > > > > > > > > > > I'm just working on a little shop and - for speed's sake - do > > > > 'ram-cache' > > > > > the article-listings and push all the Zope-Content through > > > > mod_gzip. With > > > > > this combination, I'm getting it 3x faster in Zope and 5x > > > > faster over the > > > > > wire (normally 64k). That's great so far. But I don't see > > any sense in > > > > > zipping cached content over and over, so I tried to zip the > > > > cached data in > > > > > RAMCacheManager. In my first attempt, i replaced: > > > > > > > > > > > > > > > ZCache_get (RAMCacheManager.py): > > > > > > > > > > entry.access_count = entry.access_count + 1 > > > > > return entry.data > > > > > > > > > > with: > > > > > > > > > > entry.access_count = entry.access_count + 1 > > > > > ob.REQUEST.RESPONSE.setHeader('Content-Encoding','compress') > > > > > return zlib.compress(entry.data) > > > > > > > > > > > > > > > The results: > > > > > > > > > > 1. > > > > > This works with Mozilla 0.96 only for very small entry.data. When I > > set > > > > > entry.data to 'XX', it works, but any real page gets > > > > > '<html><body></html></body>' as the result. > > > > > > > > > > 2. > > > > > This doesn't seem work in IE alltogether. (with > > > > 'Content-Encoding: gzip' - > > > > > zlib.compress != gzip!?) > > > > > > > > > > > > > > > I'm willing to spend some days of work for this feature, > > but although > > I > > > > have > > > > > 2 years experience in Python I'm quite new to Zope and > > > > Internet-RFCxxxx's > > > > > and this is definitely a point, where I would appreciate some hints > > from > > > > > more experienced Zope-programmers (or a product, of course -- or the > > > > > statement, that this won't be possible... ;o) > > > > > > > > > > > > > > > > > > > > regards > > > > > > > > > > Jan Stiller > > > > > > > > > > > > > > > P.S. sorry, if my words seems a bit confused, I'm more used > > to reading > > > > than > > > > > writing english... > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > > > > Zope-Dev maillist - Zope-Dev(a)zope.org > > > > > http://lists.zope.org/mailman/listinfo/zope-dev > > > > > ** No cross posts or HTML encoding! ** > > > > > (Related lists - > > > > > http://lists.zope.org/mailman/listinfo/zope-announce > > > > > http://lists.zope.org/mailman/listinfo/zope ) > > > > > > > > > > > > > > >

1 0

zope collector is broken
by Christopher N. Deckard 07 Dec '01

07 Dec '01

I tried submitting a new issue to the Zope collector at 11:42AM EST. Here's the traceback and error message: Zope Error Zope has encountered an error while publishing this resource. Error Type: WorkflowException Error Value: No workflow provides the "request" action. Troubleshooting Suggestions The URL may be incorrect. The parameters passed to this resource may be incorrect. A resource that this resource relies on may be encountering an error. For more detailed information about the error, please refer to the HTML source for this page. If the error persists please contact the site maintainer. Thank you for your patience. Traceback (innermost last): File /usr/local/Zope/Zope-2.4/lib/python/ZPublisher/Publish.py, line 223, in publish_module File /usr/local/Zope/Zope-2.4/lib/python/ZPublisher/Publish.py, line 187, in publish File /usr/local/Zope/Zope-2.4/lib/python/Zope/__init__.py, line 226, in zpublisher_exception_hook (Object: Zope) File /usr/local/Zope/Zope-2.4/lib/python/ZPublisher/Publish.py, line 171, in publish File /usr/local/Zope/Zope-2.4/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: collector_add_issue) File /usr/local/Zope/Zope-2.4/lib/python/ZPublisher/Publish.py, line 112, in call_object (Object: collector_add_issue) File /usr/local/Zope/Zope-2.4/lib/python/Shared/DC/Scripts/Bindings.py, line 324, in __call__ (Object: collector_add_issue) File /usr/local/Zope/Zope-2.4/lib/python/Shared/DC/Scripts/Bindings.py, line 354, in _bindAndExec (Object: collector_add_issue) File /home/Zope/guide/Products/CMFCore/FSPythonScript.py, line 187, in _exec (Object: collector_add_issue) (Info: ({'script': <FSPythonScript instance at 934af60>, 'context': <Collector 'Zope' (76 issues) at 0x8728698>, 'container': <CMFSite instance at 9199240>, 'traverse_subpath': []}, ('"mange_pasteObjects" does not look for proxy roles', ['', 'on'], 'cnd(a)purdue.edu', 'Zope', 'medium', 'bug', "Issue better defined at the following link. In short, when\nmanage_pasteObjects() is called from a proxied script in a context\nwhere the logged in user doesn't have ownership, manage_pasteObjects\nraises and Unauthorized exception.\n\nhttp://lists.zope.org/pipermail/zope/2001-December/105141.htm…", 'Zope Version (Zope 2.4.3 (source release, python 2.1, linux2), python 2.1.1, linux2) \r\nPython Version 2.1.1 (#1, Nov 23 2001, 21:55:33) [GCC 2.95.4 20011006 (Debian prerelease)] \r\nSystem Platform linux2 '), {}, None)) File Script (Python), line 17, in collector_add_issue File /usr/local/Zope/Collector/Products/CMFCollector/Collector.py, line 191, in add_issue (Object: Zope) File /usr/local/Zope/Collector/Products/CMFCollector/CollectorIssue.py, line 746, in addCollectorIssue (Object: Zope) File /usr/local/Zope/Collector/Products/CMFCollector/CollectorIssue.py, line 326, in do_action (Object: 77) File /home/Zope/guide/Products/CMFCore/WorkflowTool.py, line 506, in doActionFor (Object: portal_workflow) WorkflowException: (see above) -Chris -- -------------------------------------------------------------------- Christopher N. Deckard | Lead Web Systems Developer cnd(a)ecn.purdue.edu | Engineering Computer Network http://www.ecn.purdue.edu/ | Purdue University ---- zlib.decompress('x\234K\316Kq((-J)M\325KM)\005\000)"\005w') ---

2 1

IOError [Errno 5]
by Romain Slootmaekers 07 Dec '01

07 Dec '01

Yo dudes, our zope server shows the following behaviour. 1) runs fine for (a few hours - a few days) 2) then starts spawning the following errors: Error Type: IOError Error Value: [Errno 5] Input/output error Does anybody know what may be causing this? the tracebacks don't make sense at all but is fails somewhere in the ZODB/Connection setstate method. Tia, Sloot.

2 1

Open Letter to zope-dev
by James Johnson 07 Dec '01

07 Dec '01

<color><param>7F00,0000,0000</param>> It is the people who are in the trenches who are increasingly being > disaffected by Zope Corp, it seems as if you're not subscribed to > zope-dev, you have no voice, and for most people zope-dev is not an > appropriate forum for them to be subscribed to. <color><param>0100,0100,0100</param>This reminds me of the <FontFamily><param>Times New Roman</param><bigger>story, <underline>Swimmy</underline> by Leo Leonni http://www.fi.edu/fellows/fellow8/dec98/swimmy.html <FontFamily><param>Arial</param><smaller>. Do you know the story? Now that I've finished the Cisco Academy, I move on to my Oracle training. I plan on using Zope with Oracle, but I'm looking at the fact that oracle has a J2EE Web application server. Why am I using Zope with Oracle again? My impression has been that the elite private "school" zopers have created some fancy terms and algorithms, and have forgotton to be practical. Yeah, it's cool what I *think* Zope can do, and what it's gonna do in the future. Some say Zope needs 10x growth for next year, but I envision 80x growth. I see Zope hanging from the rearview mirror of some VW, or some kid using it, because it also comes with a cool clicker for his bicycle. I think the most beautiful thing next to my wife and kids has got to be Zope 2.5b1. I do enjoy those that sweat the details, but just the sight of Zope with Page templates and session tracking, with example folders that work on win98.... Please, I'm in heaven. I see Zope at 100x, because it should be a standard anyway. I know why I'm using it, and I happen to like the school of Zope fish. Now, go read <underline>Swimmy</underline> if you haven't. It's *Required* reading in the school I'm from. Peace, -James <nofill>

13 28