March 2001 - Zope-Dev

Re: [Zope-dev] (ZPatterns) Speeding up Skinscripts
by John Eikenberry 30 Mar '01

30 Mar '01

Phillip J. Eby wrote: > Congratulations, you're the first person (that I know of, anyway) to hit a > wall here. However, I think you may be barking up the wrong tree on your > profiling. See below. What a dubious honor. > SkinScript expressions execute using Zope security checking. If you're > using a "WITH QUERY SQLmethod() COMPUTE attributelist" statement, security > checks are applied to both the query expression and to every name > referenced in 'attributelist'. So the longer the attribute list, the > slower the execution. Although, now that I think of it, eval() is not used > if you use a pure attribute name list like "name1,name2,name3", so the > problem probably isn't on the attribute list side. This makes it much less > likely that the security machinery is to blame (unless you have a very slow > user folder which also doesn't cache well, and you haven't given your > SkinScript proxy roles to shortcut the security checks.) Yeah... I just added proxy roles to all the skinscripts and it didn't make that much of a difference (maybe a small one though). > Now, for your SQL query... is it by any chance dynamically generated in a > complex way? I mean, more complex than an sqlvar or two? Keep in mind > that when Zope caches SQL queries it does it by generating the text of the > query and using it as a cache key. That means any DTML in your query will > be executed each time. It'd have to be pretty complex, or else there'd > need to be lots of slow security lookups, to get the kind of poor > performance you're seeing, though. In the sql method in question has 1 dtml-sqlvar in it. Everything else is pre-rendered (there's a method that generates the sql methods). > 1. If you're not using LoginManager as your user folder, and maybe even if > you are, your SkinScripts should have proxy roles. Proxy roles can slash > the overhead of performing security checks, and are usually appropriate for > SkinScripts because SkinScript is "internal" to the object and Zope > security will apply to the results of the SkinScripts as well. Most of my tests have been with the builtin user folder. But as I mentioned above, proxy roles don't help much. > 2. Why are you retrieving 200 dataskins? Is this all in one transaction? > If so, you may have a design issue. Multi-object operations should > generally be implemented as domain-specific methods on the Specialist, so > they can be implementation-specific (and thereby take advantage of speedups > like a ZCatalog index or specialized SQL queries). In other words, you > probably want to iterate directly over a 200-row query result from an > SQLMethod, rather than retrieiving 200 DataSkins. (You're accessing over > 10 times as many DataSkins in one transaction as our most complex ZPatterns > application touches in its most complex type of transaction, which involves > about 4 or 5 Specialists at once!) Pretty much, if you're displaying a > list or report or doing some kind of summary analysis or mass update, it > belongs in an implementation-specific method in the Specialist. Yeah... this is mainly a design flaw. I and others at my company were working on various parts of this project in parallel. I was working on the base level ZPatterns model. They were working on client apps. We missed the communication bus at some point and they got to thinking of Specialists as DB tables. Something that needs to be addressed in the future, but it doesn't help me now. :( > 3. If you *really* need 200 dataskins in one transaction, and the overhead > is *really* in SkinScript, then you can bypass the overhead by writing an > AttributeProvider of your own in Python. I'm guessing, however, that > SkinScript per se is not the source of the slowdown, and you need to look > more closely at the things the SkinScript is calling. I think I'll have the developer just use a SQL method to get the data. This is how it should be done anyways... Thanks for the reply. -- John Eikenberry [jae(a)kavi.com] ______________________________________________________________ "A society that will trade a little liberty for a little order will deserve neither and lose both." --B. Franklin

1 0

(ZPatterns) Speeding up Skinscripts
by John Eikenberry 30 Mar '01

30 Mar '01

We have a fairly large and complex app framework built on ZPatterns. It uses MySQL for storage and the standard Specialist/Rack/DataSkin setup with skinscripts for attributes and triggers. We've found that the speed of getItem is a bit slower than we need. For instance retrieving 200 dataskins takes about 8 seconds on a P2-300. After profiling and digging around I think I've found the primary bottleneck. Its the running of eval() on the skinscript's python expression (stored in the Compute class as _fromex and Triggers as callexpr). Note that this becomes the bottleneck after the SQL Method gets cached. The query to the DB takes the most time on the first hit, but after its been cached it takes very little time. The optimization I've been looking at is changing the code from storing a string and eval()ing the string to using compile() at save time and exec() when evaluated. Profiling these 2 ways in little test programs seems to indicate about a 2.5x speedup. Not huge, but combined with better hardware should be enough. But I'm curious why this avenue wasn't taken to begin with. Seems like the way to do it to me. Am I missing something? -- John Eikenberry [jae(a)kavi.com] ______________________________________________________________ "A society that will trade a little liberty for a little order will deserve neither and lose both." --B. Franklin

4 4

Re: [ZWeb] 'All Hotfixes' page (was Re: [Zope-dev] Problem with Hotfix 2000-10-11 on 2.1.6)
by Martijn Pieters 29 Mar '01

29 Mar '01

On Thu, Mar 29, 2001 at 01:27:08PM -0500, Shane Hathaway wrote: > > Have a look at http://www.zope.org/Products/Zope/hotfixes, I think it is > > exactly what you need. > > Huh? You repeated the URL he supplied... and I'd like to know if there > is indeed a better URL. Duh. Copy, paste, send. Who cares about editing the URL..... :0 The correct URL is: http://www.zope.org/Products/Zope -- Martijn Pieters | Software Engineer mailto:mj@digicool.com | Digital Creations http://www.digicool.com/ | Creators of Zope http://www.zope.org/ ---------------------------------------------

2 1

Re: [Zope-dev] ZCatalog phrase indexing revisited
by Dieter Maurer 29 Mar '01

29 Mar '01

Oren Yosifon writes: > ... phrase searches ... > Can anyone help me ? URL:http://www.dieter.handshake.de/pyprojects/zope/near.pat is a patch that made near searches work ages ago for me. The patch is old (against Zope 2.1.6). As todays ZCatalog is quite different, it will not be easy to apply the patch to modern code. You only have a chance, when you are quite familiar with Python. Dieter

1 0

Re: [Zope-dev] Zope/OpenBSD: Too many open files
by Dieter Maurer 29 Mar '01

29 Mar '01

Ricardo Bermell writes: > Zope (started from a shell) in OpenBSD gets locked > too easyly because of the low default limit for > open file descriptors (64). Most operating systems allow you to increase this number (either at runtime or through a kernel recompilation). Dieter

3 2

ANN: Proposal for ZCatalog "drop-in" indexes
by Chris McDonough 29 Mar '01

29 Mar '01

Hi all, I'd like comments on this proposal if you've got the time: http://dev.zope.org/Wikis/DevSite/Proposals/DropInIndexes It deals with the implementation of "drop-in" ZCatalog indexes. Thanks! - C

1 0

Re: [Zope-dev] Problem with Hotfix 2000-10-11 on 2.1.6
by Gregor Hoffleit 29 Mar '01

29 Mar '01

On Thu, Mar 29, 2001 at 10:10:50PM +1000, Zac Stevens wrote: > Howdy, > > On Thu, Mar 29, 2001 at 01:28:13PM +0200, Gregor Hoffleit wrote: > | I'm having trouble installing Hotfix_2000-10-11 on Zope 2.1.6, although the > | README says: "The hotfix will work for all versions of Zope 2.2.0 and > | higher." > > I think you're just making a minor braino here - 2.1.6 isn't higher than > 2.2.0 > > The Readme is somewhat ambiguous though - it starts off stating that "all > versions up to and including 2.2.2" are affected, but then recommends "Zope > 2.2.x" sites upgrade. Perhaps it isn't even a genuine issue for 2.1.6? > (I don't know) Ooops, sorry, sorry, sorry. I had read the start of the README (like you wrote: 'Zope versions up to and including Zope 2.2.2.'), but I had quoted and paragraph that indeed implied an answer to my question ('will work for all versions of Zope 2.2.0 and higher.'). So 2.1.6 ought to be vulnerable as well, but that Hotfix won't work for it. Gregor

2 2

Re: Zope on OpenBSD 2.8 makes me cry
by Alexander Limi 29 Mar '01

29 Mar '01

andrew fabbro afabbro(a)indoctrination.com wrote: > I'm pulling out my hair trying to get Zope working on OpenBSD 2.8. I I'll repeat my original post, and the 2.7 libc can be found at http://mp3.no/openbsd/libc_r.so.3.0 (not indefinitely, though) I hope the file is OK, I have a shitty connection at the moment, and I haven't got the time to verify it right now. (from http://lists.zope.org/pipermail/zope-dev/2001-January/009137.html) For those of you wishing to run Zope on OpenBSD 2.8, here is a solution. The reason Zope doesn't work in the first place is a thread bug in the OpenBSD 2.8 release version. The solution is to use the libc_r from OpenBSD 2.7 for the Zope instance. Thanks to Jason Ish for the howto, I thought I would rewrite it to be more generic and post it to the lists. So, here we go: Python on OpenBSD 2.8 uses /usr/lib/libc_r.so.3.1. libc_r.so.3.1 is from 2.8, libc_r.so.3.0 is from 2.7 and will work. So what we do is to copy the old libc file (if you don't have it, mail me and I'll send it to you) into a directory in the Zope install dir, and add that to the path. Then we rename the old file so the system thinks it is the curent version. In this example, I assume your Zope install is in /usr/local/zope. First, we copy the old library into our Zope dir, and rename it: cp /usr/lib/libc_r.so.3.0 /usr/local/zope/libc_r.so.3.1 Then you should add something like this to the top of your "start" script: export LD_LIBRARY_PATH=/usr/local/zope Start Zope, and you will find that it works fine now. Zope also works fine in the CURRENT tree of OpenBSD, so presumably it will work out-of-the-box on OpenBSD 2.9. This is the safest way of getting it to work at the moment. Happy OpenBSD'ing :) -- Alexander Limi alexander(a)limi.net

1 0

Re: recent CVS update: dc/pyexpat, also: cPersistence stuff
by Jens Quade 28 Mar '01

28 Mar '01

Steve Spicklemire <steve(a)spvi.com> writes: > Hmm.. anybody seen this? Yes, but it's already gone... See http://lists.zope.org/pipermail/zope-checkins/2001-March/date.html for details. Anyway, I had to remove lib/python/Shared/DC/xml/pyexpat/Setup manually to build dcpyexpat.o, which was renamed last night. jens -- Shipping software is an unnatural act

1 0

Re: [Zope-dev] Memory problems with ParsedXML ?
by Dieter Maurer 28 Mar '01

28 Mar '01

Godefroid Chapelle writes: > I have imported about 2500 xml files in ParsedXML objects through the > use of : > ... > But when trying to browse the folder containing the ParsedXML instances, > a whole lot of memory is used by Zope and it took my machine down. Are you sure, you want to browse 2,500 objects? Zope must load all objects, build a really big HTML page transfer it to the user's browser which will have a hard time to render this monster. I am sure, you want to organize your objects in a sensible way.... Dieter

1 0