March 1999 - Zope-Dev

Will ZServer help me in this scenario?
by skip＠mojam.com 06 Mar '99

06 Mar '99

Glad to see this list has been created. I am currently using ZopeHTTPServer in the following architecture: Apache <-----> ZopeHTTPServer <-----> Musi-Cal DB HTTP XML-RPC Some requests from the net (searches) require a full round-trip to the database. Other things, like displaying an otherwise static DTML page, don't. The problem is that the ZopeHTTPServer is single-threaded at the moment, so if a search takes a long time (say, a couple seconds), simple stuff like calling a DTML template (which might execute in a tenth of a second), block. This, of course, causes Apache to get its britches in a twist and lots of its child processes start to pile up, memory gets short, yadda, yadda, yadda. I've had some trouble with deadlocks when trying to run ZHS multi-threaded (probably missed some object that needed to be locked). Will running ZServer with its asyncore/asynchat modules help alleviate this problem? Will I need to make changes to Fredrik Lundh's XML-RPC code to make this work? (XML-RPC is a given at this point. We're calling Musi-Cal's database from both Python and Perl at this point and it works great.) How stable is the ZServer alpha release? If it's mostly the FTP and PCGI support that is still experimental, I can live with that since I don't use either one. Thanks, Skip Montanaro | Mojam: "Uniting the World of Music" http://www.mojam.com/ skip(a)mojam.com | Musi-Cal: http://www.musi-cal.com/ 518-372-5583

2 1

TESTING TOOL & A ZOPE TEST
by Theodore Patrick 06 Mar '99

06 Mar '99

HEY DEV-ZOPE, I gather several of you out there know about this but for those who don't: <I AM A DEVELOPER & HAVE NOTHING TO DO WITH THIS PRODUCT - NO FLAMES PLEASE> We have been using this product for the last week and it has made a tremendous difference in the quality of the software we are developing with ZOPE and other web tools. As I though about it, many out there would benefit knowing about this tool, as I didn't until recently. </I AM A DEVELOPER & HAVE NOTHING TO DO WITH THIS PRODUCT - NO FLAMES PLEASE> WEB HAMMER 2.1 - Product Review It simulates web requests, LOTS OF THEM. You can configure it to do the following: Flood any URL with 40 Simultaneous connections(Get,Post,Head Requests) One Page of Settings - Easy to Use. - See attached File. **BE CAREFUL YOUR SYSTEM ADMINISTRATOR WILL FREAK OUT!!!!!** If you want to know when your server will crash, this will find that limit. We started using the application on several PERL and ZOPE apps in the office and we immediately found out where our apps were weak. In one case, one code change gave us 5X scalability on a simple script, we would have never known without the tool. We are also using it to test our form collection scripts and the database behind. Since you can do a POST we were validating middlewear and Database entries in no time, It is easy to check for errors with 1-2 Post try 40,000 --> You will find bugs. Try simulating 40,000 form submissions in 20 minutes and see where your database and server are. First time we ran the app it smoked everything (WebServer+Middlewear+Database(Oracle 8.0.5)). Nothing but Ashes!!! After some minor code changes we were handling the load fine. It costs some $85.00, but it lets you sleep at night knowing your system won't break, under the most strange & serious circumstances. Or it will tell you when you need to switch to bigger iron. You can get it at http://www.serverobjects.com/products.htm#WebHammer <http://www.serverobjects.com/products.htm#WebHammer> There is an evaluation copy that supports 2 threads per second, so you can try it for free. We have configured 2 Win boxes to simulate the loading on our servers. Each one does nothing but load test. If you are putting your reputation on the line, test to find you code's limits before your client does. It is $85 of insurance we would pay over and over. <ZOPE> We ran this tool against the Zope Server on some complex scripts, because of ZOPE's caching ability after the first couple of hits the server screamed! In our testing after ZOPE rendered the first page everything else seemed to come out of the cache. Zope also would thrash if the object: A. was not cached B. was a unique request C. involved a database connection. <INDIGO THRASH TEST> Thrash v. To cause a server to use memory and resources violently. If your system monitor goes 100%,2%,100%,15%,100% quickly you are Thrashing. As this is very OS dependent and very very code dependent. Server Hardware = Dell 410 DUAL P400 Raid0 2-9gb + 512RAM (PLENTY OF ZIP) TEST is similar code on identical Hardware 1. Linux-ZApache-ZOPE = "Min Thrash - No Death" --> It is very difficult to Kill. We ran 80 Threads across the box and killed the database but the middlewear and web server were still waiting for more requests. Quite a show for Linux/ZAP/ZOPE. Easy to code. 2. Linux-Apache-MODPERL = "Min Thrash - No Death" --> It is very very difficult to Kill. We ran 95 Threads across the box and killed the database but the middlewear and web server were still waiting for more requests. Quite a show for Linux/APACHE/MODPERL. Hard to code-requires experience. 3. NT-IIS-PERL= "MAX Thrash & Death" --> 40 Threads over 10 Minutes = IIS DEATH There is what looks to be a memory leak in IIS that causes a strange response in IIS4 sp3. It is similar to the server vomiting on itself. The memory requirement grows and then peaks and goes back to 10%, weird. We are working on tuning and some other strategies. This is really an IIS/NT problem it doesn't handle threads anywhere close to Linux/Apache. 4. NT-IIS-ZOPE= "MED Thrash & Death" --> 40 Threads over 20 Minutes = IIS DEATH SAME As PERL - There is what looks to be a memory leak in IIS that causes a strange response in IIS4 sp3. It is similar to the server vomiting on itself. The memory requirement grows and then peaks and goes back to 10%, weird. We are working on tuning and some other strategies. This is really an IIS/NT problem it doesn't handle threads anywhere close to Linux/Apache. We are working on Tuning IIS4 --> See http://www.microsoft.com/backstage/whitepaper.htm <http://www.microsoft.com/backstage/whitepaper.htm> ------------------------------------------------------------ </INDIGO THRASH TEST> <?>Is there a way to preload ZOPE's cache? Just a thought.</?> We also tested input from forms into a database from ZOPE, Although our results are specific to our implementation, typically the bottleneck is opening and closing the database connections. It caused allot of thrashing on both sides(middlewear & database). <DigCre>Hint, Hint ---> Any word on Aqueduct Integration for Zope?</DigCre> Overall the ZOPE process was as fast as any we have seen, ColdFusion and other middlewear packages. Considering the time we saved coding & configuration, Zope is worth its weight in gold. Zope also scales very well on very complex queries. <ZTABLES> Input from forms to ZTABLES is 40X-50X Faster(Tested!)than dealing with database connections. We are collecting the data into ZTABLES and then replicating it into Oracle during the wee hours. The design is faster than anything out there. ADD indexing or searching -->ZTABLES(Period!) Searching a database live in Zope takes a hit(!!!potentially every request is unique!!! = No Caching), but moving the data into the ZTABLES first makes the server sing. It is a performance boost that you have to see to believe. Why don't you guys charge for ZTABLES & AQUADUCT as components? I would not mind paying for it as I gather everyone else wouldn't. Considering the advantages are HUGE!!! </ZTABLES> </ZOPE> Just thought you ought to know! Theodore E. Patrick email: tpatrick(a)indigonetworks.com <mailto:tpatrick@indigonetworks.com> phone: 615-777-0070 ext.125 Indigo Networks - The Ultimate Choice In Who Sees You. http://www.indigonetworks.com <http://www.indigonetworks.com>

1 0

SQL methods q:
by Jordan B. Baker 05 Mar '99

05 Mar '99

Hi guys: I'm trying to write a SQL method though, and its giving me an error when I try to test it.. Error, exceptions.TypeError: keyword parameter redefined SQL used: Could not render the query template! The query I was using: SELECT searches.engineID as engine, queries.param as query, MAX(searches.timestamp) AS time, searches.position AS position, MIN(lastsearch.timestamp) AS lastupdate, lastsearch.position AS lastsearch, searches.URL as URL, 'NA' as change, engines.graphic as engine_gfx FROM searches LEFT JOIN searches as lastsearch USING (queryID,engineID), engines, queries WHERE lastsearch.timestamp BETWEEN  AND  AND searches.timestamp BETWEEN  AND  AND queries.clientID =  AND queries.queryID = searches.queryID AND engines.engineID = searches.engineID AND searches.timestamp != lastsearch.timestamp GROUP BY searches.engineID, searches.queryID ORDER BY searches.engineID ASC, searches.queryID ASC, searches.timestamp ASC, lastsearch.timestamp DESC ; Any ideas what I could be doing wrong? BTW, Zope is great! I've been kinda messing around with it for a while, also checking out PHP3 and other UNIX-based scripting languages and I've very impressed. TIA, -jordan.

1 0

Re: [Zope-dev] simple array-ish things in SQL/Zope
by Jeffrey P Shell 05 Mar '99

05 Mar '99

>From: Jeff Bauer <jbauer(a)rubic.com> > I do something similar with large text fields. Rather than > use PostgreSQL's unique (non-portable) array system, I store > collections of primitives as strings and use eval() when reading > the fields. > > >>> aList = [1, 'a', 2.5] > >>> field_value = repr(aList) > ... perform some database insert operation > ... read the field back in > >>> aList = eval(field_value) > > This works well for lists and dictionaries that contain other > lists, dictionaries, and primitive data types. It provides a > possible alternative to storing object data when you're > required to use a relational database system. Running eval on any external data source can be potentially risky. An option is to write pickles to the RDBMS. This allows storage of much more complex objects and it's harder for a third party with access to the database to write a potentially damaging pickle to a field than it is to write a potentially damaging expression. The downside is that storing a string repr of a python list or dict is still semi-searchable/readable through SQL for other operations. A stored pickle is not. >> [Julian>>] >> I am using PostgreSQL's array system for a "categories" field >> in a database, it return records in a format like: >> '{"foo"}' >> '{"foo","bar"}' >> '{"foo"}' >> >> I am trying to extract the list of unique categories - in >> this example: ["foo","bar"] >> >> This would be a complete nuisance to do in DTML. I have decided >> to read the SQL method's results into an external method and edit >> them there - but first I need to know how to call a SQL method >> without the "in" DTML tag. Do you have to store it in an array? If you can store the data in the categories field in some sort of delimited list (like comma delimited), you can use something like::  <H3></H3> <H4>Categories:</H4> <UL>  <LI></LI>  </UL> <HR NOSHADE>  Conversely, if you had a user enter data in a form with checkboxes or a SELECT list to choose categories, assuming that you use the Zope ":list" type on the input form, you can do the following in an SQL method:: INSERT INTO something (title, categories) VALUES ( , '' ) Another (and slightly better) way to do it would be to run the string.join command in the DTML Method/Document that calls the SQL method that does the insertion and pass it in as an argument. Modify the previous SQL code to look like the following:: INSERT INTO something (title, categories) VALUES ( ,  ) And use the following DTML to insert the data that came in from the form::  In essence, this is how I've used UserDB in some consulting projects that allowed the customer to add/edit users in the system. The column being used for the users roles is a comma delimited field, and the interface presented uses checkboxes to select which roles the user has. The roles are joined using the above, or split. An extra benefit of using this is that I can use regular SQL expressions to, say, list all of the users with role "Admin" by using SQL like the following: SELECT * FROM users WHERE user_roles LIKE '%%' There's a myriad of ways of approaching this situation. I'm no mega-SQL guru, but the above approach has worked fine for me in situations where I have a simple list of objects to store/retrieve from SQL. .jPS | jeffrey(a)digicool.com zope, through the web. www.zope.org

2 1

Re: [Zope] Using Zope with CORBA (new user question)
by Ross J. Reedstrom 05 Mar '99

05 Mar '99

Jeffrey P Shell wrote: > Crossing over from the Zope mailing list to zope-dev: > >>>>>> "Paul" == Paul Everitt <Paul(a)digicool.com> writes: > > > > Would it be worthwhile to start looking at and hacking ZServer? > > Yes. ZServer really restores Zope's original intent from the Bobo days: > publishing objects through a variety of protocols. I think ILU support was > once attempted back in those days. However, a Bobo app at that time could > only have one publishing mechanism (ie, CGI, PCGI, FastCGI, ILU, etc...). > ZServer allows for multiple publishing mechanisms in the same application, > allowing access to the objects through the web, through FTP, or through any > other protocol developed for ZServer/Medusa. > Hmm, I've been musing a bit about a zopefs module for linux. I couldn't code it, but I'm the Advisor for the (newly formed) Rice Linux Users Group, and some of those kids are very bright... Would Zserver be the place to start looking to hack something like that together? Or perhaps the ZClient thing I've seen mentioned... There was once something called userfs under linux, that allowed essentially arbitrary implementations of file system drivers in user space. One example that came with it was an ftpfs, so that, combined with Zserver's ftp support, gets us there. However, a cleaner implementation not involving FTP would be nice. Only problem - as far as I remember, userfs hasn't been updated since linux kernel 1.0 something (I think 1.2 broke it) Ross, (wanting to use all those crazy unix commands, like find/awk/sed/grep on his bobobase!) -- Ross J. Reedstrom, Ph.D., <reedstrm(a)rice.edu> NSBRI Research Scientist/Programmer Computer and Information Technology Institute Rice University, 6100 S. Main St., Houston, TX 77005

1 0

Properties in DTML Document
by Ze Octavio 05 Mar '99

05 Mar '99

Hello Zopians, I updated my version of Zope and have a nice surprise, the DTML Document ! Thanxs very much Zope's developers for this new fearture. When I saw that fearture I started to change my project, I had to migrate the properties from folders to new DTML Documents, ok, no problem. But I had one new problem. I have to call an lines property from a DTML Document but I am having an error and I already tried to fix that problem but was not sucessfull. Lets see the code (the part with problem):  <select name="myselect">   'Here is a Property from an atual folder 'Here I split a string from pr_days <OPTION VALUE="" 'Here is the problem SELECTED  >    </select>  pr_months is a lines Property in 2000.htm DTML Document but is not working. The error message is:  Somedoby have an idea ? Thanxs ! Ze Octavio

2 2

External method pulling data from an SQL method?
by julian＠zereau.net 05 Mar '99

05 Mar '99

I am using PostgreSQL's array system for a "categories" field in a database, it return records in a format like: '{"foo"}' '{"foo","bar"}' '{"foo"}' I am trying to extract the list of unique categories - in this example: ["foo","bar"] This would be a complete nuisance to do in DTML. I have decided to read the SQL method's results into an external method and edit them there - but first I need to know how to call a SQL method without the "in" DTML tag. Or maybe there is a simpler solution to all this, for any PostgreSQL gurus out there?? --- Julian Morrison Programmer (Zereau Ltd)

2 1

RE: [Zope-dev] External method pulling data from an SQL method?
by Michel Pelletier 05 Mar '99

05 Mar '99

> -----Original Message----- > From: julian(a)zereau.net [mailto:julian@zereau.net] > Sent: Friday, March 05, 1999 9:31 AM > To: zope-dev(a)zope.org > Subject: [Zope-dev] External method pulling data from an SQL method? > > > I am using PostgreSQL's array system for a "categories" field > in a database, it > return records in a format like: > '{"foo"}' > '{"foo","bar"}' > '{"foo"}' > > I am trying to extract the list of unique categories - in > this example: > ["foo","bar"] > > This would be a complete nuisance to do in DTML. I have > decided to read the SQL > method's results into an external method and edit them there > - but first I need > to know how to call a SQL method without the "in" DTML tag. > An SQL method is like any other method. If you want to call it from an external method, you could try this: <untested>  ... def myExternalMethod(self): self.mySQLMethod() </untested> This is just a sketch, but I hope you get the idea. You don't need the #in tag to call an SQL method, you can even call it with a #var tag, if you don't mind the output. -Michel > Or maybe there is a simpler solution to all this, for any > PostgreSQL gurus out > there?? > > --- > Julian Morrison > Programmer (Zereau Ltd) > > _______________________________________________ > Zope-Dev maillist - Zope-Dev(a)zope.org > http://www.zope.org/mailman/listinfo/zope-dev >

1 0

RE: [Zope-dev] Programmatically accessing Z SQL Methods
by Paul Everitt 05 Mar '99

05 Mar '99

Mike wrote: > myself having trouble getting started. Here's a deal: If I > get a boost up the > learning curve with a few answers, I'll put it in a How-To > once I understand > it all. Instead of answering the question (which with my IQ would be something like, "use a crescent wrench"), I'll instead suggest a counter-deal with the ZDP crowd. As you guys start moving over the ground of filling in the gaping holes in the Zope docs, you'll come up with things you can answer and things you can't. The lack of a Product API reference doc is a perfect example. I propose that you guys go ahead an collect the questions and then once a week I'll get some time from the folks here to answer the questions. Thoughts? --Paul

2 1

Mea Culpa!
by Tres Seaver 05 Mar '99

05 Mar '99

Aiyah! <sound of gnashing teeth> I got too quick on the trigger, and sent my reply to Joe VanAndel's post without editing the digest title. Tres.

1 0