- Zope-Dev - Zope lists

RE: [Zope] xmlrpc slowness
by Albert Langer 21 May '01

21 May '01

[Shilad] The new release is up on sourceforge. It should be compatible with the Zope client/server it was tested against. It is at: http://www.sourceforge.net/projects/py-xmlrpc Let me know how it goes. I'm curious to see what kind of speed increase you see. My guess is that the implementation at the other xmlrpc end will be the bottle-neck pretty soon. [Albert] Thanks *very* much for this!!! 1000 xml-rpc calls per second with commodity hardware sounds pretty attractive! I've had to just pass it on to a friend to checkout at the moment, but will try to get back to you soon with speed results, as I'm sure the other 3 CCs will. I've added the Zope list back to the addresses as there may be others there interested in trying it out now that Phil Harris has confirmed the new version interworks correctly with the Zope implementation. Also added [Zope-dev] list as Zope developers should be more interested and Matt Hamilton as issues below may be closely related to his posting "[Zope-dev] Asyncore in an external method". http://lists.zope.org/pipermail/zope-dev/2001-May/011274.html In a previous email CC you said: "I'm really excited that zope people may be using this. Let me know if you have any questions/concerns/requests." As you've also done binary releases for Windows and the various unixen that Zope does releases for, it should be suitable for actually integrating into Zope rather than just as an add-on "Product" or it's present form as just a separate python process talking to Zope rather than implementing Zope's xml-rpc itself. So here goes with some questions/concerns/requests... In the README you mention: "Some time I should document everything better, especially the nonblocking aspects of the server and client." and " * Non-Blocking IO: This becomes important when you have critical applications (especially servers) that CANNOT be held up because a client is trickling a request in or is reading a response at 1 byte per second. * Event based: This goes hand in hand with non-blocking IO. It lets you use an event based model for your client/server application, where the main body is basically the select loop on the file descriptors you are interested in. The work is done by registering callbacks for different events you received on file descriptors. This can be very nice. There is some work to be done here, but all the hooks are in place. See TO DO for a list of the features that have yet to be incorporated." (BTW there is currently no "TO DO" file.) These aspects are a very big advantage. For Zope as a client, I suspect that the trickling issue may also be very important since it could be blocking an expensive Zope thread while waiting for a response from a slow remote server to pass on some information in response to relayed request (ie http/xmlrpc relay mode or even straight xmlrpc/xmlrpc proxy mode). As you also mentioned, "the implementation at the other xmlrpc end will be the bottle-neck pretty soon". Sorry I'm not very familiar with how to do this stuff myself, so I have 3 questions which maybe you could answer when you get around to doing those docs (or even better by also providing implementations in the next release ;-) or perhaps someone else on CC list knows the answers or is planning to do something about it? 1) I'm not sure if I've got it right, but my understanding is that despite being based on a Medusa Reactor design that can handle many web hits with a single thread, Zope also maintains a small pool of threads to allow for (brief) blocking on calls to the file system for ZODB and for (also brief) external connections. I suspect these threads are "expensive" because they each end up keeping separate copies of cached web pages etc (to avoid slow python thread switching). So simply increasing the number of such threads is not recommended for improving Zope performance - performance relies on the non-blocking async Medusa Reactor design of Zserver, not on the threading, which just seems to be a sort of extra workaround. If that is correct, then a few concurrent external calls to slow external xmlrpc servers (eg for credit card authorization taking 30 seconds or more) could easily tie up a lot of Zope resources. The non-blocking py-xmlrpc client could presumably surrender it's turn in the main event loop for it's thread until a response is received and then be woken up by the response, thus improving things greatly. Unfortunately I have no idea how to do this - whether it would just happen automatically or there are built in facilities for doing that sort of thing easily in Zope already, or whether it is difficult to do. I am just guessing that there would be some special tricks needed to wake up a channel when a response comes back (eg using the stuff in Zserver/medusa/select_trigger.py and Zserver/PubCore/. which I don't fully understand, but looks relevant). Maybe I have misunderstood, but it looks to me like existing use of xmlrpc clients *from* Zope to external servers may not take this into account, but just block a thread. So even though the *server* side of xmlrpc used to "publish" zope objects is non-blocking with Zserver/Medusa, it may be just blocking a Zope thread when either an xmlrpc or http web "hit" results in a blocking xmlrpc call to an external server *from* Zope as a client. Some difficulty handling a separate set of socket selects for client calls to external servers may be the reason for this, and may relate to Matt Hamilton's problem. That might not show up in simple testing of the server side, but could become a serious bottleneck with a heavy load of such external calls arising from multiple web hits concurrently when Zope is acting as a proxy/relay to a slow external server rather than just responding itself. eg with 10 Zope threads and 30 second external delays you might end up with every thread blocked when a hit arrives every 3 seconds if only the server side of Zope is non-blocking but the client side blocks. That would be a shame if the server alone could handle 1000 calls per second, which might translate to 500 relayed or proxied calls per second, (regardless of the external delay of 30 seconds) if both sides were non-blocking (ok 30 x 500 = 15,000 micro-thread "channels" is pushing it even though they are just dormant waiting for a response most of the time - but references from the Medusa site indicate that sort of thing is at least *theoretically* possible ;-) http://www.kegel.com/c10k.html At any rate, a *lot* more than 1 every 3 seconds should be feasible, without wasting ridiculous amounts of RAM for per thread Zope caches. (It would still be less than 4 per second instead of 1000 per second even if the external delay was only 3 seconds - the critical thing is whether a non-blocking client can be integrated with Zope, taking into account the multi-threading as well as async Reactor design of Zope). Any comments? 2) Medusa has an async non-blocking "pure python" xmlrpc handler using asyncore - based on "xmlrpcserver.py" by Fredrik Lundh (www.pythonware.com) http://www.nightmare.com/medusa/ medusa-20010416.tgz//xmlrpc_handler.py As far as I can make out this seems to be for the server side only, although there are also both an rpc_server.py *and* an rpc_client.py (not using xmlrpc), so I imagine it would be easy enough to implement a client and a proxy relay for Medusa as well. The examples in the tutorial include an http client and an http proxy relay with both server and client: http://www.nightmare.com/medusa/programming.html I imagine it is quite straight forward for anyone who knows what they are doing to somehow tie the faster py-xmlrpc C version into the Medusa main event loop even though it has it's own set of sockets to select from. Unfortunately I don't know how to do this. It might be generally useful to contribute the necessary wrapper and/or installation instructions to Medusa as well as add them to py-xmlrpc itself to use it for both server and client channels with Medusa, to encourage faster takeup, and as starting point for stuff below without having to get into Zope internals. That would also provide a base reference point for speed tests of what Zope should be capable of, using py-xmlrpc for client as well as server side of plain Medusa with such proxied calls - to compare with tests doing nothing special on Zope for the Zope client side (with various numbers of 1 or more Zope threads and various delays at the remote server). Any chance of this? When? ;-) 3) As far as I can see Zope comes with the following from the top of the src tree: lib/python/xmlrpclib.py lib/python/ZPublisher/xmlrpc.py But nothing in Zserver despite that containing and being based on Medusa. So it is extending the Medusa implementation of xmlrpclib rather than just layering above it. Is anyone planning to provide a replacement based on py-xmlrpc? If so, when - and will it include a non-blocking client side as described above? I've tried to track down some documentation on how XML-RPC fits into Zope. Here's the list I came up with in case it helps or prompts somebody to point out docs or threads with other info. The start of some documentation on how Zope publishes with xml-rpc server is: http://www.zope.org/Members/teyc/howtoXMLRPC2 Others include: http://www.zope.org/Members/Amos/XML-RPC http://www.zope.org/Members/teyc/pipermailXMLRPCWoes http://www.zope.org/Members/teyc/howxmlrpc There is also an XMLRPCProxy: http://www.zope.org/Members/dshaw/XMLRPCProxy And quite a few other references from the search button on www.zope.org for both "xmlrpc" and xml-rpc". Anyway, thanks again for 1000 calls per second!

2 1

Woah! Weirdy Catalog Error :-S
by Chris Withers 21 May '01

21 May '01

Automatic Zope Response wrote: > > Zope reported an error for: > Error Type: TextIndex.QueryError > > Error Value: Malformed query > > Traceback: > Traceback (innermost last): > File /usr/local/zope/2.3.2_base/lib/python/ZPublisher/Publish.py, line 171, in publish > File /usr/local/zope/2.3.2_base/lib/python/ZPublisher/mapply.py, line 160, in mapply > (Object: search) > File /usr/local/zope/2.3.2_base/lib/python/ZPublisher/Publish.py, line 112, in call_object > (Object: search) > File /usr/local/hosting/accounts/swishdot/Products/Squishdot/SquishSite.py, line 798, in search > (Object: Traversable) > File /usr/local/hosting/accounts/swishdot/Products/Squishdot/SquishSite.py, line 903, in __call__ > (Object: Traversable) > File /usr/local/zope/2.3.2_base/lib/python/Products/ZCatalog/ZCatalog.py, line 535, in searchResults > (Object: Traversable) > File /usr/local/zope/2.3.2_base/lib/python/Products/ZCatalog/Catalog.py, line 651, in searchResults > File /usr/local/zope/2.3.2_base/lib/python/Products/ZCatalog/Catalog.py, line 542, in _indexedSearch > File /usr/local/zope/2.3.2_base/lib/python/SearchIndex/UnTextIndex.py, line 513, in _apply_index > File /usr/local/zope/2.3.2_base/lib/python/SearchIndex/UnTextIndex.py, line 576, in query > File /usr/local/zope/2.3.2_base/lib/python/SearchIndex/UnTextIndex.py, line 640, in evaluate > File /usr/local/zope/2.3.2_base/lib/python/SearchIndex/UnTextIndex.py, line 601, in get_operands > File /usr/local/zope/2.3.2_base/lib/python/SearchIndex/UnTextIndex.py, line 653, in evaluate > TextIndex.QueryError: Malformed query > ---------------------- > REQUEST object dump: > ---------------------- > SERVER_SOFTWARE => Zope/Zope 2.3.1 (source release, python 1.5.2, linux2) ZServer/1.1b1 Huh? Why does that say 2.3.1 when the paths above suggest Zope 2.3.2?! > URL => http://www.squishdot.org/search > author => > batch_size => 20 > body => Error Value: frexp() result out of range > op => comments > reviewed => 1 > subject => > textToSearch => Error Value: frexp() result out of range You can reproduce this by going to Squishdot.org and searching for "Error Value: frexp() result out of range". Please don't do it too many times as I get an email for each one. If you do, I'll hunt down your IP address and kill you ;-) Any ideas? cheers, Chris

2 3

ZSQL Methods in Z Classes
by Shai Berger 21 May '01

21 May '01

Hi Guys, I wanted to use ZSQL Methods in Z Classes to share code between different parts of a large project. The different parts need to use similar, but separate, database namespaces, but I still wanted to use the same methods. Now, ZSQL Methods treat their Database connection ID, for some reason, on a par with their source -- it is not a parameter. You are not supposed to run one ZSQL Method on two connections. (One can try to acquire, restricting yourself to the same ID name in different folders; I think this will not work, and the ZSQL methods will get to their original DB connection first). I tried to bypass this limitation, by defining a forwarding database adapter. The idea was that this adapter's connection string would point to a resource (property or method) which will specify the actual connection. The idea was that in a ZClass, I could have this be a pythonscript which will look into an instance property sheet, thereby allowing different instances to use different connections. This seemed to work very nicely at first: After the ZClass with the forwarding adapter and pythonscript was installed into a folder, all its methods worked like magic. I was very happy. Then we restarted Zope, and ever since, it seems like the Adapter is being called by the ZSQL Methods without any acquisiotion context. Could you help show me what went wrong? BTW: I intended to make the forwarding DA a product, and to ship with it a base ZClass for ZSQL methods-using ZClasses, which I wanted to call "ZSQL Class". But even before the current problems, I was halted by the fact that if you define a database connection in a ZClass, ZSQL Methods in derived classes cannot use it. In fact, even ZSQL Methods in the same class as the adapter act funny -- you can set it as their connection, but in the source they say "connection not found", and you can't test them in the class context (even when they don't rely on instance resources). Hope this is informative enough, and that others have thought of these problems, Shai.

1 1

Re: [Zope-dev] Quick Persistence Question
by Chris Withers 21 May '01

21 May '01

Casey Duncan wrote: > > Chris Withers wrote: > > > > if self.data is a BTree and I do: > > > > obj = self.data[id] > > obj.revsub = obj.revsub + 1 > > AFAIK _p_changed is only needed when you change a mutable attribute you > want stored like adding a dict key or appending to a list. revsub looks > like an int which is not mutable. The persistance machinery should pick > up on this change automagically. Thanks, I had thought that maybe changing obj (which is stored in a dictionary-like thing) might have an effect that needed self._p_changed, but I found stupid programmer error was actually the cause *grinz* Chris

1 0

Fwd: [Zope] ZCatalog + Directory product question
by J�lio Dinis Silva 21 May '01

21 May '01

Any hints? Best Regards, Julio Dinis Silva >From: "Júlio Dinis Silva" <juliodinis(a)hotmail.com> >To: zope(a)zope.org >Subject: [Zope] ZCatalog + Directory product question >Date: Tue, 15 May 2001 18:29:37 +0100 > >Hi, > >supose you have a product with two classes: > >MyCategory > | > ---------- MyLink > >Both classes are CatalogAware. >The ideia of this classes is to implement >a Directory like yahoo (think about the Yiahw product). > >Now I start creating my Directory. >My classes have a property "active" that I use >to activate and deactivate some parts of my Directory. > >Now I want to search in my Catalog in which I have MyCategory and >MyLink instances catalogued. > >When I do a search I want to retrieve only results with the >"active" property "on". This is ease no big deal. But I want to guarante >that all PARENTS of a given Mylink are all with active property "on". Yes >no >big deal, for each result from my search I >run trough the getPath(data_record_id) or even PARENTS and check for each >Parent if they are all with active==on. > >Well the problem arises when you have 30 results in your search and >all are Mylinks in a directory depth path of 10 category... The search >becomes to heavy,i.e, scalability is compromised. > >I'm thinking I could use the zope security engine to do the active/deactive >trick, i.e, when I deactivate one MyCategory I uncheck some security entry, >then listing my directory I do a dtml-in with skip-unauthorized, and when I >do a search I try to getObject MyLink's >results as a anonymous role and then skip_unauthorized. > >I think this skip_unauthorized could be the solution for this >ZCatalog+Directory product scalability, because instead of running >trough the PARENTS and check a property, I will be using zope's >internal security mecanism. > >Any hints, >Júlio Dinis Silva >_________________________________________________________________________ >Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com. > > >_______________________________________________ >Zope maillist - Zope(a)zope.org >http://lists.zope.org/mailman/listinfo/zope >** No cross posts or HTML encoding! ** >(Related lists - >http://lists.zope.org/mailman/listinfo/zope-announce >http://lists.zope.org/mailman/listinfo/zope-dev ) _________________________________________________________________________ Get Your Private, Free E-mail from MSN Hotmail at http://www.hotmail.com.

3 2

Re: [Zope] ZCatalog + Directory product question
by Tres Seaver 21 May '01

21 May '01

"R. David Murray " <bitz(a)bitdance.com> wrote > > On Thu, 17 May 2001, Casey Duncan wrote: > >> You should create an instance method (It will need to be a Python >> script, DTML methods cannot be indexed)to return the result of the value > > > To my understanding, this is not true. DTML Methods *can* be indexed, > they just can't make use of REQUEST in whatever processing they do. > However, I can't think of a good reason to use a DTML Method as a > Catalog index. The catalog doesn't do any of ZPublisher's "argument sniffing"; if the attribute corresponding to the index is callable, it calls it with no arguments. DTMLMethods *must* be called with at least the client and namespace arguments; they therefore cannot be the target of an '_apply_index' call. Tres. -- =============================================================== Tres Seaver tseaver(a)digicool.com Digital Creations "Zope Dealers" http://www.zope.org

1 0

Quick Persistence Question
by Chris Withers 21 May '01

21 May '01

if self.data is a BTree and I do: obj = self.data[id] obj.revsub = obj.revsub + 1 obj's class subclasses Persistant, as does self's Do I need to let the persistence mechanism know something has changed? If so, do I do: obj._p_changed=1 ...or... self._p_changed=1 (which feels like it's gonna store much more data than it needs to...) ...or something else? cheers, Chris

1 0

ZCatalog/TextIndex: searching for the exact phrase "word1 word2"
by Erik Enge 21 May '01

21 May '01

Hiya, it basically says it in the subject. How can I search for "word1 word2" without ZCatalog/TextIndex interpreting it as "word1" or/and "word2"?

2 7

MySQL (Zope) Problem on Sun Solaris machines ?
by Holger Lehmann 21 May '01

21 May '01

Hi Andy, and all ! I was developing a little webapp under Zope. I was using the following setup: Linux 2.4.x Zope 2.3.0 MySQL 3.22.32 ZMySQLDA-2.0.6 MySQL-python-0.3.5 Everything went smooth that way. Small SQL queries and complex SQL queries, queries on empty tables and on full ones, they all went well. Now I exported the website and dumped the tables structure. I the impoerted the website and created the tables on a: Sun Solaris (don't ask me what release) - Sparc processor. Zope 2.3.0 MySQL 3.23.32 (or 3.23.33, I am not sure) ZMySQLDA-2.0.6 MySQL-python-0.3.5 And this happened: Querying MySQL like this: -- show tables; -- or -- select * from data; -- all went ok. But running this statement on *empty* tables (mark you, just no data) left no python process running, they all died: -- select a.a b.b c.c from a b c where a.fk_b = b.pk_b and a.fk_c = c.pk_c order by a.a; -- It looks to me there is an uncaught "NULL pointer exception" deep down somewhere in the code.... Anyone hints or solutions ? I am going to put some valid data into the database and try to run the queries again. I have not done this yet. - Thanks in advance, Holger -- ----------------------------------- catWorkX GmbH Dipl.-Ing. Holger Lehmann Stresemannstr. 364 22761 Hamburg Tel: +49 (0700) catWorkX Tel: +49 (40) 890 646-0 Fax: +49 (40) 890 646-66 mailto:lehmann@catworkx.de http://www.catworkx.de http://www.catbridge.de

1 0

another Zope hanging..
by cs＠comlounge.net 21 May '01

21 May '01

Hi! I also now have some problems with a hanging zope. I got something like this in my stupid logfile: 2001-05-03T13:44:39 ERROR(200) ZServer uncaptured python exception, closing channel <select-trigger (pipe) at 81d64c8> (exceptions.OSError:[Errno 11] Die Ressource ist zur Zeit nicht verf?gbar [/opt/Zope-2.3.1-src/ZServer/medusa/asyncore.py|poll|83] [/opt/Zope-2.3.1-src/ZServer/medusa/asyncore.py|handle_read_event|335] [/opt/Zope-2.3.1-src/ZServer/medusa/select_trigger.py|handle_read|77] [/opt/Zope-2.3.1-src/ZServer/medusa/asyncore.py|recv|287] [/opt/Zope-2.3.1-src/ZServer/medusa/asyncore.py|recv|475]) Now my question is simply if I can catch this exception somewhere and get some traceback etc. Actually I am quite confused from the inner workings of medusa so I dunno where to start searching. Can anyone help? (I know about the -M logging but unfortunately this does not help that much..) cheers, Christian -- COM.lounge http://comlounge.net/ communication & design info(a)comlounge.net

4 20