remote procedure calls to manage functions
Have been delving into ZPublisher.Client to make remote procedure calls under Zope 2.2.1. Many of the manage_... functions work just fine through this interface (once the obvious bug on line 212 of Client.py is fixed, anyway ;-). Check out the example utilities/load_site.py if you want to get started in this very productive use of Zope. However, manage_addZSQLMethod refuses to work, always returning a 404 not found error. Don't have time at the moment to find out why, but I wondered if there was a simple reason for it (nothing seems simple at 3am does it)? In the meantime, calling it via an external method to act as glue is working fine.
andrew wrote:
Have been delving into ZPublisher.Client to make remote procedure calls under Zope 2.2.1.
Did someone slip 2.2.1 without mentioning it?! I thought XML-RPC was now favoured over ZClient? cheers, Chris
Chris Withers wrote:
andrew wrote:
Have been delving into ZPublisher.Client to make remote procedure calls under Zope 2.2.1.
Did someone slip 2.2.1 without mentioning it?!
I thought XML-RPC was now favoured over ZClient?
It's not really favored, both are quite useful. xml-rpc is more for when you want two different system to interoperate. ZClient is very zope specific and probably gives you a bit more functionality than xml-rpc (because it has the 'remote object' abstraction). Also, ZClient is much faster i've found, probably due to the marshalling/unmarshalling necessary for xml-rpc. -Michel
Michel Pelletier wrote:
Chris Withers wrote:
andrew wrote:
Have been delving into ZPublisher.Client to make remote procedure calls under Zope 2.2.1.
Did someone slip 2.2.1 without mentioning it?!
I thought XML-RPC was now favoured over ZClient?
It's not really favored, both are quite useful. xml-rpc is more for when you want two different system to interoperate. ZClient is very zope specific and probably gives you a bit more functionality than xml-rpc (because it has the 'remote object' abstraction).
I don't think ZClient is really Zope specific. At least it's not supposed to be. It does have a richer API, including features like authentication and header support.
Also, ZClient is much faster i've found, probably due to the marshalling/unmarshalling necessary for xml-rpc.
That's interesting......Hm. ZClient has to marshal. I suspect that xml-rpc wants some sort of optimization. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Jim Fulton wrote:
Michel Pelletier wrote:
I thought XML-RPC was now favoured over ZClient?
It's not really favored, both are quite useful. xml-rpc is more for when you want two different system to interoperate. ZClient is very zope specific and probably gives you a bit more functionality than xml-rpc (because it has the 'remote object' abstraction).
I don't think ZClient is really Zope specific. At least it's not supposed to be. It does have a richer API, including features like authentication and header support.
My bad, you're right.
Also, ZClient is much faster i've found, probably due to the marshalling/unmarshalling necessary for xml-rpc.
That's interesting......Hm. ZClient has to marshal. I suspect that xml-rpc wants some sort of optimization.
I made a test script to make about 300 xmlrpc calls to various manage_ methods. I had to kill it after about a half-hour cuz it was takin so long. Here's the profiler output: ncalls tottime percall cumtime percall filename:lineno(function) 702883/2151 108.980 0.000 288.580 0.134 DT_InSV.py:379(__getitem__) 7704 79.050 0.010 243.900 0.032 xmllib.py:201(goahead) 3852 42.300 0.011 43.160 0.011 FileStorage.py:706(_finish) 158831/158827 38.010 0.000 70.140 0.000 DT_Var.py:258(render) 732178 37.580 0.000 49.450 0.000 re.py:112(match) 495239/51912 36.690 0.000 307.200 0.006 HTTPRequest.py:741(__getitem__) 586827 29.040 0.000 29.040 0.000 re.py:335(group) 499291 22.900 0.000 30.530 0.000 re.py:95(search) 875347 19.500 0.000 19.500 0.000 re.py:290(__init__) 45981 14.090 0.000 39.000 0.001 xmllib.py:539(parse_starttag) 748592 10.680 0.000 10.680 0.000 exceptions.py:65(__init__) 541924 9.420 0.000 9.420 0.000 re.py:297(start) 456273 9.190 0.000 9.190 0.000 re.py:306(end) 107831 8.520 0.000 8.520 0.000 urllib.py:898(quote) 127647/10757 7.400 0.000 300.080 0.028 :0(?) 49113 7.160 0.000 8.240 0.000 User.py:140(getRolesInContext) 127647/4302 6.350 0.000 303.910 0.071 DT_Util.py:325(eval) 45981 5.360 0.000 19.870 0.000 xmllib.py:615(parse_endtag) 111258 5.240 0.000 7.030 0.000 xmllib.py:699(handle_charref) 3856 5.100 0.001 17.020 0.004 BaseRequest.py:224(traverse) 284515 4.510 0.000 4.510 0.000 xmlrpclib.py:387(data) I think you're right about optimizing, why is it spending most of it's time in DT_InSV? -Michel
Michel Pelletier wrote:
Jim Fulton wrote:
Michel Pelletier wrote:
(snip)
Also, ZClient is much faster i've found, probably due to the marshalling/unmarshalling necessary for xml-rpc.
That's interesting......Hm. ZClient has to marshal. I suspect that xml-rpc wants some sort of optimization.
I made a test script to make about 300 xmlrpc calls to various manage_ methods. I had to kill it after about a half-hour cuz it was takin so long.
This sounds pretty fishy. Are you saying that you made less than 300 calls in half an hour? This doesn't seem right at all.
Here's the profiler output:
ncalls tottime percall cumtime percall filename:lineno(function) 702883/2151 108.980 0.000 288.580 0.134 DT_InSV.py:379(__getitem__) 7704 79.050 0.010 243.900 0.032 xmllib.py:201(goahead) 3852 42.300 0.011 43.160 0.011 FileStorage.py:706(_finish)
Hm. This says that the FileStorage _finish method was called 3852 times. This says that 3852 transactions were committed. Since transactions only get committed on write requests, then this suggests 3852 write requests. There should have been way more than 3852 requests to have this many write requests.....
158831/158827 38.010 0.000 70.140 0.000 DT_Var.py:258(render) 732178 37.580 0.000 49.450 0.000 re.py:112(match) 495239/51912 36.690 0.000 307.200 0.006 HTTPRequest.py:741(__getitem__) 586827 29.040 0.000 29.040 0.000 re.py:335(group) 499291 22.900 0.000 30.530 0.000 re.py:95(search) 875347 19.500 0.000 19.500 0.000 re.py:290(__init__) 45981 14.090 0.000 39.000 0.001 xmllib.py:539(parse_starttag) 748592 10.680 0.000 10.680 0.000 exceptions.py:65(__init__) 541924 9.420 0.000 9.420 0.000 re.py:297(start) 456273 9.190 0.000 9.190 0.000 re.py:306(end) 107831 8.520 0.000 8.520 0.000 urllib.py:898(quote) 127647/10757 7.400 0.000 300.080 0.028 :0(?) 49113 7.160 0.000 8.240 0.000 User.py:140(getRolesInContext) 127647/4302 6.350 0.000 303.910 0.071 DT_Util.py:325(eval) 45981 5.360 0.000 19.870 0.000 xmllib.py:615(parse_endtag) 111258 5.240 0.000 7.030 0.000 xmllib.py:699(handle_charref) 3856 5.100 0.001 17.020 0.004 BaseRequest.py:224(traverse) 284515 4.510 0.000 4.510 0.000 xmlrpclib.py:387(data)
I think you're right about optimizing, why is it spending most of it's time in DT_InSV?
First, it's not spending most of it's time in DT_InSV __getitem__, although it's spending more time in that method than in any other single method. The reason it spends so much time in that method is that the method gets called 700 thousand times. That is, it's used alot, because alot of in-tag special variables are used in your test. This is also a pretty non-trivial method, so it's reasonable that it might take a while. All of this seems to me to be somewhat beside the point. The original comment was regarding ZClient vs XML-RPC. It would be more interesting to: - Compare ZClient and XML-RPC for some realistic tests, like yours, - Compare them for simple Python methods that don't do much but return some data, something like: def test(x,y,z): return 'hello world' * 100 - If there is a big difference, use the profiler to try to figure out why XML-RPC is taking so long. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
Jim Fulton wrote:
Michel Pelletier wrote:
I made a test script to make about 300 xmlrpc calls to various manage_ methods. I had to kill it after about a half-hour cuz it was takin so long.
This sounds pretty fishy. Are you saying that you made less than 300 calls in half an hour? This doesn't seem right at all.
My bad, my fingers dropped a zero. I was gonna say 3000. I don't know the exact number but I can find out, i just wrapped a for loop around the code that uploads the book content to zope.org; effectively uploading the book 100 times. I don't know the exact time either, but in the time it ran before i killed it I recompiled and reconfigured a linux kernel three times and had at least four cups of coffee.
Hm. This says that the FileStorage _finish method was called 3852 times. This says that 3852 transactions were committed.
That may be the number of calls it made before i killed it.
Since transactions only get committed on write requests, then this suggests 3852 write requests. There should have been way more than 3852 requests to have this many write requests.....
You mean 300?
First, it's not spending most of it's time in DT_InSV __getitem__, although it's spending more time in that method than in any other single method. The reason it spends so much time in that method is that the method gets called 700 thousand times. That is, it's used alot, because alot of in-tag special variables are used in your test.
I'm not sure why though, I'm not executing any DTML, just uploading a bunch of structured text into File objects (and three DTML methods to view the book).
This is also a pretty non-trivial method, so it's reasonable that it might take a while.
All of this seems to me to be somewhat beside the point.
The original comment was regarding ZClient vs XML-RPC. It would be more interesting to:
- Compare ZClient and XML-RPC for some realistic tests, like yours,
- Compare them for simple Python methods that don't do much but return some data, something like:
def test(x,y,z): return 'hello world' * 100
- If there is a big difference, use the profiler to try to figure out why XML-RPC is taking so long.
Ok, in my spare time i'll cobble up a much better benchmark. -Michel
On 10 Aug 2000 08:10:01 -0500, Jim Fulton wrote:
Michel Pelletier wrote:
Jim Fulton wrote:
Michel Pelletier wrote:
Also, ZClient is much faster i've found, probably due to the marshalling/unmarshalling necessary for xml-rpc.
That's interesting......Hm. ZClient has to marshal. I suspect that xml-rpc wants some sort of optimization.
I made a test script to make about 300 xmlrpc calls to various manage_ methods. I had to kill it after about a half-hour cuz it was takin so long.
This sounds pretty fishy. Are you saying that you made less than 300 calls in half an hour? This doesn't seem right at all.
I'm probably way off base here, but I use xml-rpc to transfer records to a Zope-based application from a bunch of Macs attached to ABI gene sequencers. The data is simple lists of lists and it *appears* to take about 2-3 seconds for the Macs to xml-encode and transfer the data as-is, but it takes typically *3 minutes* for the xmlrpc server to decode and rebuild the table data. This is for about 6000 records representing about 200k of text data (Sun Ultra 5 server). However, if I simply build a string representation of the table data, it takes about 3-4 seconds for the client to stringify, transfer the string data, and the server to reconstruct the table. Guess which method I use? The tests I ran using a SAX-based parser show it might parse and rebuild the data at least 50x faster than the current Python-only xmlrpclib. Now maybe something else is going on wrt manage* methods and xmlrpc, (I'm using external methods for managing the data transfer and am not storing the raw data in the Zope database), but my external methods-based calls rip along at a pretty good clip as long as I convert the data to a (few) string(s) before sending.
Jim Fulton wrote:
It's not really favored, both are quite useful. xml-rpc is more for when you want two different system to interoperate. ZClient is very zope specific and probably gives you a bit more functionality than xml-rpc (because it has the 'remote object' abstraction).
I don't think ZClient is really Zope specific. At least it's not supposed to be. It does have a richer API, including features like authentication and header support.
The problem I have with ZClient is that the result of method calls is always a string - while with XML-RPC I can get lists/dictionaries/integers, if that's what the method returned. -- Itamar S.T. itamar@maxnm.com Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C
Itamar Shtull-Trauring wrote:
Jim Fulton wrote:
It's not really favored, both are quite useful. xml-rpc is more for when you want two different system to interoperate. ZClient is very zope specific and probably gives you a bit more functionality than xml-rpc (because it has the 'remote object' abstraction).
I don't think ZClient is really Zope specific. At least it's not supposed to be. It does have a richer API, including features like authentication and header support.
The problem I have with ZClient is that the result of method calls is always a string - while with XML-RPC I can get lists/dictionaries/integers, if that's what the method returned.
I didn't claim that it was better for RPC, but that it simply wasn't Zope specific and has some features that XML-RPC lacks. Frankly, I'd very much like to see the missing features in CML-RPC or SOAP. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
On Wed, 09 Aug 2000 16:06:36 -0400, Jim Fulton <jim@digicool.com> wrote:
That's interesting......Hm. ZClient has to marshal. I suspect that xml-rpc wants some sort of optimization.
xmlrpclib will by default use python's standard library xmllib.py for parsing, which is terribly slow. There is support for an optional C extension, which gave me a 12x speed increase. Check out the source for xmlrpclib.py for pointers. Toby Dickenson tdickenson@geminidataloggers.com
participants (7)
-
andrew -
Chris Withers -
Itamar Shtull-Trauring -
Jim Fulton -
kent＠tiamat.goathill.org -
Michel Pelletier -
Toby Dickenson