Zope Persistence (was: XML-RPC within ZOPE)
Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously. However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is. However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products. Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried. Ole 2005/12/15, Dieter Maurer <dieter@handshake.de>:
Jan-Ole Esleben wrote at 2005-12-11 19:10 +0100:
Is it at all impossible to use XML-RPC _within_ a ZOPE architecture?
In principle yes.
Be careful however: it is easy to introduce deadlocks!
When during request processing you call back into the same Zope via XML-RPC, the calling out request will not complete until the XML-RPC returns a result (this always holds for calls, XML-RPC or other, to an external or internal server).
Zope has a limited amount of workers (the default is 4) able to execute requests. If there are no free workers, requests have to wait for one.
Now imagine that as many requests arrive as there are workers and each of them wants to perform an XML-RPC against the same Zope. Then you have a deadlock: none of the XML-RPC requests will finish, because there are no free workers. An no worker will ever become free again, because each of them waits for its XML-RPC to finish.
Therefore, you should directly call internal resources (rather than use XML-RPC).
-- Dieter
Jan-Ole Esleben wrote:
Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously.
However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is.
However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products.
Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried.
Ole
You could try XMLRPCMethod. It creates its own worker thread and has a configurable timeout. Michael [1] http://www.zope.org/Members/EIONET/XMLRPC -- http://zope.org/Members/d2m http://planetzope.org
Thanks, I will definitely look into that for my immediate problems. Ole 2005/12/16, Michael Haubenwallner <michael@d2m.at>:
Jan-Ole Esleben wrote:
Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously.
However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is.
However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products.
Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried.
Ole
You could try XMLRPCMethod. It creates its own worker thread and has a configurable timeout.
Michael
[1] http://www.zope.org/Members/EIONET/XMLRPC
-- http://zope.org/Members/d2m http://planetzope.org
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude? - C On Dec 16, 2005, at 6:40 AM, Jan-Ole Esleben wrote:
Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously.
However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is.
However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products.
Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried.
Ole
2005/12/15, Dieter Maurer <dieter@handshake.de>:
Jan-Ole Esleben wrote at 2005-12-11 19:10 +0100:
Is it at all impossible to use XML-RPC _within_ a ZOPE architecture?
In principle yes.
Be careful however: it is easy to introduce deadlocks!
When during request processing you call back into the same Zope via XML-RPC, the calling out request will not complete until the XML-RPC returns a result (this always holds for calls, XML-RPC or other, to an external or internal server).
Zope has a limited amount of workers (the default is 4) able to execute requests. If there are no free workers, requests have to wait for one.
Now imagine that as many requests arrive as there are workers and each of them wants to perform an XML-RPC against the same Zope. Then you have a deadlock: none of the XML-RPC requests will finish, because there are no free workers. An no worker will ever become free again, because each of them waits for its XML-RPC to finish.
Therefore, you should directly call internal resources (rather than use XML-RPC).
-- Dieter
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude?
I'm sorry, but I don't understand _that_ question. What am I trying to _exclude_? XML-RPC and (the concept of) persistence aren't incompatible. XML-RPC (on a ZOPE server) and ZOPE persistence are, to the extent described by others in this thread, not by me. I just described a couple more cases (and specifically one case) where it is bad that things are this way (and cannot be solved by better design). My point is: this doesn't happen _within_ single "programs", but everywhere there is even the slightest bit of communication there is a chance of it happening. Ole
- C
On Dec 16, 2005, at 6:40 AM, Jan-Ole Esleben wrote:
Thanks; this is a problem we are well aware of. Our solution is to increase the amount of workers, obviously.
However, I'm increasingly getting a feeling that for a rather big range of unlikely situations that are nonetheless to be expected, Zope doesn't work _at all_. In a WebServices setting, situations like the one I described, with one server calling back to another server within a call from that latter server and both not knowing anything about the implementation of the other, it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them for persistent objects that actually _do_ change state (unlike mine). The solution is to not have state in your objects, and thus lose instantly most of what Zope is.
However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation (which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products.
Please tell me if I'm wrong with my assumption above, and why. I'm not trying to be inflamatory, this just has me really worried.
Ole
2005/12/15, Dieter Maurer <dieter@handshake.de>:
Jan-Ole Esleben wrote at 2005-12-11 19:10 +0100:
Is it at all impossible to use XML-RPC _within_ a ZOPE architecture?
In principle yes.
Be careful however: it is easy to introduce deadlocks!
When during request processing you call back into the same Zope via XML-RPC, the calling out request will not complete until the XML-RPC returns a result (this always holds for calls, XML-RPC or other, to an external or internal server).
Zope has a limited amount of workers (the default is 4) able to execute requests. If there are no free workers, requests have to wait for one.
Now imagine that as many requests arrive as there are workers and each of them wants to perform an XML-RPC against the same Zope. Then you have a deadlock: none of the XML-RPC requests will finish, because there are no free workers. An no worker will ever become free again, because each of them waits for its XML-RPC to finish.
Therefore, you should directly call internal resources (rather than use XML-RPC).
-- Dieter
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Dec 16, 2005, at 8:02 PM, Jan-Ole Esleben wrote:
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude?
I'm sorry, but I don't understand _that_ question. What am I trying to _exclude_?
You said: """ it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them """
XML-RPC and (the concept of) persistence aren't incompatible. XML-RPC (on a ZOPE server) and ZOPE persistence are, to the extent described by others in this thread, not by me. I just described a couple more cases (and specifically one case) where it is bad that things are this way (and cannot be solved by better design).
AFAICT, people have told you to not use XML-RPC here and when you said it was not possible to avoid the use of XML-RPC, they provided suggestion about how to accomplish what you wanted anyway. So I'm not sure what the exact problem is.
My point is: this doesn't happen _within_ single "programs", but everywhere there is even the slightest bit of communication there is a chance of it happening.
The chance of what happening, sorry? I'm still trying to understand the problem. The only problem that was noted so far was a deadlock potential by Dieter which presumed you were doing XML-RPC requests to the same system which accepts them. This is an unusual thing to do; it wouldn't happen under "normal" circumstances. - C
I don't understand the problem. How is using XML-RPC incompatible with persistence? What are you trying to exclude? I'm sorry, but I don't understand _that_ question. What am I trying to _exclude_? You said: """ it would most certainly be extremely hard to foresee the exact setup of such situations and impossible to exclude them """
That's probably an idiomatic error on my part, sorry. I meant "avoid" (in German, it's the same word).
AFAICT, people have told you to not use XML-RPC here and when you said it was not possible to avoid the use of XML-RPC, they provided suggestion about how to accomplish what you wanted anyway. So I'm not sure what the exact problem is.
The problem is a different one now, and I was referring to the _reasons_ people had for telling me not to use XML-RPC. Part of the problem I have now is that no application on the web is isolated from others, and that ZOPE specifically _comes_ with XML-RPC capabilities on the server's part.
My point is: this doesn't happen _within_ single "programs", but everywhere there is even the slightest bit of communication there is a chance of it happening. The chance of what happening, sorry? I'm still trying to understand the problem. The only problem that was noted so far was a deadlock potential by Dieter which presumed you were doing XML-RPC requests to the same system which accepts them. This is an unusual thing to do; it wouldn't happen under "normal" circumstances.
The problem setup is this; I explained it above, but it this has become a long thread: I write a ZOPE product. I want to make use of other software on the internet and the services that software provides. So I use the methods exposed by that software via SOAP, XML-RPC, whatever. One of those methods actually calls my product back, maybe because the developer has learned that my product itself exposes some (or all) of its functionality via XML-RPC. If this is all inside one call, which I can't avoid explicitly as the developer of just my product, I have a broken transaction on my hands that isn't easy to fix (and maybe impossible). This holds true for the whole product, even if the method called by the second server changed some completely unrelated data. I hope that clears it up a little. Ole
The problem setup is this; I explained it above, but it this has become a long thread:
I write a ZOPE product. I want to make use of other software on the internet and the services that software provides. So I use the methods exposed by that software via SOAP, XML-RPC, whatever.
One of those methods actually calls my product back, maybe because the developer has learned that my product itself exposes some (or all) of its functionality via XML-RPC. If this is all inside one call, which I can't avoid explicitly as the developer of just my product, I have a broken transaction on my hands that isn't easy to fix (and maybe impossible). This holds true for the whole product, even if the method called by the second server changed some completely unrelated data.
I see what you're saying, but how is this specific to Zope? If you write a Perl program and expose it via mod_perl on Apache, and the program calls out to a service that calls back in to the mod_perl program (no matter how broken of a pattern this was), wouldn't the Apache process that was waiting on data be tied up in the same way? - C
Jan-Ole Esleben wrote at 2005-12-16 12:40 +0100:
... However, as I see it, the problem is that what Zope actually _is_ (i.e. mostly the ZODB) is an unhealthy way of coupling data and implementation
Coupling data and behaviour (what you might call "implementation") is the main feature of object orientedness. I find this very productive (especially multiple inheritance). Apart from that: for all solutions you have to make tradeoffs based on your aims. The ZODB aimed at ease of integration and ease of implementing simple multi-threaded applications (such as Web application servers). To make the second easier, it removed a major fault class: locking faults in application code. With the ZODB, you need not to worry about setting locks to make the modification of persitent objects safe. Because of that, you will usually not have problems with deadlocks or data inconsistencies due to forgotten locking. However, this ease has a price: the quite costly ZODB cache maintained for each connection in a connection pool. To limit RAM consumption, the ZODB versions before 3.4 had a limit on the number of connections in the pool. With ZODB 3.4, the number of connections are no longer limited. Connections exceeding the pool size become temporary connections the cache of which is destroyed when the connection is closed. Thus, you can now have a rather small number of ZODB connections (to hold RAM consumption in check) and a rather large number of workers to reduce the risk of deadlocks caused by resource exhaustion. If you are ready to modify ZServer a bit, you could even create workers on demand (usually, they are preallocated). However, this will cost you much of your control over resource usage...
(which is _exactly_ why my implementation didn't work immediately). This of course comes from its origins in TTW development where there wouldn't actually have been many user made products.
I do not think that it has anything to do with TTW development. It is just that Zope did not target arbitrary web services implementations but much simpler web applications such as blogs, CMS, portals, ... This applications usually do have a precise notion of interior and exterior of themselves -- no need to call back to itself via HTTP rather than caller back directly. If you are smart (and you do not want to replace Zope by a different system altogether), then you will implement your own WebServices dispatcher -- one that recognizes when the call it in fact internal and then does make an internal call. If such calls are frequent, then you will gain performance as well. -- Dieter
participants (4)
-
Chris McDonough -
Dieter Maurer -
Jan-Ole Esleben -
Michael Haubenwallner