Re: [Zope] Zope Persistence (was: XML-RPC within ZOPE)
It's not about the threads or processes being tied up and waiting, it's about the transaction breaking: because the internal call (the one from the second server back to the first) changes the object on the first server, and thus when the first server checks wether the object has changed after the transaction should close (during the last return), it finds that indeed it has, and before it could write to it, so it raises a conflict error invariably. Ole 2005/12/17, Chris McDonough <chrism@plope.com>:
The problem setup is this; I explained it above, but it this has become a long thread:
I write a ZOPE product. I want to make use of other software on the internet and the services that software provides. So I use the methods exposed by that software via SOAP, XML-RPC, whatever.
One of those methods actually calls my product back, maybe because the developer has learned that my product itself exposes some (or all) of its functionality via XML-RPC. If this is all inside one call, which I can't avoid explicitly as the developer of just my product, I have a broken transaction on my hands that isn't easy to fix (and maybe impossible). This holds true for the whole product, even if the method called by the second server changed some completely unrelated data.
I see what you're saying, but how is this specific to Zope? If you write a Perl program and expose it via mod_perl on Apache, and the program calls out to a service that calls back in to the mod_perl program (no matter how broken of a pattern this was), wouldn't the Apache process that was waiting on data be tied up in the same way?
- C
On 12/17/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
It's not about the threads or processes being tied up and waiting, it's about the transaction breaking: because the internal call (the one from the second server back to the first) changes the object on the first server, and thus when the first server checks wether the object has changed after the transaction should close (during the last return), it finds that indeed it has, and before it could write to it, so it raises a conflict error invariably.
This is still not a problem that has anything to do with Zope or persistenace, but it is quite simply just a conflict error. It will happen anywhere you do things like this. It will also only happen when you, in the processor of modifying an object, calls a method on another server, which, before that call finishes, makes a call back and modifies the *same* object. I can't currently dream up any scenario where this happens, but I'm happy that Zope raises an error when it does. A system that does NOT raise an error in this situation is broken. If you get problems like this, you need to make changes to the software so that this does not happen. The right way to do that depends on your application. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. But in ZOPE, it is definitely the case that data and program are coupled in an implicit way that makes these cases much harder to debug and avoid, because if the two methods in my example operate on different sets of data, which they probably would, or if one of them did a read before calling the external method and then afterwards a write (on an SQL database maybe), nothing would happen if I used explicit data storage! In Zope, it's just the whole object that's tainted. That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. This had never occured to me as a _real_ problem before. Of course, if you just create self contained code on a small scale that doesn't talk to other programs on the web, you'll likely not run into any great problems because of this. Ole 2005/12/17, Lennart Regebro <regebro@gmail.com>:
On 12/17/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
It's not about the threads or processes being tied up and waiting, it's about the transaction breaking: because the internal call (the one from the second server back to the first) changes the object on the first server, and thus when the first server checks wether the object has changed after the transaction should close (during the last return), it finds that indeed it has, and before it could write to it, so it raises a conflict error invariably.
This is still not a problem that has anything to do with Zope or persistenace, but it is quite simply just a conflict error. It will happen anywhere you do things like this.
It will also only happen when you, in the processor of modifying an object, calls a method on another server, which, before that call finishes, makes a call back and modifies the *same* object.
I can't currently dream up any scenario where this happens, but I'm happy that Zope raises an error when it does. A system that does NOT raise an error in this situation is broken. If you get problems like this, you need to make changes to the software so that this does not happen. The right way to do that depends on your application.
-- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
On 12/17/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true.
You *will* run into these problems in exactly the same cases in any other tool.
But in ZOPE, it is definitely the case that data and program are coupled in an implicit way that makes these cases much harder to debug and avoid, because if the two methods in my example operate on different sets of data, which they probably would, or if one of them did a read before calling the external method and then afterwards a write (on an SQL database maybe), nothing would happen if I used explicit data storage!
It will not happen i Zope in this case either.
In Zope, it's just the whole object that's tainted.
You just said "different sets of data". That reasonably must mean different objects, unless you envision having huge objects with only marginally connected sets of data all stored as attributes. And then you have other problems. :-)
That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code.
Eh, no they aren't. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool.
I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the "internal" call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break.
But in ZOPE, it is definitely the case that data and program are coupled in an implicit way that makes these cases much harder to debug and avoid, because if the two methods in my example operate on different sets of data, which they probably would, or if one of them did a read before calling the external method and then afterwards a write (on an SQL database maybe), nothing would happen if I used explicit data storage! It will not happen i Zope in this case either.
No. It does happen, and it _did_ happen in my original problem. Zope doesn't even care if anything actually changes, it just considers an object changed that set self._p_changed=1.
In Zope, it's just the whole object that's tainted. You just said "different sets of data". That reasonably must mean different objects, unless you envision having huge objects with only marginally connected sets of data all stored as attributes. And then you have other problems. :-)
No. An object usually binds together different sets of data (as in the above example - it has several fields, and that is true for almost any given object). What you are saying is "don't program the ZOPE way", and it would eventually lead to the conclusion that your product classes should not have any persistent data, which is the conclusion I have come to.
That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. Eh, no they aren't.
Please don't just claim stuff. ZOPE has persistent classes. Persistent classes store data that in ordinary programs would not survive as long. In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code. Of course, you could do the same in ZOPE, but I addressed that above, and you could mess up ordinary code, but I think we're agreed that it is not something you should aim for (and I think I have made it clear why I do not believe it's as easy to mess up generic Python code in this way). Ole
Am Samstag, den 17.12.2005, 14:56 +0100 schrieb Jan-Ole Esleben:
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool.
I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the "internal" call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break.
This is wrong IMHO. dict and list are just columns of the same tuple if you speak RDBMS. And there are very few (if any?) databases which do locks only per column. In RDBMS you distribute in different tables to avoid such (e.g. normalize) in ZODB, you just make subclasses of Persistent for your subobjects. (Attributes) HTH Tino
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the "internal" call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. This is wrong IMHO. dict and list are just columns of the same tuple if you speak RDBMS. And there are very few (if any?) databases which do locks only per column. In RDBMS you distribute in different tables to avoid such (e.g. normalize) in ZODB, you just make subclasses of Persistent for your subobjects. (Attributes)
I'm not saying that it absolutely can't be done, it is just very difficult to do it right from the start because most of time you are not even aware you are fiddling with the data model. This of course isn't - I stressed that before - a problem for small scale development. But if you do more complex stuff with ZOPE it's a bit like going back to C - most of the mistakes in your design don't come back to you until much later. However, unlike C, ZOPE is very high level, and the problems stem from too much implicitness and too little separation of said data and code. When you design your model, you know that you are designing data and you handle those data as data explicitly. When you add a field to your class, you have to explicitly check yourself. And of course you could now answer "You should be doing that anyway" and that nothing hppens in perfectly designed code etc., but that's just not how it really works. You should be able to design stuff incrementally with a little experimentation along the way without constantly impending danger of it all crashing down on you. That's how Python works, and RoR etc. In ZOPE, we're back to the temptation to "just stuff a bunch of data into my object". And it's not even obvious that this is a problem, because everything is so tightly interdependent. It's exactly what Python usually avoids ("explicit is better than implicit"). Ole
Am Samstag, den 17.12.2005, 16:43 +0100 schrieb Jan-Ole Esleben:
That ZOPE raises an error is fine. That I _might_ run into such situations with other tools is true. You *will* run into these problems in exactly the same cases in any other tool. I'm sorry, but that's just wrong, and I have given examples of such situations. To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the "internal" call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break. This is wrong IMHO. dict and list are just columns of the same tuple if you speak RDBMS. And there are very few (if any?) databases which do locks only per column. In RDBMS you distribute in different tables to avoid such (e.g. normalize) in ZODB, you just make subclasses of Persistent for your subobjects. (Attributes)
I'm not saying that it absolutely can't be done, it is just very difficult to do it right from the start because most of time you are not even aware you are fiddling with the data model. This of course isn't - I stressed that before - a problem for small scale development. But if you do more complex stuff with ZOPE it's a bit like going back to C - most of the mistakes in your design don't come back to you until much later. However, unlike C, ZOPE is very high level, and the problems stem from too much implicitness and too little separation of said data and code. When you design your model, you know that you are designing data and you handle those data as data explicitly. When you add a field to your class, you have to explicitly check yourself. And of course you could now answer "You should be doing that anyway" and that nothing hppens in perfectly designed code etc., but that's just not how it really works. You should be able to design stuff incrementally with a little experimentation along the way without constantly impending danger of it all crashing down on you. That's how Python works, and RoR etc. In ZOPE, we're back to the temptation to "just stuff a bunch of data into my object". And it's not even obvious that this is a problem, because everything is so tightly interdependent. It's exactly what Python usually avoids ("explicit is better than implicit").
Partly agreed. But then in zope, simple things are simple and complex things complex. Dont say interlocking transactions are simple anywhere else :-) If you want to do advanced stuff you need to get deeper into the concepts. Unfortunately there is no easy way. But now we figured it out, didnt we? :-) Thats why these lists exist - nobody can probably get things right from the start. Happy coding Tino
On 12/17/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
You should be able to design stuff incrementally with a little experimentation along the way without constantly impending danger of it all crashing down on you.
I don't undertand why you say that this isn't possible in Zope.
That's how Python works, and RoR etc. In ZOPE, we're back to the temptation to "just stuff a bunch of data into my object". And it's not even obvious that this is a problem, because everything is so tightly interdependent. It's exactly what Python usually avoids ("explicit is better than implicit").
I agree that there is too much implicitness in Zope 2. I don't agree that persistance is a part of that. It isn't implicit at all. Maybe it's not easy to understand, but it isn't particularily implicit, and neither is it uncontrollable, as you seem to say. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
On 12/17/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked.
This statement is incorrect.
So if I have, say, a list field that contains some data and a dictionary field that contains some other data, and the "internal" call changes the dict while the original call changes the list, that breaks the transaction, while in usual situation in a database, nothing would break.
If you use all non-persistent aware dicts and lists, yes. Using only dicts and lists is suboptimal in most OO-languages, and that goes for Python too. It is correct that with Zope, the negative impact of that type of pre'ogramming-style is increased. This is however simply solved by not doing that type of suboptimization, and instead using persistant-aware objects.
No. It does happen, and it _did_ happen in my original problem. Zope doesn't even care if anything actually changes, it just considers an object changed that set self._p_changed=1.
No, se above.
You just said "different sets of data". That reasonably must mean different objects, unless you envision having huge objects with only marginally connected sets of data all stored as attributes. And then you have other problems. :-)
No. An object usually binds together different sets of data (as in the above example - it has several fields, and that is true for almost any given object).
Yes. You said "different sets of data". That reasonably means different objects. If it doesn't, yuo need to take a long hard look at your object hierarchy.
What you are saying is "don't program the ZOPE way",
No, in fact, I'm saying "program the Zope way", and I am beginning to suspect that the problem is that you are not.
and it would eventually lead to the conclusion that your product classes should not have any persistent data, which is the conclusion I have come to.
Then you didn't read what I said.
That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code.
Eh, no they aren't.
Please don't just claim stuff.
You do.
ZOPE has persistent classes. Persistent classes store data that in ordinary programs would not survive as long.
Not unless you tell them to. If you don't want the data stored, then do not set it as an attribute on a persistent object.
In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code.
What is implicit with it?
Of course, you could do the same in ZOPE, but I addressed that above, and you could mess up ordinary code, but I think we're agreed that it is not something you should aim for (and I think I have made it clear why I do not believe it's as easy to mess up generic Python code in this way)
Well, sorry, it's not clear for me. To summarize: If you have two different processes changing the same set of data, you will get confllict errors. You claim that you will not, but this is false. You claim that you can't control what a persistent class store. That is false. You claim that the whole Zope product gets locked, that is false. It's on a object-basis, not a product basis. You claim that things get locked that shouldn't be locked in Zope. It is not clear to me why you say that. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. This statement is incorrect.
You're right. It should be "during any transaction where there are potential changes to the object". And I agree, it's actually a lot more complicated than I'd thouht at first. What I describe can happen, but it happens under rather magical circumstances. I have an example where it actually happens; this is code from a product that is instantiated twice, one of those instances called TPCDest, the other TestPChanged; the method called directly is "TestPChanged.test()": security.declareProtected("Use TestPChanged", "test_internal") def test_internal(self, args=None, args2=None): "Called by test via XML-RPC" if args2: self.a.append(1) self._p_changed = 1 if not args: import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) s.TestPChanged.test_internal(1) return self.a security.declareProtected("Use TestPChanged", "test") def test(self, args2=None): "Called directly via ZOPE" if args2: self.a.append(1) self._p_changed = 1 import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) return s.TPCDest.test_internal() Zope obviously looks at the code, because if you remove the "if args2" (which is _never_ true), then a ConflictError happens. Maybe I'm stupid, but could someone point out _where_ exactly I am?
No. It does happen, and it _did_ happen in my original problem. Zope doesn't even care if anything actually changes, it just considers an object changed that set self._p_changed=1. No, se above.
You're right that it doesn't always, but see my example.
Yes. You said "different sets of data". That reasonably means different objects. If it doesn't, yuo need to take a long hard look at your object hierarchy.
Simply not true. What if I have a field titles (that is a hash of titles to Book objects) and a field comments (that is an array of comments on the library)? Is that so obviously not a sane example?
That's my whole point. I think it is a very significant point nonetheless, because this is just an extreme case of what happens when you couple data and programs, and persistent classes are just that: application data inside program code. Eh, no they aren't. Please don't just claim stuff. You do.
I don't. I at least try to give reasons wherever I make a statement that could be challenged.
In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code. What is implicit with it?
See the example for some major implicitness. It's also implicit because you have no control over what a transaction considers a tainted object. (You have no real control over the transaction).
If you have two different processes changing the same set of data, you will get confllict errors. You claim that you will not, but this is false.
I am explicitly talking about changing different sets of data within the same object. You noted that above yourself.
You claim that you can't control what a persistent class store. That is false.
I don't. I say that if I want to avoid such problems as I describe altogether, I cannot store anything in a persistent class and thus lose most of what makes ZOPE so interesting.
You claim that the whole Zope product gets locked, that is false. It's on a object-basis, not a product basis.
I never said that, or at least I didn't mean it. If it sounded like that, I apologize (maybe I used Product metonymically for instance somewhere, which would not be a good idea, I agree).
You claim that things get locked that shouldn't be locked in Zope. It is not clear to me why you say that.
See the example. Ole
On 12/18/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
To simplify, in ZOPE, for any given product, during a transaction the product is effectively locked. This statement is incorrect.
You're right. It should be "during any transaction where there are potential changes to the object".
No, this statement is still correct. Replace "product" with "object" and you are right.
And I agree, it's actually a lot more complicated than I'd thouht at first.
Or simpler. ;-)
but it happens under rather magical circumstances. I have an example where it actually happens; this is code from a product that is instantiated twice, one of those instances called TPCDest, the other TestPChanged; the method called directly is "TestPChanged.test()":
security.declareProtected("Use TestPChanged", "test_internal") def test_internal(self, args=None, args2=None): "Called by test via XML-RPC" if args2: self.a.append(1) self._p_changed = 1 if not args: import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) s.TestPChanged.test_internal(1) return self.a
security.declareProtected("Use TestPChanged", "test") def test(self, args2=None): "Called directly via ZOPE" if args2: self.a.append(1) self._p_changed = 1 import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) return s.TPCDest.test_internal()
Zope obviously looks at the code, because if you remove the "if args2" (which is _never_ true), then a ConflictError happens.
Maybe I'm stupid, but could someone point out _where_ exactly I am?
I don't understand the question. I would like to point out that you set self._p_changed even when you don't change anything. :-) It seems to me that you say that if you process calls test and another calls test_internal, you get a conflict error. This is correct, and as noted before, that code would create a conflict in any type of environment. If you have two processes trying to modify the same dataset, which is what your example seems to do, then you get a conflict. What you claim, is that if you have two processes in Zope change two different datasets, you get a conflict. That statement is still false.
Yes. You said "different sets of data". That reasonably means different objects. If it doesn't, yuo need to take a long hard look at your object hierarchy.
Simply not true. What if I have a field titles (that is a hash of titles to Book objects) and a field comments (that is an array of comments on the library)? Is that so obviously not a sane example?
Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency...
In ordinary systems, you would have to find a way to store the data and retrieve it, thus having a model that isn't implicit and entangled with your code.
What is implicit with it?
See the example for some major implicitness.
What is implicit with it?
It's also implicit because you have no control over what a transaction considers a tainted object. (You have no real control over the transaction).
Thats still not true.
I am explicitly talking about changing different sets of data within the same object. You noted that above yourself.
Then these different sets should be different objects, and the object should be an object container: Problem solved.
You claim that you can't control what a persistent class store. That is false.
I don't. I say that if I want to avoid such problems as I describe altogether, I cannot store anything in a persistent class and thus lose most of what makes ZOPE so interesting.
This is still not correct.
You claim that things get locked that shouldn't be locked in Zope. It is not clear to me why you say that.
See the example.
I don't see how the example examplifies this. It is obvious to me that you have misunderstood something. I don't know what yet, though. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
but it happens under rather magical circumstances.
That's my problem. Can you, from the ZOPE documentation, predict that the example below will cause a ConflictError? It doesn't if you alter it only slightly! Please try!
security.declareProtected("Use TestPChanged", "test_internal") def test_internal(self, args=None, args2=None): "Called by test via XML-RPC" if args2: self.a.append(1) self._p_changed = 1 if not args: import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) s.TestPChanged.test_internal(1) return self.a
security.declareProtected("Use TestPChanged", "test") def test(self, args2=None): "Called directly via ZOPE" if args2: self.a.append(1) self._p_changed = 1 import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) return s.TPCDest.test_internal()
I don't understand the question. I would like to point out that you set self._p_changed even when you don't change anything. :-)
I know. This is just example code. Just imagine that both methods change completely unrelated sets of data in addition to not changing self.a.
It seems to me that you say that if you process calls test and another calls test_internal, you get a conflict error.
It doesn't really in most cases. And of course I can see now where it does in this instance, but this is based on Zope looking at the code, determining that self.a is changed before the _p_changed, and marking the object tainted because of that.
This is correct, and as noted before, that code would create a conflict in any type of environment. If you have two processes trying to modify the same dataset, which is what your example seems to do, then you get a conflict. What you claim, is that if you have two processes in Zope change two different datasets, you get a conflict. That statement is still false.
Actually, I don't think we're getting anywhere with this same dataset/different dataset distinction. It wouldn't happen in a database using application because there would be no transaction for "self.a". You see, nothing happens to it, so why would there be one? This only happens when you mix your data with your code and have implicit transactions handled by the server.
Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency...
But that's part of my point: I need to go out of my way to circumvent Python, and I need to be really careful, because using dicts and lists might still work. Nothing is enforced, and where it breaks is hard to predict.
See the example for some major implicitness. What is implicit with it?
I explained this above. Transaction handling in Zope (someone else pointed that out in this thread), Zope looking at the code to determine that self.a has changed (which isn't really documented anywhere obvious).
You claim that things get locked that shouldn't be locked in Zope. It is not clear to me why you say that. See the example. I don't see how the example examplifies this.
I have tried to make it a little clearer in this message.
It is obvious to me that you have misunderstood something. I don't know what yet, though.
I think we might be misunderstanding each other because we both place different value on implicitness and explicit design of data inside code. I am mostly talking about what is, pragmatically, good programming and a supportive environment. Ole
On 12/18/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
I know. This is just example code. Just imagine that both methods change completely unrelated sets of data in addition to not changing self.a.
Well, yes, but since both a and also b is non-persistence aware lists, that means that you in fact change neither a or b, but self. If however, you do change unrelated sets of data, such as the persistence aware object self.a, and the persistance aware object self.b, then you do NOT get a conflict error.
Actually, I don't think we're getting anywhere with this same dataset/different dataset distinction. It wouldn't happen in a database using application because there would be no transaction for "self.a". You see, nothing happens to it, so why would there be one?
There isn't one in this case either, unless you set self._p_changed = 1, which you of course do...
This only happens when you mix your data with your code and have implicit transactions handled by the server.
There is no mixing of data and code going on, so we are definitely not going anywhere with TAHT distinction. ;-)
Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency...
But that's part of my point: I need to go out of my way to circumvent Python, and I need to be really careful, because using dicts and lists might still work. Nothing is enforced, and where it breaks is hard to predict.
No, it's dead easy to predict, as soon as you understand that you should not modify non-persistent aware attributes, and expect that to work optimally. You may be right that doing that should raise an error, but I also don't exactly see how to make that happen.
See the example for some major implicitness.
What is implicit with it?
I explained this above. Transaction handling in Zope (someone else pointed that out in this thread), Zope looking at the code to determine that self.a has changed (which isn't really documented anywhere obvious).
I'm sorry, I still don't understand what implicitness you are talking about.
It is obvious to me that you have misunderstood something. I don't know what yet, though.
I think we might be misunderstanding each other because we both place different value on implicitness and explicit design of data inside code. I am mostly talking about what is, pragmatically, good programming and a supportive environment.
No, I think the misunderstanding is that you are overcomplicating something that is really quite simple. But I'm not sure. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
What you say is all perfectly true except: 1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)! 2. You shouldn't use lists and dicts - it should say this "on the front page". It is never really mentioned in any way that intuitively leads to such problems as we are now talking about. It isn't very obvious that things work like this when you look at the documentation, and 3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though! All the stuff that you claim as being obvious really isn't all that obvious. And of course, once you've been burned by something like this and the confusion it engenders, you will only use PersistentMapping etc., but I still don't know what _exactly_ to expect from ZOPE in terms of behaviour with mutable objects that aren't Persistent (because of the XML-RPC inconsistency and the self._p_changed inconsistency both mentioned above). Ole 2005/12/18, Lennart Regebro <regebro@gmail.com>:
On 12/18/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
I know. This is just example code. Just imagine that both methods change completely unrelated sets of data in addition to not changing self.a.
Well, yes, but since both a and also b is non-persistence aware lists, that means that you in fact change neither a or b, but self.
If however, you do change unrelated sets of data, such as the persistence aware object self.a, and the persistance aware object self.b, then you do NOT get a conflict error.
Actually, I don't think we're getting anywhere with this same dataset/different dataset distinction. It wouldn't happen in a database using application because there would be no transaction for "self.a". You see, nothing happens to it, so why would there be one?
There isn't one in this case either, unless you set self._p_changed = 1, which you of course do...
This only happens when you mix your data with your code and have implicit transactions handled by the server.
There is no mixing of data and code going on, so we are definitely not going anywhere with TAHT distinction. ;-)
Yes, sorry, having non-persistent aware dictionaries or arrays, and then complaining that you have problems with persitency...
But that's part of my point: I need to go out of my way to circumvent Python, and I need to be really careful, because using dicts and lists might still work. Nothing is enforced, and where it breaks is hard to predict.
No, it's dead easy to predict, as soon as you understand that you should not modify non-persistent aware attributes, and expect that to work optimally. You may be right that doing that should raise an error, but I also don't exactly see how to make that happen.
See the example for some major implicitness.
What is implicit with it?
I explained this above. Transaction handling in Zope (someone else pointed that out in this thread), Zope looking at the code to determine that self.a has changed (which isn't really documented anywhere obvious).
I'm sorry, I still don't understand what implicitness you are talking about.
It is obvious to me that you have misunderstood something. I don't know what yet, though.
I think we might be misunderstanding each other because we both place different value on implicitness and explicit design of data inside code. I am mostly talking about what is, pragmatically, good programming and a supportive environment.
No, I think the misunderstanding is that you are overcomplicating something that is really quite simple. But I'm not sure.
-- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
Jan-Ole Esleben wrote:
security.declareProtected("Use TestPChanged", "test_internal") def test_internal(self, args=None, args2=None): "Called by test via XML-RPC" if args2: self.a.append(1) self._p_changed = 1 if not args: import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) s.TestPChanged.test_internal(1) return self.a
security.declareProtected("Use TestPChanged", "test") def test(self, args2=None): "Called directly via ZOPE" if args2: self.a.append(1) self._p_changed = 1 import xmlrpclib s = xmlrpclib.ServerProxy("http://USER:PASSWORD@localhost:8080/", allow_none=True) return s.TPCDest.test_internal()
1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)!
I strongly doubt it. Zope does not "inspect code". There must be a problem in your testing. Note that if self.a is a standard list, the self.a.append(1) doesn't have any impact on the persistence mechanism or transactions either.
2. You shouldn't use lists and dicts - it should say this "on the front page". It is never really mentioned in any way that intuitively leads to such problems as we are now talking about. It isn't very obvious that things work like this when you look at the documentation, and
You should understand the persistence mechanism of a framework before using it. Zope's is simple to understand in the case where you work with a single transaction. Here you have a remote server that does a callback to the local and thus starts a new transaction parallel to the current one. In addition it touches the same object. Frankly that's a bizarre use case, nothing is "simple" about it.
3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though!
The successive XML-RPC call you describe provoke new transactions, surely you're aware of that? Whereas just calling a function of course doesn't. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)! I strongly doubt it. Zope does not "inspect code". There must be a problem in your testing. Note that if self.a is a standard list, the self.a.append(1) doesn't have any impact on the persistence mechanism or transactions either.
Please, try it out. Delete the if clause and the append in it.
3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though! The successive XML-RPC call you describe provoke new transactions, surely you're aware of that? Whereas just calling a function of course doesn't.
I'm aware of that. But ZOPE offers XML-RPC and as there is nothing in the documentation about such complex interactions. Also, I was directly responding to what the previous poster had written. Ole
On 12/18/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
I strongly doubt it. Zope does not "inspect code". There must be a problem in your testing. Note that if self.a is a standard list, the self.a.append(1) doesn't have any impact on the persistence mechanism or transactions either.
Please, try it out. Delete the if clause and the append in it.
No, Zope does not do any code analysis. Code gets executed or not, and that's it. If you observe behaviour you can only explain by assuming un-executed code to have side-effects, you will have to re-examine your assumptions. Get a good debugger instead. -- Martijn Pieters
On 12/19/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)! I strongly doubt it. Zope does not "inspect code". There must be a problem in your testing. Note that if self.a is a standard list, the self.a.append(1) doesn't have any impact on the persistence mechanism or transactions either.
Please, try it out. Delete the if clause and the append in it.
Little bit tricky to try out as testers need to guess what all the missing code is. Also, for this kind of code demonstration, rather than directions for commenting/uncommenting and relying on the tester to restart the server between trails, provide different methods or even different classes. Meanwhile I'm camping over at I-strongly-doubt-it.
3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls. From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though! The successive XML-RPC call you describe provoke new transactions, surely you're aware of that? Whereas just calling a function of course doesn't.
I'm aware of that. But ZOPE offers XML-RPC and as there is nothing in the documentation about such complex interactions. Also, I was directly responding to what the previous poster had written.
Zope provides for XML-RPC as a server. And it does so with internal transaction support. But there is no extra transactional framework for XML-RPC clients (which your example has). Or for that matter acting as an XML-RPC server nested within some external transaction (which your example has). Either design to avoid this or get your hands dirty with transactions. Some starting points for examples would include any of the relational database adapters. Also in the same space is MaildropHost. michael
Little bit tricky to try out as testers need to guess what all the missing code is.
Any standard persistent ZOPE product wrapped around this will do. These are the only methods in a ZOPE product that inherits from Item, Persistent, RoleManager and Implicit.
Also, for this kind of code demonstration, rather than directions for commenting/uncommenting and relying on the tester to restart the server between trails, provide different methods or even different classes.
If I did that, people would say "somethings wrong with your testing" and they would have to do a line-by-line comparison and still have room to doubt my statement.
But there is no extra transactional framework for XML-RPC clients (which your example has). Or for that matter acting as an XML-RPC server nested within some external transaction (which your example has).
I was initially talking about a more complex situation that has been gradually simplified. ZOPE should be able to act as an XML-RPC client to the outside world. As to me not understanding what you all are saying: I believe I understand, I know that theoretically - from a merely "technical" viewpoint, it works, I just look at it from a different angle. Of course you have to know a framework and its limitations, but this error I stumbled upon strikes me as really complex (even though you keep repeating it's very simple if you just acknowledge xyz), for instance because the docs "aren't perfect" and say that "You must explicitly signal any changes made to mutable attributes (such as instances, lists, and dictionaries) or use persistent versions of mutable objects, like ZODB.PersistentMapping (see below for more information on PersistentMapping.)" - the _first_ option being mutable attributes! Also, there is no "below". I believe you when you say that it is really very simple and I shouldn't have to worry if I follow the right principles, that you then go on to state (principles that aren't really that explicit anywhere - such as "don't use XML-RPC to talk to ZOPE from ZOPE" and "don't use mutable subobjects at all"). However, this is a further bit of distance between a ZOPE user and the Python he knows, and it also isn't as true for complex situations with complex requirements that I could probably just jot down in "dumber" frameworks such as RoR or Django or TurboGears. Because I know without having to check myself where I am designing Data, I explicitly _make_ changes (to the DB) and am thus very aware of where complex interactions might occur etc. This is probably one of those cases where no one is really "right" because a mixture of viewpoints is involved, I feel a bit as if we were talking about wether Lisp is better than Smalltalk. However, even if it is only the docs that are lacking I think it would be sensible to acknowledge that as a problem. It obviously was a problem for me, as of course was the complexity of my setup and several other factors. I am not an inexperienced programmer, so I would think that just because other people rarely stumble upon it it is not an insignificant problem. Imagine designing a large system based on some only marginally false assumptions about ZOPE transactions, persistence and XML-RPC interaction, and then running into this in a key component. Ole
On 12/19/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
Little bit tricky to try out as testers need to guess what all the missing code is.
Any standard persistent ZOPE product wrapped around this will do. These are the only methods in a ZOPE product that inherits from Item, Persistent, RoleManager and Implicit.
Also, for this kind of code demonstration, rather than directions for commenting/uncommenting and relying on the tester to restart the server between trails, provide different methods or even different classes.
If I did that, people would say "somethings wrong with your testing" and they would have to do a line-by-line comparison and still have room to doubt my statement.
/me shurgs. I prefer code that runs. :-)
But there is no extra transactional framework for XML-RPC clients (which your example has). Or for that matter acting as an XML-RPC server nested within some external transaction (which your example has).
I was initially talking about a more complex situation that has been gradually simplified. ZOPE should be able to act as an XML-RPC client to the outside world.
As to me not understanding what you all are saying: I believe I understand, I know that theoretically - from a merely "technical" viewpoint, it works, I just look at it from a different angle. Of course you have to know a framework and its limitations, but this error I stumbled upon strikes me as really complex (even though you keep repeating it's very simple if you just acknowledge xyz), for instance because the docs "aren't perfect" and say that "You must explicitly signal any changes made to mutable attributes (such as instances, lists, and dictionaries) or use persistent versions of mutable objects, like ZODB.PersistentMapping (see below for more information on PersistentMapping.)" - the _first_ option being mutable attributes! Also, there is no "below". I believe you when you say that it is really very simple and I shouldn't have to worry if I follow the right principles, that you then go on to state (principles that aren't really that explicit anywhere - such as "don't use XML-RPC to talk to ZOPE from ZOPE" and "don't use mutable subobjects at all"). However, this is a further bit of distance between a ZOPE user and the Python he knows, and it also isn't as true for complex situations with complex requirements that I could probably just jot down in "dumber" frameworks such as RoR or Django or TurboGears. Because I know without having to check myself where I am designing Data, I explicitly _make_ changes (to the DB) and am thus very aware of where complex interactions might occur etc. This is probably one of those cases where no one is really "right" because a mixture of viewpoints is involved, I feel a bit as if we were talking about wether Lisp is better than Smalltalk. However, even if it is only the docs that are lacking I think it would be sensible to acknowledge that as a problem. It obviously was a problem for me, as of course was the complexity of my setup and several other factors. I am not an inexperienced programmer, so I would think that just because other people rarely stumble upon it it is not an insignificant problem. Imagine designing a large system based on some only marginally false assumptions about ZOPE transactions, persistence and XML-RPC interaction, and then running into this in a key component.
Joining many transactional systems into one mega transactional beast is requires attention to details. How about getting away from this notion a mega transaction with nested transactions? For example, make use of transaction.commit() before making XML-RPC calls. Though you well need to decide what happens in the event of a ConflictError being thrown from somewhere else and causing the initial request to be retried. michael
On 12/18/05, Jan-Ole Esleben <esleben@gmail.com> wrote:
What you say is all perfectly true except:
1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error. With the ineffectual code above it (that never gets executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)!
Possibly, but that is irrelevant. If you try to force a conflict error in a place where there should not be one in a real situation, you can hardly complain that there is no conflict error. ;-)
2. You shouldn't use lists and dicts - it should say this "on the front page". It is never really mentioned in any way that intuitively leads to such problems as we are now talking about. It isn't very obvious that things work like this when you look at the documentation, and
Well, there may be a documentation problem in this area, I haven't read any docs for years. :-) But it is pretty clearly stated that objects should be aware of persistency, and dicts and lists aren't. That's where there are PersistentLists and PersistentDicts.
3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls.
I don't think it is.
From what you say, it should be the same within the ZOPE system as when using XML-RPC. It gets more complicated with XML-RPC though!
No, it doesn't. Really. It doesn't. Stop overcomplicating things. It's much easier than you think. We just need the right way to explain it to you and we haven't found it yet. We will soon.
All the stuff that you claim as being obvious really isn't all that obvious. And of course, once you've been burned by something like this and the confusion it engenders, you will only use PersistentMapping etc., but I still don't know what _exactly_ to expect from ZOPE in terms of behaviour with mutable objects that aren't Persistent (because of the XML-RPC inconsistency and the self._p_changed inconsistency both mentioned above).
You should know not to mutate them. Full stop. Do not use mutable objects unless they are persistent aware. I still think you are over-complicating things. It may not be obvious, but its is pretty darn easy. :-) -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.cps-project.org/
Jan-Ole Esleben wrote:
1. In the example, just setting _p_changed=1 does _not_ lead to a conflict error.
I can assure you it will if any other connection to that ZODB does the same thing on the same object...
executed) it _does_. So there _is_ some implicit magical stuff going on and ZOPE tries to take care that only subobjects change (but incompletely)!
Rubbish. You're likely misunderstanding your own example, or not understanding the context in which it's executing...
2. You shouldn't use lists and dicts - it should say this "on the front page".
Why? You want us to rewrite all the docs just to cater for your weird use cases? Tell you what, how about you jost google for "zope conflict error" and have a read?!
3. It is especially confusing that ZOPE behaves differently when using XML-RPC calls.
It doesn't. It behaves exactly the same as for any other request, w.r.t. zodb transactions...
XML-RPC though! All the stuff that you claim as being obvious really isn't all that obvious.
Maybe for you ;-)
PersistentMapping etc., but I still don't know what _exactly_ to expect from ZOPE in terms of behaviour with mutable objects that aren't Persistent (because of the XML-RPC inconsistency
There is NO xml-rpc inconsistency...
and the self._p_changed inconsistency both mentioned above).
_p_changed is pretty hugely documented in all the zodb docs I've ever read. I dunno, maybe you're stupid, lazy, or both, but please quite whining... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Jan-Ole Esleben wrote:
That's my problem. Can you, from the ZOPE documentation, predict that the example below will cause a ConflictError?
Certainly, try and ask a simple version of these questions on zodb-dev@zope.org and let Tim or Dieter have some fun ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Jan-Ole Esleben wrote:
Yes. You said "different sets of data". That reasonably means different objects. If it doesn't, yuo need to take a long hard look at your object hierarchy.
Simply not true. What if I have a field titles (that is a hash of titles to Book objects) and a field comments (that is an array of comments on the library)? Is that so obviously not a sane example?
You have to understand the Zope persistence mechanism if you want to have write access to subparts of an objects in parallel transactions not cause write conflict errors. Basically, if you want to have that, your object can't be a single persistent object, but be composed of different persistent subobjects.
See the example for some major implicitness. It's also implicit because you have no control over what a transaction considers a tainted object. (You have no real control over the transaction).
Of course you do. An object is "tainted" either if it's Persistent and one of its attribute is updated, or if it's not Persistent and reachable from a Persistent object that got "tainted". "Persistent" means a subclass of persistence.Persistent.
If you have two different processes changing the same set of data, you will get confllict errors. You claim that you will not, but this is false.
I am explicitly talking about changing different sets of data within the same object. You noted that above yourself.
If you want that, again, then your "different sets of data" have to be designed using the Persistent base class. PersistentList and PersistentMapping are examples. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
Ole, I'm getting pretty tired of this thread, even reading it is annoying. You're whining about stuff which isn't a problem, and making out like Zope/ZODB has all these big critical flaws, casting huge assertions without having enough understanding to make them... Jan-Ole Esleben wrote:
security.declareProtected("Use TestPChanged", "test_internal") def test_internal(self, args=None, args2=None): "Called by test via XML-RPC" if args2: self.a.append(1) self._p_changed = 1
What the hell are you doing that for? If self.a is a simple python list, of course you're in trouble. Make it a PersistentList, or better yet, an OOTreeSet, and you'll be much better off...
Zope obviously looks at the code,
I can assure you Zope does ZERO code introspection... Seriously, your examples are overcomplicated, your use cases are bizarre and your attitude of "it must be Zope's fault" is more than a little wearing. No-one's forcing you to use Zope, I'd implore you to go and use another framework if you're so convinced Zope "does it wrong" ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
participants (7)
-
Chris Withers -
Florent Guillaume -
Jan-Ole Esleben -
Lennart Regebro -
Martijn Pieters -
Michael Dunstan -
Tino Wildenhain